Service back up! Or why we need more admins

Continuing the discussion from SocialHub needs active administration:

Service bakc up!

So, I want to apologize for the recent outrage and unavailability of the SocialHub. For what happened was entirely my responsibility: the crontab that serve to handle local backups of the critical service files has a line to keep only the last two months of archives locally. On this line, there was a hard to catch typo that would prevent the backup rotation from happening, leaving the hard disk filling up with useless backup files over time.

BKP_HOME="..."

...

# Keep only 2 months worth of backups to save some space
find $KBP_HOME -name \*.tar.gz -atime +60 -delete

Now that you know it, and because of the excerpt context, you should be able to find the culprit. I did immediately once I found that the files were not deleted…

Correct answer is...

$KBP_HOME is an empty string, so the find command fails.
$BKP_HOMEis the right path to backup files…

:person_facepalming:

Or why we need more admins…

So, that brings again the question of who is taking care of all this.

As you may know, only I (@how) and @nightpool are currently in charge of administering the server. And we’re both quite busy elsewhere, although we, at least, cover a number of timezones.

In recent discussions, especially after @aschrijver burned out and left the moderators team (fortunately he’s been active anyway and very present in this community), we have insisted that the community, at some point, should be able to take care of itself. @weex has stepped up to organize the #standards:fep and animate the community. But we need more committed involved[1] people to form a steering committee and ensure that daily, or hourly tasks of sysadmin, well-being and animation are taken care of without anyone burning out.

So, here we go again…


  1. The difference between committed and involved is best illustrated by a hamburger: the cow is committed, while the chicken is involved. ↩︎

I am moderating :slight_smile: just not trying to ‘animate’ a community on same activity level as before. Wrt the FEP there’s currently this team:

See: FEP Editors on Codeberg.

2 Likes

Can you provide more information about the admin role(s), the skills required, estimated time requirements, and so on? Thanks!

(note that the typo was intentional :wink: )

Admin requirements

  1. Know *NIX, especially a bit of basic sysadmin (Debian, bash, sudo, troubleshooting)
  2. SSHing (we use ed25519 keys with strong passphrases)
  3. Know Discourse (to be able to upgrade from the console or run tasks)
  4. Be discreet and never peek at people’s data.
  5. Knowledge or Ruby and Rails a plus.

Time constraints

  • Once in a while (it takes minutes to hours a month).
  • If we’re enough people: we can cover the timezones for rapid action.

Once we have a sysadmin team

We can work on making the service more automated, scalable.
We can eventually improve the service (e.g., plug into the Github issues, plug into the Fediverse using a custom ActivityPub plugin for Discourse that we’ve been willing for a while)
Maybe the ActivityPub plugin thing is orthogonal to sysadmin though… :wink:

2 Likes

:wave: I can certainly chip in a few hours a month to help with sysadmin. I’ve got numbers 1, 2, and 4 covered. Through first and foremost an artist, I’ve been supporting myself doing web development for a number of years. Could pick up numbers 3 and 5 if you can tolerate a bit of a learning curve.

1 Like

I think @stevebate also stepped up on the Fediverse. I guess @edwardficklin and @stevebate can proceed to send me or @nightpool an SSH public key that we can add to the server.

Process would be like:

  1. Let the community know who is stepping up, in case someone has good reasons to block them
  2. Create user accounts, including README files replicating the doc already present on the server
  3. Make the accounts accessible via SSH
  4. Add users to the sudo group
  5. Make them admin here.
  6. Set up a call between sysadmin to onboard with a live tmux session for Q&A
  7. Hope for the best :wink:

What’s your preferred way to send the public key?

Encrypted email (0x0B1496B7E33603EB) is good, or a PM here or on Matrix.

I’ve changed my mind.

Please disregard my previous offer.

1 Like