Service back up! Or why we need more admins

Community
1

Continuing the discussion from SocialHub needs active administration:

Service bakc up!

So, I want to apologize for the recent outrage and unavailability of the SocialHub. For what happened was entirely my responsibility: the crontab that serve to handle local backups of the critical service files has a line to keep only the last two months of archives locally. On this line, there was a hard to catch typo that would prevent the backup rotation from happening, leaving the hard disk filling up with useless backup files over time.

BKP_HOME="..."

...

# Keep only 2 months worth of backups to save some space
find $KBP_HOME -name \*.tar.gz -atime +60 -delete

Now that you know it, and because of the excerpt context, you should be able to find the culprit. I did immediately once I found that the files were not deleted…

Correct answer is...

$KBP_HOME is an empty string, so the find command fails.
$BKP_HOMEis the right path to backup files…

:person_facepalming:

Or why we need more admins…

So, that brings again the question of who is taking care of all this.

As you may know, only I (@how) and @nightpool are currently in charge of administering the server. And we’re both quite busy elsewhere, although we, at least, cover a number of timezones.

In recent discussions, especially after @aschrijver burned out and left the moderators team (fortunately he’s been active anyway and very present in this community), we have insisted that the community, at some point, should be able to take care of itself. @weex has stepped up to organize the #standards:fep and animate the community. But we need more committed involved[1] people to form a steering committee and ensure that daily, or hourly tasks of sysadmin, well-being and animation are taken care of without anyone burning out.

So, here we go again…

  1. The difference between committed and involved is best illustrated by a hamburger: the cow is committed, while the chicken is involved. ↩︎

2

I am moderating :slight_smile: just not trying to ‘animate’ a community on same activity level as before. Wrt the FEP there’s currently this team:

See: FEP Editors on Codeberg.

2 Likes
3

Can you provide more information about the admin role(s), the skills required, estimated time requirements, and so on? Thanks!

4

(note that the typo was intentional :wink: )

5

Admin requirements

  1. Know *NIX, especially a bit of basic sysadmin (Debian, bash, sudo, troubleshooting)
  2. SSHing (we use ed25519 keys with strong passphrases)
  3. Know Discourse (to be able to upgrade from the console or run tasks)
  4. Be discreet and never peek at people’s data.
  5. Knowledge or Ruby and Rails a plus.

Time constraints

  • Once in a while (it takes minutes to hours a month).
  • If we’re enough people: we can cover the timezones for rapid action.

Once we have a sysadmin team

We can work on making the service more automated, scalable.
We can eventually improve the service (e.g., plug into the Github issues, plug into the Fediverse using a custom ActivityPub plugin for Discourse that we’ve been willing for a while)
Maybe the ActivityPub plugin thing is orthogonal to sysadmin though… :wink:

2 Likes
6

:wave: I can certainly chip in a few hours a month to help with sysadmin. I’ve got numbers 1, 2, and 4 covered. Through first and foremost an artist, I’ve been supporting myself doing web development for a number of years. Could pick up numbers 3 and 5 if you can tolerate a bit of a learning curve.

1 Like
7

I think @stevebate also stepped up on the Fediverse. I guess @edwardficklin and @stevebate can proceed to send me or @nightpool an SSH public key that we can add to the server.

Process would be like:

  1. Let the community know who is stepping up, in case someone has good reasons to block them
  2. Create user accounts, including README files replicating the doc already present on the server
  3. Make the accounts accessible via SSH
  4. Add users to the sudo group
  5. Make them admin here.
  6. Set up a call between sysadmin to onboard with a live tmux session for Q&A
  7. Hope for the best :wink:
8

What’s your preferred way to send the public key?

9

Encrypted email (0x0B1496B7E33603EB) is good, or a PM here or on Matrix.

10

I’ve changed my mind.

Please disregard my previous offer.

1 Like
11

Hey @how @nightpool, I’m willing to have a go at this, but I will need a bit of training, or at least some hand holding until I get my feet under me. Looking at the list of admin requirements, I tick at least some of the boxes.

  1. I know my way around basic BASH, been using desktop GNU/Linux for about 15 years.

  2. Have done this for Bridge Seat stuff, need to learn to manage a consistent SSH key robustly anyway.

  3. No experience here, but I’ve installed and upgraded GNU/Linux installs many times, and I’m a quick study.

  4. 1000% respect for privacy.

  5. I did a pair programming project about 5 years ago where we dug through a RoR package.

3 Likes
12

I would be happy to have a sysadmin meeting one of these days, and that would be a good moment to look around and clarify tasks. I won’t be much available though before at least one week.

13

No worries, I wish I’d checked in and realised this need about 6 months ago when the thread started. Because in Aotearoa we’re just heading into summer, which means I’ll be off working at outdoor events and my net access and time to dig into new tech stuff will be extremely limited from about the middle of next month. However, I do plan to take a laptop with me on my travels, and keep as many plates spinning as I can.

If you folks could do everything that needs doing over your winter to define the process for bringing on a new sysadmin, I might be able to carve out some time in Feb for some orientation. That way I can hit the ground running in late March/ early April when the outdoor events season is over.

1 Like