So, I want to apologize for the recent outrage and unavailability of the SocialHub. For what happened was entirely my responsibility: the crontab that serve to handle local backups of the critical service files has a line to keep only the last two months of archives locally. On this line, there was a hard to catch typo that would prevent the backup rotation from happening, leaving the hard disk filling up with useless backup files over time.
BKP_HOME="..."
...
# Keep only 2 months worth of backups to save some space
find $KBP_HOME -name \*.tar.gz -atime +60 -delete
Now that you know it, and because of the excerpt context, you should be able to find the culprit. I did immediately once I found that the files were not deleted…
Correct answer is...
$KBP_HOME is an empty string, so the find command fails. $BKP_HOMEis the right path to backup files…
Or why we need more admins…
So, that brings again the question of who is taking care of all this.
As you may know, only I (@how) and @nightpool are currently in charge of administering the server. And we’re both quite busy elsewhere, although we, at least, cover a number of timezones.
In recent discussions, especially after @aschrijver burned out and left the moderators team (fortunately he’s been active anyway and very present in this community), we have insisted that the community, at some point, should be able to take care of itself. @weex has stepped up to organize the #standards:fep and animate the community. But we need more committed involved[1] people to form a steering committee and ensure that daily, or hourly tasks of sysadmin, well-being and animation are taken care of without anyone burning out.
Once in a while (it takes minutes to hours a month).
If we’re enough people: we can cover the timezones for rapid action.
Once we have a sysadmin team
We can work on making the service more automated, scalable.
We can eventually improve the service (e.g., plug into the Github issues, plug into the Fediverse using a custom ActivityPub plugin for Discourse that we’ve been willing for a while)
Maybe the ActivityPub plugin thing is orthogonal to sysadmin though…
I can certainly chip in a few hours a month to help with sysadmin. I’ve got numbers 1, 2, and 4 covered. Through first and foremost an artist, I’ve been supporting myself doing web development for a number of years. Could pick up numbers 3 and 5 if you can tolerate a bit of a learning curve.
I think @stevebate also stepped up on the Fediverse. I guess @edwardficklin and @stevebate can proceed to send me or @nightpool an SSH public key that we can add to the server.
Process would be like:
Let the community know who is stepping up, in case someone has good reasons to block them
Create user accounts, including README files replicating the doc already present on the server
Make the accounts accessible via SSH
Add users to the sudo group
Make them admin here.
Set up a call between sysadmin to onboard with a live tmux session for Q&A