Please consider adding [email protected] to your address book, which will ensure that our messages reach you and not your spam box!

Read and share online: https://www.fsf.org/blogs/sysadmin/look-behind-the-scenes-of-the-fsf-tech-team

Dear Free Software Supporter,

This article gives a glimpse behind the scenes of recent work done by the FSF tech team. Read about Prometheus, AMT data import, OS upgrades, and more.

The FSF tech team has continued to keep busy since we last wrote an appeal article! In addition to other ongoing projects, on the network security front we're working to replace our old Nagios alerting stack with Prometheus, a more modern tool that we're currently adapting to our network. We plan on using plain Prometheus, with some of our own automation to configure graphs and site-wide monitoring, rather than integrating with third party systems that typically offer proprietary editions.

There's a fair bit of work going into this change, but it will be worth the effort. Prometheus will highlight the most important issues in alerts based on the time of day, so we can focus on the really important issues when we're out of the office. This will help us with faster issue response times. We've also been fortunate to have very few critical problems in the last several months that would have required us to physically visit the colocation facility.

Some of the other work we're wrapping up is the AMT ("associate membership tool") data import. AMT was our old, internally developed member database. We switched to CiviCRM, an AGPLv3 constituent relationship manager, several years ago, and it really helped with fundraising and emailing people who want to learn more about free software. We had a bit more data that needed to be exported from our old membership database into the new one before we could call it complete. And aside from updating our internal documentation, we're finally done with that data migration. We had the original virtual machine (VM) turned off, and we're using a newer Trisquel installation running the MariaDB database software for working with, filtering, and converting data into an importable format. It's a good feeling to fully replace an outdated system, and, going forward, we can now focus on completing the last few edge cases.

We've also been upgrading between major releases of Trisquel for some of our older systems via apt dist-upgrade. This often works well for systems that don't have a lot of moving parts. For others, we tend to spin up a new VM so we can test the new version of the application after migrating the data from the old server. For those who are curious, I'll briefly cover some details here. Once everything is working on the new machine, we find a time to turn off public facing services on the old server and then migrate the data and update the Domain Name System (DNS) record. We find that this tends to involve a lot more work, but it's more reliable in the cases where we don't know if software that doesn't come from Trisquel itself will be compatible with the new OS dependencies, and whether the data migration will run smoothly. We try to keep downtime to a minimum, but are looking into ways to improve the speed of our major system upgrades. This is necessary because we are only a three-person tech team maintaining over 140 servers, along with help from dedicated GNU volunteers.

We are getting ready to continue work on modernizing our fsf.org site as well. We're in the process of comparing various backend web software that we may use and getting them up to date. For a future iteration of this site, we're considering using a static site generator for simplicity and easy maintenance of upgrades.

In parallel to these bigger projects, we are keeping up the daily work. Part of this is answering questions sent to us by you and others in the free software community. When anyone sends a technical-, licensing-, or campaigns-related question, it gets queued in a ticket system called "RT," which stands for Request Tracker and runs entirely on free software. All technical questions that FSF associate members and free software community members send go to our sysadmin ticket queue. This helps us ensure that we've responded to all the questions coming in and do so in an efficient manner.

We are proud to say that we closed a lot of tickets in the FSF sysadmin ticket queue over the last six months -- 1,115 to be precise! We have many more tickets to work on, but we're doing our best to keep response times low while prioritizing the ones with the greatest impact. That being said, don't hesitate to reach out to us. We will listen to your request and do the best we can with the resources we have. While our RT queues are not a general support service, we do take the time to educate and assist volunteers, especially those who help with GNU and FSF infrastructure.

We take our support position for the GNU project very seriously, and we hope that we can some day expand our team so we can even better support the work of all of the generous volunteers who keep writing GNU software and keep GNU/Linux moving forward as a system.

Can you join as an FSF associate member to help us support GNU, our technical work, the FSF campaigns, and the Licensing and Compliance Lab? The associate member program started in November of 2002 to maintain the core work of the free software movement, independent from major individual or corporate donors. The FSF wanted to be sustained by the community we serve. As of today, membership dues make up most of the FSF's operational costs. Without members we would not be able to carry out the important work the FSF does for the free software movement.

The membership program keeps us working, and the FSF tech team is nonstop working for the free software community. Together with your help, we can strive towards a world where computer user freedom is universal. You can start an FSF associate membership for as little as $10 per month ($5 for students), or $120 per year.

You'll be able to enjoy all the member benefits, which include merchandise discounts, a 16GB bootable membership card, and use of our associate member videoconferencing server. Plus, your membership will count towards achieving our spring goal of 175 new associate members before July 21. When you join as an annual associate member at $120 or more, you'll also be eligible for this year's sustainable and stylish, genuine wood GNU head sticker.

Happy hacking,

Andrew Engelbrecht
Senior Systems Administrator