What happens behind the technical scenes at the Free Software
Foundation (FSF)? The FSF SysOps team consists of two full-time tech
team employees and a handful of dedicated volunteers that work to keep
our technical infrastructure operational, as well as take on new
projects. The FSF SysOps team has worked tirelessly over the past six
months to ensure the smooth functioning of the FSF's systems and we
will highlight some of the major projects done. From fighting
Distributed Denial of Service (DDoS) attacks to downsizing our office
space, here are some highlights from the last few months.
The FSF SysOps team successfully fought off several DDoS attacks
that threatened to disrupt our operations. These attacks result in
slow response times for users visiting our sites. We have blocked
countless addresses and have written several abuse reports this year
alone. We will have to continue to persevere and adapt, as the attacks
are ongoing and mutate over time. We do not always hear back from
sites that we send abuse reports to, but we were happy to receive this
response from Digital Ocean last month: "We appreciate your efforts in
helping to clean up the internet!" You can read more about how we ward
off DDoS attacks in our recent Free Software Bulletin article "FSF
SysOps cleaning up the Internet." Fighting abuse is often
thankless work, as well as frustrating because of the nature and abuse
of services. It also keeps the team preoccupied. Proactive measures
must be taken to prevent these attacks from taking our services
offline.
In August, we migrated our physical office. If you had visited
the FSF office at our old location on 51 Franklin St., you know that
the tech team had accumulated a large amount of equipment over the
years. With the help of our intern, Anush V., we ended up using tech
markets, the Internet, recycling organizations, a flea market style
office party, and junk collection. We were able to recycle and sell
all of our old tech that we no longer needed and organize what we had
left. And we cataloged the historically important memorabilia among
our tech equipment for the memorabilia auction that will happen
in March so that you can get unique, personal souvenirs of the FSF.
In order to move all of the essential tech and RYF equipment to
their new homes, we spread our equipment out over both a storage space
and the data centers. We had to do a lot of preparation, ranging from
designing the layout of the storage space with Inkscape to rethinking
how we manage our software that runs our operations. We also built new
servers to replace the ones held in the office, upgraded our hardware,
ordered sturdy shelving with wheels, and made some improvements to our
stacks.
We moved the services that were housed in the old office closet into
the data center. To shield these services once they were moved to a
publicly accessible place, we had to implement new security measures.
Apart from doing a lot of mental work, we also had to complete the
physical act of moving our remaining essential equipment to the
storage unit. We were able to do this with team work and splitting up
the main tasks within the tech team. I drove the rental van to and
from the old office, the data center, and the storage unit and moved
all of the gear with the help of Anush, Andrew, and Craig. We designed
the storage unit with nine racks in a way that one person can access
anything in the unit by themselves. Ian focused on the data center
migration where we needed to install new servers and migrate services
and data to them. As a team, we were able to do the majority of the
move within twenty-four hours.
Ian and I do not always divide and conquer like we did on moving day.
After the that day, we ran into some technical trouble migrating our
telephone system running on an Asterisk server from a physical
machine to a virtual machine (VM). Our Asterisk server was one
dedicated machine with Plain Old Telephone Service (POTS) copper
connections running into a specialized FXS PCIe card. The server was
in need of operating system upgrades and the file system was too old
to just copy it to the data center. If you are not familiar with
Asterisk, the project is very old, the documentation very dense, and
the configurations (configs.) can get complicated. We used pair
programming to team up and get this task done quickly. We shared a
Tmux session on the target VM while talking to each other through
our Mumble instance, which we use for encrypted, low-latency
voice chats. Tmux is a terminal multiplexer that allows us to run and
control several terminals, each running different programs, from a
single window.
On our new server, we installed a fresh VM of the latest Trisquel
version and installed the Asterisk package. We meticulously compared
all of the new default config. files with our old config. files. We
moved all of the relevant configs. to the new server, updated them to
match the new ones when necessary, and left anything that we no longer
needed commenting out in the new configs. On top of that, we had to
strengthen the security of the user configs. on the new framework. We
had to relearn how Asterisk expects prerecorded audio files to be
formatted and document this process. The entire undertaking took about
one night of hacking together. Finally, we ran into some configuration
issues with Linphone, the desktop client we use for Voice Over IP
calls, and shared a workaround with staff to get things working again.
Our phone lines are now back up and operational through teamwork!
As you can tell, the FSF SysOps team, one of our newest initiatives to
increase transparency as well as find ways for volunteers to help us,
has had their hands full these past few months. We still have a lot of
work to do. We really want to put more time and energy into working
more closely with the community. While we have published our internal
documentation publicly, been more active on IRC, and built an IRC bot
to help those that are not always connected to IRC to participate, we
are still working on becoming more transparent and finding new ways to
work together. We're always looking for volunteers, so if you have
technical skills and some spare time, you can find information about
joining the FSF SysOps team at https://www.fsf.org/volunteer/.
We know not everyone is in a position to, but if you can, can you
support our efforts by making a donation? Or, an associate
membership is a great show of support we can rely on, and an annual
FSF associate membership translates to a mere $2.69 USD per week,
or $0.38 USD per day! We need more resources to continue our work, but
my request is even bigger, because we have to do more. Associate
members will also be able to enjoy all the associate member
benefits. Will you help us reach our year-end fundraising
goal of $400,000 USD this year-end? If you join as an associate
member this period, we will send you a set of five unique postcards to
help you promote computer user freedom.
Technological freedom is important more than ever!
