| Please consider adding [email protected] to your address book, which
will ensure that our messages reach you and not your spam box!
Read and share online:
https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots
Dear Free Software Supporter,
The FSF SysOps team consists of two full-time tech team employees and
a handful of dedicated volunteers. A large part of our work is running
the software and physical servers that host websites and other
services for GNU, FSF, and other free software projects, including
virtual machines for the browser extension JShelter, the desktop
environment and software collection KDE, and Sugar Labs, an
organization that creates learning tools for children. We recently
counted seventy different services, and have a dozen physical servers
across two Boston-area data centers.
Since we last wrote, much has happened, and while I'd love to
talk about all of it, including the process of deploying four new
servers to our data centers, I want to focus on the huge task of
maintaining our services in the face of ongoing (and increasing)
distributed denial of service (DDoS) attacks. A DDoS attack typically
happens when attackers control thousands or millions of machines and
get them all to send requests or other traffic to a target
server. Then, the server gets overwhelmed with processing those
requests and fails to respond to requests from legitimate users. A
common way of defending against a DDoS attack, which we often use, is
to figure out a way of identifying which IP addresses are sending
requests as part of the DDoS, and then have the server ignore requests
from those IP addresses.
Our infrastructure has been under attack since August 2024. Large
Language Model (LLM) web crawlers have been a significant source of
the attacks, and as for the rest, we don't expect to ever know what
kind of entity is targeting our sites or why.
In the fall Bulletin, we wrote about the August attack on
gnu.org. That attack continues, but we have mitigated it. Judging
from the pattern and scope, the goal was likely to take the site down
and it was not an LLM crawler. We do not know who or what is behind
the attack, but since then, we have had more attacks with even
higher severity.
To begin with, GNU Savannah, the FSF's collaborative software
development system, was hit by a massive botnet controlling about five
million IPs starting in January. As of this writing, the attack is
still ongoing, but the botnet's current iteration is mitigated. The
goal is likely to build an LLM training dataset. We do not know who or
what is behind this.
Furthermore, gnu.org and ftp.gnu.org were targets in a new
DDoS attack starting on May 27, 2025. Its goal seems to be to take
the site down. It is currently mitigated. It has had several
iterations, and each has caused some hours of downtime while we
figured out how to defend ourselves against it. Here again, the goal
was likely to take our sites down and we do not know who or what is
behind this.
In addition, directory.fsf.org, the server behind the Free
Software Directory, has been under attack since June 18. This likely
is an LLM scraper designed to specifically target Media Wiki sites
with a botnet. This attack is very active and now partially mitigated.
As we developed programs to identify IP addresses belonging to the
botnet, they sometimes misidentified legitimate user's IP
addresses. We've removed them from the list of DDoS IP addresses and
improved our defenses to be more precise. If you do not have access to
gnu.org right now, please send us an email at [email protected]
with your IP address and we will look into it. If you are having
trouble with a VPN (virtual private network), try switching exit nodes
and skip writing us -- we know our attackers use VPNs, which leads us
to block the ones they are using.
More recently, automated software build systems have become an issue
for us. These usually go by the non-obvious term CI/CD, which stands
for "continuous integration or continuous deployment." They send
automated requests to check for new code on Savannah in order to
rebuild their software. They often send far more requests than is
necessary, which looks and acts like a DDoS attack even though it is
not intended to be. The CI/CD tooling does not typically have contact
information labeling their traffic, so we do not have a way to contact
them if there is a problem outside of banning their addresses or
sending abuse reports if we can find a place to send them. We had to
block some of these IP addresses, which often prompts them to search for
better ways to accomplish the same goals.
On top of all of that, we have our run-of-the-mill standard crawlers,
SEO (search engine optimization) crawlers, crawlers pretending to be
normal users, crawlers pretending to be other crawlers, uptime
systems, vulnerability scanners, carrier-grade network address
translation, VPNs, and normal browsers hitting our sites. It is taxing
for our sites and for our team of staff and volunteers, since we have
to figure out a specific defense approach for each attack. Some of the
abuse is not unique to us, and it seems that the health of the
web has some serious problems right now.
When you visit a website, it might send your browser one or more
JavaScript programs. These JavaScript programs are usually
proprietary. We explain this more in "The JavaScript Trap." If a
website sends you a free JavaScript program, you can develop a
modified version, share that with other people so they can benefit,
and you can configure your browser to run your modified version
instead of what the website sends. But some JavaScript programs are
malware, which do things like spy on you, and the only modification
any user would want is to stop it from ever running.
Some web developers have started integrating a program called
Anubis to decrease the amount of requests that automated
systems send and therefore help the website avoid being DDoSed. The
problem is that Anubis makes the website send out a free JavaScript
program that acts like malware. A website using Anubis will respond to
a request for a webpage with a free JavaScript program and not the
page that was requested. If you run the JavaScript program sent
through Anubis, it will do some useless computations on random numbers
and keep one CPU entirely busy. It could take less than a second or
over a minute. When it is done, it sends the computation results back
to the website. The website will verify that the useless computation
was done by looking at the results and only then give access to the
originally requested page.
At the FSF, we do not support this scheme because it conflicts with
the principles of software freedom. The Anubis JavaScript program's
calculations are the same kind of calculations done by crypto-currency
mining programs. A program which does calculations that a user does
not want done is a form of malware. Proprietary software is
often malware, and people often run it not because they want to,
but because they have been pressured into it. If we made our website
use Anubis, we would be pressuring users into running malware. Even
though it is free software, it is part of a scheme that is far too
similar to proprietary software to be acceptable. We want users to
control their own computing and to have autonomy, independence, and
freedom. With your support, we can continue to put these principles
into practice.
Even though we are under active attack, gnu.org,
ftp.gnu.org, and savannah.gnu.org are up with normal
response times at the moment, and have been for the majority of this
week, largely thanks to hard work from the Savannah hackers Bob,
Corwin, and Luke who've helped us, your sysadmins. We've shielded
these sites for almost a full year of intense attacks now, and we'll
keep on fighting these attacks for as long as they continue.
Our full-time FSF tech staff is just two systems administrators, and
we currently lack the funds to hire more tech staff any time soon. I
know many of the readers support the free software movement in a
variety of ways which we appreciate greatly, but in order to improve
our staffing situation we need more associate members.
Can you join us in our crucial work to guard user freedom and defy
dystopia? Become an associate member today! Every associate
member counts, and every new member will help us reach our fundraising
goal of 200 new members. By supporting us today, you help defy
the dystopia Big Tech is trying to bring on us.
We know not everyone is in a position to donate $140 USD or more,
which is why we also offer the Friends membership at $35 USD that
comes with a few less benefits. In addition, you can now apply
to receive a sponsored FSF membership.
Thank you for supporting the tech team!
Happy hacking,
Michael McMahon & Ian Kelling
Your FSF Systems Administrators
| 