Starting on February 15th, our primary network AS401720 was hit with a series of L3/L4 DDoS attacks from a Mirai botnet. For those not aware, a DDoS attack is designed to take down services and networks (not breach something). The attacks peaked at over 100 Mpps (million packets per second) and over 10 Gbps in traffic bursts. The primary target was our Matrix homeserver.
We're confident that we know who is responsible. We believe this was not a state-sponsored attack. We're not going to name the individual or group publicly at this time, but we are preserving all of the evidence. At this time no one has publicly come forward to claim responsibility.
This post covers what happened, what we did to mitigate it, and what we're doing to harden our infrastructure going forward.
AS401720 is our core network. It hosts most of our public-facing Secure Infrastructure (SI) services, including the Unredacted Matrix server, XMPP.is, and several internal systems. Our other networks and services were not affected, such as NoiseNet - our experimental anycast network designed for privacy.
Our core network is typically low in traffic volume, and built with reliability in mind. However, our router could not handle the influx of high packet volume & low packet size attacks that we received. These types of attacks are specifically designed to exhaust CPU cycles, and that's exactly what they achieved.
The primary target was our Matrix homeserver. However, it affected nearly all of our core services due to the CPU exhaustion it caused on our router.
You can always check the status of our services at status.unredacted.org.
The timeline above covers the sequence of events, so we won't repeat it here. The short version is that we migrated all affected services behind Cloudflare's Magic Transit in about three hours, and withdrew our original prefix announcements to prevent the attacker from pivoting around it. No services went down after the migration.
What didn't work was relying on our upstream's DDoS mitigation. We were caught at a bad time with their systems being reworked, but it exposed a dependency we shouldn't have had. Our router was also never specced for this kind of packet rate, and that's something we should have planned for.
This was a targeted attack against our Matrix homeserver. We're confident that we know who is responsible. We're not disclosing that information publicly right now, but this was not the work of a government or state-sponsored actor. No one has publicly come forward to claim responsibility.
The attack used a Mirai botnet, which is widely available and commonly rented. The targeting of our Matrix homeserver specifically, combined with other indicators we've observed, makes us confident in who did this.
This attack exposed weaknesses in our core network that we're now addressing.
Magic Transit stays. We're keeping all of our core services behind Cloudflare's Magic Transit going forward. It proved itself during this incident, and we're not going back to relying on upstream mitigations alone (at least until we are very confident they work well).
Router hardening. Our router was the single point of failure. It was never built to handle 100Mpps+ of low-size packet floods, ideally we can fix that. We're evaluating hardware upgrades that can handle high packet rates without CPU exhaustion. We're not completely sure that this is something we'll do due to rising hardware costs. However, we'll be looking into it.
Reviewing what else belongs behind NoiseNet. NoiseNet was designed in part to distribute load and mitigate DDoS attacks through its anycast edge. Our core services on AS401720 were not behind it. As we stabilize NoiseNet, we'll evaluate whether some of these services should be routed through it for the additional resilience it provides. However, this will take additional time as we have yet to develop proper automation for NoiseNet.
Running infrastructure that helps people access the open Internet and communicate privately means accepting that it will be attacked. We've dealt with abuse reports, threats, and now sustained DDoS attacks. None of it changes what we do or why we do it.
We spent countless hours building out our infrastructure and network to be resilient, and we're working on making them even more so. Although we’re just a small non-profit, the reliability of our services like our Matrix server, XMPP.is, FreeSocks, and our Tor relays is crucial to us. We aim to be as transparent as possible when things go wrong.
If you want to help us build more resilient infrastructure, consider making a donation. If you have questions, join one of our community chats.
The Internet as we know it is in danger and democracy is under attack. Around the world state censors are working to restrict access to information, and prevent people from sharing their own. With growing access to technology, we need to ensure that information remains free at serious infrastructure scale – and we think we…
Internet censorship and surveillance have gotten worse every year. Security matters more than ever as AI-driven vulnerability discovery gets better. The people we build for depend on our machines to reach the open Internet safely, and every one of those machines runs Linux. The kernel underneath is the part that arguably matters most. Our Tor…
The Matrix.org Foundation posted yesterday that “public Matrix servers will also have to uphold age verification laws, as misguided as they might be.” This is true for them. They’re based in the UK and subject to the Online Safety Act. But it is not true as a general statement, and it’s not true for us.…
