Internet censorship and surveillance have gotten worse every year. Security matters more than ever as AI-driven vulnerability discovery gets better. The people we build for depend on our machines to reach the open Internet safely, and every one of those machines runs Linux.

The kernel underneath is the part that arguably matters most. Our Tor exit relays, FreeSocks, our proxies, XMPP.is, our Matrix server – they all sit on top of it. Compromise the kernel and nothing above it is safe.

So we built our own. linux-hardened-unredacted is a hardened kernel we build, sign, and publish as .deb packages for Debian 13 (trixie), with Ubuntu coming soon, on amd64. It starts from upstream Linux, applies the linux-hardened patch set maintained by Levente Polyak (anthraxx), and adds our own config, signing, and an automated, reproducible build pipeline.

Repository: git.unredacted.org/unredacted/linux-hardened-unredacted

Why we built it

We run a lot of servers – over 300 of them. Hundreds of terabytes a month move across our Tor exits and our censorship-evasion and secure-infrastructure services. Most of that runs on Debian, increasingly on hardware we own and rack ourselves.

Those machines carry other people’s traffic. A kernel bug on one of them is about the worst case we can think of, which is why the kernel sat near the top of our list to harden. We built it for ourselves, ran it across nearly the whole fleet, then released it for anyone else on Debian.

Finding kernel bugs used to mean a person reading the code, spotting the flaw, and writing the exploit. Less so now. Fuzzing and automated analysis turn up bugs faster every year, and they work for attackers exactly as well as they work for us – more serious bugs, less time between a flaw going public and someone pointing it at a real person.

We respond on two fronts. We cut the attack surface so fewer bugs are reachable and the survivors are harder to use, and when a fix lands upstream we rebuild, re-sign, and ship as fast as we can.

Where linux-hardened came from

We didn’t write the hardening patches. linux-hardened was started by Daniel Micay – the developer behind GrapheneOS – as the mainline-Linux counterpart to that work. It builds on grsecurity/PaX and the kernel’s own Kernel Self-Protection Project, and some of its ideas have since reached mainline. The slab freelist hardening in every kernel today began as Micay’s patches before it landed upstream.

These days Levente Polyak (anthraxx), an Arch Linux developer, maintains it. GrapheneOS archived its own copy and points to his as the project’s home; it’s what Arch ships, and it’s what we build on. Tracking a tree that plenty of other people already run beats maintaining a private fork only we ever look at. What we add on top is small enough to audit in an afternoon.

What the hardened kernel changes

This kernel makes the bugs that slip through harder to exploit, and rips out features that mostly serve as a way in. A kernel runs to millions of lines of code, most of which never executes on boxes like ours, and every piece we don’t need is one more door. We keep what we use, harden it, and drop the rest.

It clears memory both when it hands it out and when it takes it back. Normally freed memory keeps whatever was in it – a password, a key, part of someone’s traffic – until something overwrites it. Ours zeroes it on the way out and again on the way back, so a bug that reads memory it shouldn’t finds zeroes, not leftovers. Same for the kernel’s own stack: when a syscall returns, the space it used is wiped, so stale arguments and pointers aren’t sitting there for the next call to leak.

It treats memory errors as attacks. A big share of kernel exploits come down to one thing: a write past the end of a buffer, or a copy with the wrong length. This kernel checks the size of every copy between itself and your programs, randomizes where objects land on the heap, and wraps allocations in canaries, so an overflow trips a check instead of becoming an exploit. A lighter check runs the entire time the machine is up, sampling a fraction of allocations to catch use-after-free and out-of-bounds bugs in real traffic – cheap enough to leave on in production. And it loads itself at a random address on every boot, so an attacker can’t assume anything sits where it did last time.

Nothing gets to modify the running kernel. Kernel memory is never writable and executable at once, so injected code has nowhere to run. Drivers won’t load unless they carry a signature we created, which keeps tampered modules out. The kernel runs in integrity lockdown from boot – on every machine, not just Secure Boot ones – closing the interfaces that could modify it at runtime, root included. On a Secure Boot machine, the signed boot chain goes further and stops anything tampering with the kernel before it even loads.

We also cut whole features such as direct access to physical memory (/dev/mem), live kernel memory dumps (/proc/kcore), loading a new kernel at runtime (kexec), hibernation (which writes an unverified image of the kernel to disk), and a pile of old, rarely-used syscalls. Heavy tracing tools – perf, BPF, ptrace – are limited to the administrator instead of every process.

All of this comes with tradeoffs, and you should know them before installing:

• Out-of-tree modules (NVIDIA, ZFS, VirtualBox) need to be rebuilt against our headers and signed. DKMS handles this with a per-machine key you enroll once.
• Rootless Docker and Podman, the Chromium sandbox, Snap, and Flatpak all want unprivileged user namespaces, which we leave off. A single config line turns them back on.
• kexec, kdump, and hibernation are gone (a problem for laptops).

Every workaround we know about lives in the incompatibilities doc. And none of this is permanent: the hardened kernel installs alongside your existing one, GRUB lists both, and if something breaks you reboot into the stock kernel and you’re back where you started.

How it compares to a stock Debian kernel

Debian’s kernel is already hardened. Out of the box it ships stack-protector-strong, KASLR, FORTIFY_SOURCE, hardened usercopy, W^X (write-xor-execute) memory, slab freelist hardening, and heap zeroing on allocation. Plain Debian is a good place to start. We keep all of it and push further:

“Reject unsigned modules: always” means a tampered driver won’t load on our kernel even on a machine without Secure Boot, where stock Debian would load it anyway. “Removed” means more than “disabled”: the code isn’t compiled into the kernel at all, so it can’t be turned back on at runtime.

None of this makes Debian’s kernel a bad choice. It’s a sensible default that has to boot on every laptop, server, and obscure embedded board anyone might own. We’re making narrower choices. These are our machines, so dropping hibernation and kexec costs us nothing.

How it works

Most of the actual work went into the build process and automation.

The config

We don’t keep a finished .config in the repo. A checked-in config goes stale – it drifts from upstream and quietly sheds protections one kernel version at a time, and you don’t notice until you go looking for something else.

Instead we build it fresh each time from two pieces. The base is anthraxx’s own config, the one that tracks the patch set; we pin it to an exact commit and hash and check its GPG signature before the build touches it. On top goes our overlay – around ninety lines pinning the hardening options we care about. A couple dozen of them diverge from anthraxx. Some go further: we refuse unsigned modules, run lockdown in integrity mode, and zero registers on function return, among others. The rest are either the same protection done differently or a deliberate trade-off, like dropping a plugin that would break reproducible builds. The remaining lines pin defenses we want kept – including ones anthraxx already sets – so a later change can’t silently drop one.

Then the build checks that every line of our overlay actually made it into the final config. Miss one and the build fails inside two minutes, naming the option that didn’t land. That check has earned its keep more than once: it caught an upstream rename of a stack-hardening option, and a missing build dependency that had silently dropped a whole family of protections. A second check diffs each build against our last release and fails if an inherited protection went missing – so we pick up new hardening from anthraxx automatically and don’t lose old hardening by accident.

The build

Push a version tag and the pipeline takes over. It pulls the kernel source from kernel.org and checks the signature against the release keys, downloads the linux-hardened patch and checks it against anthraxx’s key, and stops if either fails. Everything is re-hashed before use, so nothing can be swapped between steps. Then it builds the config, runs the fidelity check above, and compiles.

Out comes a kernel image around 230 MB, headers, an optional debug package (~1.3 GB), and small meta-packages – so you install one package name and apt carries you forward to new kernels over time. The pipeline signs the image for Secure Boot, signs the modules during the build, records everything in a manifest, and attaches it all to a draft release for a person to review. Timestamps come from the tag’s date and the build container is pinned by digest, so the result is reproducible.

Signing and Secure Boot

On a Secure Boot machine you trust our kernel once, by enrolling our certificate through MOK at the next reboot. There’s no Microsoft-signed shim – we left it out on purpose.

Microsoft’s program requires the signing key to live in a hardware security module (HSM), or an equivalent FIPS 140-2 device, with documented two-factor authorization. Ours doesn’t – it lives in our CI’s secret store and loads only for the build job. That’s what lets the pipeline sign on its own and turn an upstream fix into a signed kernel within hours. A key that needs a human at a hardware token for every signature can’t move that fast.

The cost is real: both signing keys are online, so anyone who fully compromised our build infrastructure could sign a malicious kernel. Our security policy says exactly that. It’s the price of automated builds without an HSM, for now.

Reproducible builds

Because the key is online, we give you a way to check what we shipped. Every release carries a manifest: the exact build container, every input and its hash, the hash of each package before signing, the signing certificate’s fingerprint, and a full parts list of the build environment down to the compiler version. Take the manifest, rebuild from the same public inputs and toolchain, strip the signatures off both sides, and the bytes should match. No private key needed. We run that check every night.

It’s also why we turn off one optional plugin that mixes randomness into the build: it would break a byte-for-byte match, and proving what we shipped matters more to us than the bit of entropy it adds.

Responding to CVEs

The pipeline watches upstream and handles the repetitive parts; the decisions stay with us. Every hour it checks anthraxx for a new release, and when one lands it opens a tracking issue with a best-effort list of the CVEs fixed since the version we’re on. It won’t start a build by itself – a person verifies the upstream signature first. Once a week it checks anthraxx’s config and opens a pull request if anything changed. The release is hand-done too: the build hands a person a draft to review and boot-test, and because the kernel is a root of trust for boot, a human signs off before it ships.

For a serious CVE we aim to queue a build within 24 hours of the upstream fix and publish within 48 once the reproducibility check passes. That timeline is what the automation buys us – and donations are what pay the people who run it.

Shipping it

Packages go to our own Debian repository behind a CDN, which keeps the load off our infrastructure. The repository metadata is signed with a separate key you verify out of band before your first install.

One caveat: CDNs usually log which IP pulled which kernel version, and when. The packages are signed and tamper-evident, but if your kernel version is something you’d rather not advertise, that download timing is still metadata. Pull updates over Tor or a VPN, or mirror the repository locally. For us it’s a fair trade for getting updates to the people who need them, ourselves included.

Where does it go from here?

There’s plenty left to do.

• The toolchain isn’t fully pinned yet – the last step toward a build anyone can reproduce exactly.
• It’s amd64-only on Debian today. Ubuntu, Debian 12, arm64, and more kernel flavors are next.
• An HSM would eventually close the online-key gap and let us ship a Microsoft-signed shim, so Secure Boot users could skip the manual key enrollment. That one comes down to funding and time.

We’ll keep writing about it as it develops, like we do with our other projects.

We need your help

Unredacted is a 501(c)(3) non-profit. We build free and open infrastructure that helps people evade censorship and protect their privacy. All of it runs on donations, and responding to a kernel CVE in hours takes real engineering time.

If you find any of this useful, please consider making a donation. If you’re an organization that wants to collaborate, get in touch.

And if you run Debian: install it, read the config policy, rebuild a release, and check your bytes against ours. Tell us if they don’t match. We appreciate every one of you testers.

A huge and special thank you to our donors for making this work possible.