Category: Tor

  • How we accidentally broke our Tor exit relays

    Making technology work how you expect it to, and keep it working that way can be difficult at times. Changes in configuration or software updates can put services into a broken or half-broken state, and we had the latter happen to us.

    In the spirit of transparency, we are writing about the painful discovery that many of our Tor exit relays (at least 1/3) have been broken for (at the very least) weeks, and possibly months without our knowing.

    I’d like to thank the team at Tor Project for letting us know about the issue, and how to reproduce it, which ultimately led to it being discovered and fixed.

    Root cause

    Credit: https://itsfoss.community/t/its-always-dns-lol/10820

    Based on the image, you might suspect what the issue was.

    It was DNS. Specifically, it was Tailscale’s MagicDNS feature. DNS queries were not getting resolved for some reason that is unknown to us. This means that anyone who ended up connecting to our Tor exit relays failed to connect to nearly every domain/subdomain by failing to resolve hostnames. Connecting to IPs worked just fine.

    Before we go on blaming Tailscale, I want to state that we don’t know why MagicDNS failed in the way we observed, we just know that it did. Ultimately, disabling MagicDNS on our exit relays resolved the issue entirely. When we enabled it, and tested again, it failed. As a result, we’ve left it off.

    Technical analysis

    We knew the symptom was that DNS resolution seemed to fail, which was noted by the nice people at Tor.

    We started by attempting to reproduce the issue ourselves. This required pinning our Tor instance/daemon on our local computer to a specific exit relay’s fingerprint that was exhibiting this strange behavior. For example, by modifying our Tor daemon’s torrc file and adding the below line to it, we could force our local Tor daemon to exit all traffic on that relay.

    ExitNodes F34EE673122518873E717C128E35A389B72C7837

    This fingerprint corresponds to our UnredactedSnowden relay.

    We then pointed one of our browsers to use the local SOCKS proxy the Tor daemon listens on (127.0.0.1 port 9050) to send traffic through Tor.

    When attempting to connect to any website, it failed, but the reason was unclear and did not appear to display an error related to a DNS resolution issue.

    As DNS was still the suspect here, the easiest thing to do was to SSH into that exit relay and run a tcpdump to capture all inbound and outbound packets that used TCP or UDP port 53, such as the one below.

    tcpdump -i any -n port 53

    Once we did that, we discovered that nearly all DNS queries originated from Tor seemingly went out to the 100.100.100.100 MagicDNS IP, but nothing was returned on most queries. We knew at this moment, that it was indeed a DNS resolution problem.

    An anonymized example of what we saw:

    17:11:52.164204 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 40707+ A? domain.com. (45)
17:11:52.172049 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 33552+ A? domain.com. (33)
17:11:52.203409 tailscale0 In  IP 100.100.100.100.53 > 100.69.x.x.22065: 47343 NXDomain 0/1/0 (119)
17:11:52.321235 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 45366+ A? domain.com. (28)
17:11:52.321271 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 16617+ A? domain.com. (35)
17:11:52.321303 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 39612+ A? domain.com. (29)
17:11:52.352491 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 63111+ A? domain.com. (34)
17:11:52.383332 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 15513+ A? domain.com. (35)
17:11:52.501714 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 16308+ A? domain.com. (29)
17:11:52.532238 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 7697+ A? domain.com. (40)
17:11:52.540674 tailscale0 In  IP 100.100.100.100.53 > 100.69.x.x.22065: 22472 0/1/0 (89)
17:11:52.544683 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 8052+ A? domain.com. (47)

    We tried several things;

    • Switching the MagicDNS nameservers to other ones & restarting Tor.
    • Rebooting the exit relay we were testing with to see if it was a strange Tailscale daemon/interface/routing issue.
    • Using dig via CLI on the test relay which queried the MagicDNS IP (100.100.100.100) which worked without issue.

    We were at a loss, and couldn’t figure out what was happening. We then decided to disable MagicDNS on the test relay to see what would happen. It worked, DNS queries started flowing and getting resolved responses via the same nameservers directly.

    We subsequently disabled MagicDNS on the rest of the exit relays with an adhoc Ansible shell command.

    Our conclusion

    The problem appeared to be with the abstraction that MagicDNS does, and queries originating from Tor did not appear to work 99%+ of the time when the feature was enabled. However, queries from dig via CLI appeared to always work. We suspect that MagicDNS fails in some sort of way when too many queries are directed at its 100.100.100.100 IP which is seemingly routed out the tailscale0 interface (& subsequently onto the physical interface). However, this doesn’t make complete sense, as we would expect queries from dig to fail as well.

    We may never know what happened exactly, and we don’t want to leave it in a broken state long enough to figure it out. At this point, it’s safe to say that we are leaving MagicDNS disabled on our Tor exit relays for the foreseeable future.

    Shortly after resolving the issue, our Tor exit relay traffic rate shot up beyond previously normal levels and hit our full capacity (as of writing this).

    In the near future, we will explore running our own local DNS resolver on each exit relay, which we’ve done in the past – but had to move away from due to an overload of bogus queries originated from Tor which also resulted in DNS resolution failures. DNS over HTTPS (DoH) or DNS over TLS (DoT) are also great options we may explore further.

    We hope you found this interesting and insightful. If you enjoy what we do, please consider making a donation. Unredacted is a non-profit organization that provides free and open services that help people evade censorship and protect their right to privacy.

  • New Tor bridge types for Operation Envoy

    In July of last year (2023) we launched Operation Envoy, our effort to deliver packets to and from the Tor network which helps defeat Internet censorship. This is achieved by Unredacted operating Tor bridges, also known as Pluggable Transports. Tor bridges obfuscate (bridge) the connection a user makes when connecting to Tor so that it looks like any normal connection and disguises the fact that they are connecting to the Tor network. Each Pluggable Transport has its own unique way of obfuscating the connection, such as WebTunnel which mimics HTTPS traffic, one of the most common types of traffic on the Internet.

    What a connection to the Tor network looks like with a bridge in the path
    Credit: robertheaton.com/2019/04/06/how-does-tor-work/

    Historically, and for a long time we’ve focused our efforts on deploying dedicated snowflake proxies around the world in strategic locations close to Internet users that face a high level of Internet censorship in their countries. Today, we’ve added a WebTunnel and meek bridge into the mix. Adding more Tor bridge types means that users have more ways to connect to the Tor network in the event that one protocol / obfuscation technique gets blocked.

    Our meek bridge

    How meek works, click the image to learn more

    To deploy our meek bridge, we worked with the team at Tor after volunteering to run a new bridge. Due to how meek works with Tor, there is some setup on their end as well because they use a technique called domain fronting. This is a technique to disguise a connection and route it through popular, and more painful to block CDN networks like Microsoft Azure. Meek bridges remain a crucial method to connect to the Tor network in several countries.

    To see our new meek bridge statistics, you can click here.

    Our WebTunnel bridge

    How HTTPT works, the proxy behind WebTunnel technology

    As described earlier in the post, WebTunnel is a bridge type which mimics HTTPS traffic, one of the most common types of traffic on the Internet. It’s based on HTTPT which resists active probing attacks that censors use to block censorship circumvention techniques. WebTunnel will likely, and ultimately become a very important bridge type for Tor as it rolls out and gains popularity due to the protocol it disguises itself as and its resistance to active probing.

    Our new WebTunnel bridge uses a unique configuration that we came up with to hide the IP of the bridge behind a TCP proxy service. This allows us to easily switch the ‘front’ of the WebTunnel bridge in case its IP gets blocked. In the future, we plan to write about how we did this once we’ve confirmed its stability over time.

    To see our new WebTunnel bridge statistics, you can click here.

    Current Operation Envoy stats

    As it stands today, we have a collective of virtual machines consisting of 31 CPU cores, 40GB of RAM and multi-gigabit unmetered links dedicated to serving Tor bridge traffic across the world.

    Past 7 days of CPU and memory usage, click the image to see live stats

    On an average day, we are pushing almost 2TB of symmetrical bandwidth per day. That’s almost 60TB per month!

    Past 7 days of bandwidth usage, click the image to see live stats

    We can’t make all of this possible without your help. If you like what we do, please consider making a donation. As time goes on, and with more funding we’ll continue to expand our Operation Envoy footprint by deploying more Tor bridges across the world. Your help can make a real impact for Internet censorship circumvention.

  • Operation Envoy: Defeating Censors

    Operation background

    Accessing the uncensored Internet in some countries has never been so difficult. Internet censorship is rising across the world, and content filtering is becoming more difficult to circumvent as technology and censors evolve. Even in countries you wouldn’t expect. However the worst offenders are the ones you would typically suspect, China, Russia and countries who rank low on the World Press Freedom Index.

    The organization, OONI (Open Observatory of Network Interference) monitors internet censorship around the world and produces reports which show that censorship is on the rise. Government censors (governments who implement Internet censorship) are insatiable in their quest to restrict Internet access and keep their citizenry blind and oppressed, just how they like it.


    The question is, what are we doing about it? That’s where Operation Envoy comes in. We want to help deliver messages (network packets) to and from the Tor network. For quite a while now, we’ve been running Tor exit relays which provide valuable bandwidth and processing power to the Tor network which helps people in heavily censored countries access services and information that people in the western world take for granted. While exit relays are an integral part of the Tor network, there’s another part that is critical for accessing it in many countries. Tor bridges and snowflake proxies are the first entry point into the Tor network for many people. What are they you might be wondering? Well, many countries block access to Tor and they’re very good at it, which makes Tor hard to access. That’s where Tor bridges and snowflake proxies step in, and so do we. Bridges and snowflake proxies allow Tor users to access the network via an obfuscated and seemingly normal-looking connection to the bridge or proxy. That bridge or proxy then acts as a literal bridge to the Tor network.

    Censors have even gotten so audacious that they’ve identified specific signatures of user to snowflake proxy traffic and blocked it. Thanks to the anti-censorship team at Tor, they are hyperaware of these issues and always trying to be a step ahead of the censors.

    Where the operation stands

    So, that’s where we’ve been focusing most of our censorship evasion efforts. The Tor network has plenty of bandwidth, but it has problems with accessibility and bridges/snowflake proxies help with that. At the time of writing we’ve ramped up to 29 high-bandwidth servers around the world that run Tor snowflake proxies 24/7/365. We have 34 CPU cores and 58GB of RAM at our disposal. Some servers are in strategic locations that help users within censored countries access the proxies themselves.

    Over the past 30 days, we’ve pushed over 93TB of symmetrical traffic on our bridges & proxies.

    See our stats

    The future of Operation Envoy

    Our goal with this operation is to run as many high quality dedicated bridges and snowflake proxies as possible, and become one of the largest operators. We believe Operation Envoy is essential, as many of the snowflake proxies are run via home networks which typically do not provide high upload and download speeds.

    To scale our growing bridge and snowflake proxy server infrastructure, we use automation software called Ansible and have started writing our own Ansible role to help with that. This allows us to update and maintain our Tor bridge and proxy fleet.

    To succeed in our mission, we ask for your help via donation. With your help, we can deploy more and more censorship evasion servers around the world. In an effort to fund our operations, if you make a recurring donation of $10/mo or more after reading this post, be sure to contact us and let us know – we will deploy a Tor bridge or snowflake proxy in your name!

    We plan to release updates on our operation as it expands, so stay tuned.

    Thanks for your support,
    Zach

  • What we’re doing in response to the invasion of Ukraine

    https://blog.torproject.org/tor-and-the-humans-who-use-it/

    The situation in Ukraine is unsettling, and requires the world to step in and help on every front. Whether you do your part and help with donations, use your cybersecurity skills or attending & staging protests, anything helps.

    In response to the invasion of Ukraine by the Russian military, we have expanded our operations on the Tor network.

    What exactly does this do, and how does it help, you might ask?

    Tor is a network of virtual tunnels that allows you to improve your privacy and security on the Internet. Tor works by sending your traffic through three random servers (also known as relays) in the Tor network. The last relay in the circuit (the exit relay) then sends the traffic out onto the public Internet.

    Source: https://tb-manual.torproject.org/about/

    This makes Tor critical infrastructure to those living in oppressive countries. Without Tor, they can’t access many sites and services that provide views that their governments don’t want them to see.

    While Tor is accessible in Ukraine (despite internet outages), and not being actively blocked, it is not the same in Russia. There are many Russians who do not agree with the decision to invade Ukraine, and they have been staging protests across Russia. As such, it’s very important that Russian protestors have a way to access the uncensored internet. For many years, Roskomnadzor, a Russian agency focused on censoring and controlling the media Russian’s consume, has been cutting Russians off from websites and services they deem unpalatable. Right now, Tor is blocked in Russia, and we want to help unblock it through anti-censorship Tor bridges. Everyone should have access to an uncensored internet.

    How specifically are we helping combat this now?

    Well, since the start of the invasion we’ve deployed 5 additional Tor bridges (our focus), and 4 exit relays. Our Tor bridges were deployed in strategic locations, close to but outside of Russia, for optimal latency.

    What exactly are Tor bridges?

    Tor bridges are the first hop onto the Tor network for many users in countries enforcing internet censorship. They use obfuscation to disguise Tor traffic to and from a user, & make it look unsuspicious to would-be snoopers looking to block connections to the Tor network.

    Since the deployment of our Tor bridges, we’ve seen high usage across the board. This doesn’t surprise us, as Tor usage has been spiking in Ukraine, and bridge usage is up in Russia.

    Source

    Since the invasion of Ukraine on February 24th to now (Feb 27th), our own metrics show that we have pushed over 100 TB of symmetrical traffic to and from the Tor network via all of our Tor relays and bridges.

    Source

    How can you help?

    You might be wondering, how can I help this effort to provide uncensored internet access. Well, it’s quite simple actually, and you don’t need to be tech-savvy at all. You can install Tor’s Snowflake browser add-on which helps censored users access the Tor network.

    https://snowflake.torproject.org/

    Additionally, you can donate to Unredacted. We will use any funds during the conflict to spin up new Tor bridges and expand our Tor footprint to help those experiencing internet censorship.

    Donate here: unredacted.org/donate

    You can also see the real world impact of your funding here (although Tor bridges produce much less traffic than other relay types): https://grafana.unredacted.net/d/ce-tor-bridges/unredacted-tor-bridge-metrics?orgId=1

    We wish the best for the people of Ukraine and Russia alike.

    Найкращі побажання
    Zach

  • Unredacted, a new Tor relay operator

    Unredacted is a new and special project. We’re focused on operating resilient, automated and security focused infrastructure. We’re dedicated to giving back to the community through open source and privacy focused projects. We’ve ran projects such as XMPP.is, a free and open XMPP/Jabber server and other projects under the Crypto World label for quite some time.

    We’ve maintained and operated Tor relays in the past, however, never under our own network. As an ARIN member, recently expanding our IP space we registered an ASN, AS399532. Putting it to good use, we established a BGP session with BuyVM who provides our bandwidth and connectivity. We love stability, so inbound traffic to our network is DDoS protected by Path.

    We have 3 locations in which we operate Tor relays:

    • Las Vegas, NV
    • New York, NY
    • Roost, LU

    All of which can be seen on our status page.

    As a start, we currently run 6 Tor exit relays and plan to add more: metrics.torproject.org

    • Our relays are named after whistleblowers, activists and people who uphold the values that we hold dear to our hearts.
    • For the configuration management of these Tor relays, we use Ansible, which makes it extremely flexible and easy to maintain a consistent configuration across all of our relays.
    • Along with a collection of our own custom Ansible roles and playbooks, we use a fantastic role called ansible-relayor which allows for easy config management and offline keys for our relays.
    • For monitoring we use UptimeRobot, Prometheus and AlertManager, all of which give us different levels of metrics and insight into performance and network latency.

    We hope that you enjoy our project! We’re proud to provide netizens of the world with stable and fast access to and from the Tor network through our relays.

    Til next time 🙂

    Zach

Donate