Category: Operations

  • Operation Envoy’s 1st year anniversary

    Governments across the world continue to block & restrict access to the uncensored Internet, with many of them blocking & restricting the use of the Tor network as a result. Over a year ago, we launched Operation Envoy, an effort to help defeat those Internet censors. Operation Envoy originally helped with our vast deployment of Tor bridges & snowflake proxies, which help to pass messages (IP packets) back and forth from users and the Tor network. These messengers, or envoys as we call them, allow people to access the uncensored Internet and disguise their use of Tor from prying eyes.

    Obfuscation of the messages that our envoys carry to and from uncensored networks are incredibly important in keeping users safe. In many countries, it’s outright illegal or highly discouraged to use these technologies to bypass Internet censorship. Some people could be in real danger if their government found out that they are circumventing Internet censorship. This is morally wrong, and with governments across the world continuing to abuse their powers and limit the free flow of information, we’ll continue fighting against it.

    It’s no secret that people in countries such as Russia and Iran (& some in China) heavily depend on censorship-resistant bridge & proxy technologies according to Tor’s metrics. To help people in even more countries, and in more ways, we want to expand our vision of what Operation Envoy is.

    Tor bridge usage metrics from June to August 2024

    Redefining what an envoy is

    After we originally launched Operation Envoy, we launched FreeSocks – a service that provides free, open & uncensored Outline (Shadowsocks) proxies to people in countries experiencing a high level of Internet censorship. We also launched Unredacted Proxies, which allow people to connect to messaging services such as Signal and Telegram, without exposing the fact to their ISP or government.

    Today, we are redefining what an envoy is to us – it’s any of our services that pass messages (IP packets or TLS wrapped application layer data) back and forth between a user and the uncensored Internet. These services should all obfuscate those messages in a way where anyone monitoring a user’s Internet usage would not be able to tell what those messages might contain. In other words, they all should use an obfuscated protocol of some kind.

    Operation Envoy now includes:

    These services currently all fall under our Censorship Evasion (CE) services.

    Operation Envoy does not include:

    Operation Envoy metrics

    Operation Envoy started with 34 CPU cores and 58 GiB of RAM, deployed all over the world. We’ve since scaled the operation, and we currently have 61 CPU cores (nearly double), and 70 GiB of RAM dedicated to delivering uncensored access to the Internet (excluding our Tor exit relays). We’re working to expand that on a regular basis, and continue growing the number of envoys at our disposal.

    To collect anonymized metrics on all of ours envoys, we created a new Grafana dashboard which details the hourly bandwidth usage of all envoys combined. Over the last 30 days (at the time of writing) we pushed over 152 TiB of bandwidth across all of our envoys. That’s a lot of data!

    We need your help!

    Unredacted Inc is a 501(c)(3) non-profit organization, and we directly depend on generous donors like you to fund our operations. If you like what we do, and want to support our mission, please consider donating. We couldn’t fund Operation Envoy, and many of our services without your help.

    As a special promotion, if you donate $10 USD/mo (or more) to us on a recurring basis after reading this blog post, we’ll deploy an envoy of your choice in honor of your generosity. If you do this, please contact us afterwards and we’ll coordinate with you.

    Thank you!

  • How we accidentally broke our Tor exit relays

    Making technology work how you expect it to, and keep it working that way can be difficult at times. Changes in configuration or software updates can put services into a broken or half-broken state, and we had the latter happen to us.

    In the spirit of transparency, we are writing about the painful discovery that many of our Tor exit relays (at least 1/3) have been broken for (at the very least) weeks, and possibly months without our knowing.

    I’d like to thank the team at Tor Project for letting us know about the issue, and how to reproduce it, which ultimately led to it being discovered and fixed.

    Root cause

    Credit: https://itsfoss.community/t/its-always-dns-lol/10820

    Based on the image, you might suspect what the issue was.

    It was DNS. Specifically, it was Tailscale’s MagicDNS feature. DNS queries were not getting resolved for some reason that is unknown to us. This means that anyone who ended up connecting to our Tor exit relays failed to connect to nearly every domain/subdomain by failing to resolve hostnames. Connecting to IPs worked just fine.

    Before we go on blaming Tailscale, I want to state that we don’t know why MagicDNS failed in the way we observed, we just know that it did. Ultimately, disabling MagicDNS on our exit relays resolved the issue entirely. When we enabled it, and tested again, it failed. As a result, we’ve left it off.

    Technical analysis

    We knew the symptom was that DNS resolution seemed to fail, which was noted by the nice people at Tor.

    We started by attempting to reproduce the issue ourselves. This required pinning our Tor instance/daemon on our local computer to a specific exit relay’s fingerprint that was exhibiting this strange behavior. For example, by modifying our Tor daemon’s torrc file and adding the below line to it, we could force our local Tor daemon to exit all traffic on that relay.

    ExitNodes F34EE673122518873E717C128E35A389B72C7837

    This fingerprint corresponds to our UnredactedSnowden relay.

    We then pointed one of our browsers to use the local SOCKS proxy the Tor daemon listens on (127.0.0.1 port 9050) to send traffic through Tor.

    When attempting to connect to any website, it failed, but the reason was unclear and did not appear to display an error related to a DNS resolution issue.

    As DNS was still the suspect here, the easiest thing to do was to SSH into that exit relay and run a tcpdump to capture all inbound and outbound packets that used TCP or UDP port 53, such as the one below.

    tcpdump -i any -n port 53

    Once we did that, we discovered that nearly all DNS queries originated from Tor seemingly went out to the 100.100.100.100 MagicDNS IP, but nothing was returned on most queries. We knew at this moment, that it was indeed a DNS resolution problem.

    An anonymized example of what we saw:

    17:11:52.164204 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 40707+ A? domain.com. (45)
17:11:52.172049 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 33552+ A? domain.com. (33)
17:11:52.203409 tailscale0 In  IP 100.100.100.100.53 > 100.69.x.x.22065: 47343 NXDomain 0/1/0 (119)
17:11:52.321235 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 45366+ A? domain.com. (28)
17:11:52.321271 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 16617+ A? domain.com. (35)
17:11:52.321303 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 39612+ A? domain.com. (29)
17:11:52.352491 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 63111+ A? domain.com. (34)
17:11:52.383332 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 15513+ A? domain.com. (35)
17:11:52.501714 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 16308+ A? domain.com. (29)
17:11:52.532238 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 7697+ A? domain.com. (40)
17:11:52.540674 tailscale0 In  IP 100.100.100.100.53 > 100.69.x.x.22065: 22472 0/1/0 (89)
17:11:52.544683 tailscale0 Out IP 100.69.x.x.22065 > 100.100.100.100.53: 8052+ A? domain.com. (47)

    We tried several things;

    • Switching the MagicDNS nameservers to other ones & restarting Tor.
    • Rebooting the exit relay we were testing with to see if it was a strange Tailscale daemon/interface/routing issue.
    • Using dig via CLI on the test relay which queried the MagicDNS IP (100.100.100.100) which worked without issue.

    We were at a loss, and couldn’t figure out what was happening. We then decided to disable MagicDNS on the test relay to see what would happen. It worked, DNS queries started flowing and getting resolved responses via the same nameservers directly.

    We subsequently disabled MagicDNS on the rest of the exit relays with an adhoc Ansible shell command.

    Our conclusion

    The problem appeared to be with the abstraction that MagicDNS does, and queries originating from Tor did not appear to work 99%+ of the time when the feature was enabled. However, queries from dig via CLI appeared to always work. We suspect that MagicDNS fails in some sort of way when too many queries are directed at its 100.100.100.100 IP which is seemingly routed out the tailscale0 interface (& subsequently onto the physical interface). However, this doesn’t make complete sense, as we would expect queries from dig to fail as well.

    We may never know what happened exactly, and we don’t want to leave it in a broken state long enough to figure it out. At this point, it’s safe to say that we are leaving MagicDNS disabled on our Tor exit relays for the foreseeable future.

    Shortly after resolving the issue, our Tor exit relay traffic rate shot up beyond previously normal levels and hit our full capacity (as of writing this).

    In the near future, we will explore running our own local DNS resolver on each exit relay, which we’ve done in the past – but had to move away from due to an overload of bogus queries originated from Tor which also resulted in DNS resolution failures. DNS over HTTPS (DoH) or DNS over TLS (DoT) are also great options we may explore further.

    We hope you found this interesting and insightful. If you enjoy what we do, please consider making a donation. Unredacted is a non-profit organization that provides free and open services that help people evade censorship and protect their right to privacy.

  • New Tor bridge types for Operation Envoy

    In July of last year (2023) we launched Operation Envoy, our effort to deliver packets to and from the Tor network which helps defeat Internet censorship. This is achieved by Unredacted operating Tor bridges, also known as Pluggable Transports. Tor bridges obfuscate (bridge) the connection a user makes when connecting to Tor so that it looks like any normal connection and disguises the fact that they are connecting to the Tor network. Each Pluggable Transport has its own unique way of obfuscating the connection, such as WebTunnel which mimics HTTPS traffic, one of the most common types of traffic on the Internet.

    What a connection to the Tor network looks like with a bridge in the path
    Credit: robertheaton.com/2019/04/06/how-does-tor-work/

    Historically, and for a long time we’ve focused our efforts on deploying dedicated snowflake proxies around the world in strategic locations close to Internet users that face a high level of Internet censorship in their countries. Today, we’ve added a WebTunnel and meek bridge into the mix. Adding more Tor bridge types means that users have more ways to connect to the Tor network in the event that one protocol / obfuscation technique gets blocked.

    Our meek bridge

    How meek works, click the image to learn more

    To deploy our meek bridge, we worked with the team at Tor after volunteering to run a new bridge. Due to how meek works with Tor, there is some setup on their end as well because they use a technique called domain fronting. This is a technique to disguise a connection and route it through popular, and more painful to block CDN networks like Microsoft Azure. Meek bridges remain a crucial method to connect to the Tor network in several countries.

    To see our new meek bridge statistics, you can click here.

    Our WebTunnel bridge

    How HTTPT works, the proxy behind WebTunnel technology

    As described earlier in the post, WebTunnel is a bridge type which mimics HTTPS traffic, one of the most common types of traffic on the Internet. It’s based on HTTPT which resists active probing attacks that censors use to block censorship circumvention techniques. WebTunnel will likely, and ultimately become a very important bridge type for Tor as it rolls out and gains popularity due to the protocol it disguises itself as and its resistance to active probing.

    Our new WebTunnel bridge uses a unique configuration that we came up with to hide the IP of the bridge behind a TCP proxy service. This allows us to easily switch the ‘front’ of the WebTunnel bridge in case its IP gets blocked. In the future, we plan to write about how we did this once we’ve confirmed its stability over time.

    To see our new WebTunnel bridge statistics, you can click here.

    Current Operation Envoy stats

    As it stands today, we have a collective of virtual machines consisting of 31 CPU cores, 40GB of RAM and multi-gigabit unmetered links dedicated to serving Tor bridge traffic across the world.

    Past 7 days of CPU and memory usage, click the image to see live stats

    On an average day, we are pushing almost 2TB of symmetrical bandwidth per day. That’s almost 60TB per month!

    Past 7 days of bandwidth usage, click the image to see live stats

    We can’t make all of this possible without your help. If you like what we do, please consider making a donation. As time goes on, and with more funding we’ll continue to expand our Operation Envoy footprint by deploying more Tor bridges across the world. Your help can make a real impact for Internet censorship circumvention.

  • Operation Envoy: Defeating Censors

    Operation background

    Accessing the uncensored Internet in some countries has never been so difficult. Internet censorship is rising across the world, and content filtering is becoming more difficult to circumvent as technology and censors evolve. Even in countries you wouldn’t expect. However the worst offenders are the ones you would typically suspect, China, Russia and countries who rank low on the World Press Freedom Index.

    The organization, OONI (Open Observatory of Network Interference) monitors internet censorship around the world and produces reports which show that censorship is on the rise. Government censors (governments who implement Internet censorship) are insatiable in their quest to restrict Internet access and keep their citizenry blind and oppressed, just how they like it.


    The question is, what are we doing about it? That’s where Operation Envoy comes in. We want to help deliver messages (network packets) to and from the Tor network. For quite a while now, we’ve been running Tor exit relays which provide valuable bandwidth and processing power to the Tor network which helps people in heavily censored countries access services and information that people in the western world take for granted. While exit relays are an integral part of the Tor network, there’s another part that is critical for accessing it in many countries. Tor bridges and snowflake proxies are the first entry point into the Tor network for many people. What are they you might be wondering? Well, many countries block access to Tor and they’re very good at it, which makes Tor hard to access. That’s where Tor bridges and snowflake proxies step in, and so do we. Bridges and snowflake proxies allow Tor users to access the network via an obfuscated and seemingly normal-looking connection to the bridge or proxy. That bridge or proxy then acts as a literal bridge to the Tor network.

    Censors have even gotten so audacious that they’ve identified specific signatures of user to snowflake proxy traffic and blocked it. Thanks to the anti-censorship team at Tor, they are hyperaware of these issues and always trying to be a step ahead of the censors.

    Where the operation stands

    So, that’s where we’ve been focusing most of our censorship evasion efforts. The Tor network has plenty of bandwidth, but it has problems with accessibility and bridges/snowflake proxies help with that. At the time of writing we’ve ramped up to 29 high-bandwidth servers around the world that run Tor snowflake proxies 24/7/365. We have 34 CPU cores and 58GB of RAM at our disposal. Some servers are in strategic locations that help users within censored countries access the proxies themselves.

    Over the past 30 days, we’ve pushed over 93TB of symmetrical traffic on our bridges & proxies.

    See our stats

    The future of Operation Envoy

    Our goal with this operation is to run as many high quality dedicated bridges and snowflake proxies as possible, and become one of the largest operators. We believe Operation Envoy is essential, as many of the snowflake proxies are run via home networks which typically do not provide high upload and download speeds.

    To scale our growing bridge and snowflake proxy server infrastructure, we use automation software called Ansible and have started writing our own Ansible role to help with that. This allows us to update and maintain our Tor bridge and proxy fleet.

    To succeed in our mission, we ask for your help via donation. With your help, we can deploy more and more censorship evasion servers around the world. In an effort to fund our operations, if you make a recurring donation of $10/mo or more after reading this post, be sure to contact us and let us know – we will deploy a Tor bridge or snowflake proxy in your name!

    We plan to release updates on our operation as it expands, so stay tuned.

    Thanks for your support,
    Zach

Donate