How to Detect a Government’s Hand Behind Internet Shutdowns

Internet shutdowns that affect entire regions or countries and cost billions of dollars annually have become a widespread phenomenon, especially as various governments wield them like a blunt club to restrict citizens’ access to online information.

Some governments deploy Internet shutdowns in an attempt to suppress protests, while Iraq’s Ministry of Education even orders shutdowns to prevent cheating during national school exams. The trick for independent observers trying to keep track of it all involves figuring out the difference between government-ordered shutdowns versus other causes of Internet outages.

In early 2020, the five-person team behind the nongovernmental organization NetBlocks was watching dips in Internet connectivity happening in a particular region of China over several months. That could have sparked suspicion that China’s online censors—who restrict access to certain online content as part of China’s “Great Firewall”—were perhaps throttling some popular online services or social media networks. But the NetBlocks team’s analysis showed that such patterns likely had to do with businesses shutting down or limiting operations to comply with government efforts aimed at containing the coronavirus outbreak that has since become a pandemic.

“When you’re investigating an internet shutdown, you need to work from both ends to conclusively verify that incident has happened, and to understand why it’s happened,” says Alp Toker, executive director of NetBlocks. “This means ruling out different types of outages.”

NetBlocks is among the independent research groups trying to keep an eye on the growing prevalence of Internet shutdowns. Since it formed in 2016, the London-based NetBlocks has expanded its focus from Turkey and the Middle East to other parts of the world by using remote measurement techniques. These include analytics software that monitors how well millions of phones and other devices can access certain online websites and services, along with both hardware probes plugged into local routers and an Internet browser probe that anyone can use to check their local connectivity.

But NetBlocks also relies upon what Toker describes as a more hands-on investigation to manually check out various incidents. That could mean checking in with local engineers or Internet service providers who are in a position to help confirm or rule out certain lines of inquiry. This combined approach has helped NetBlocks investigate all sorts of causes of Internet shutdowns, including major hurricanes, nationwide power outages in Venezuela and cuts in undersea Internet cables affecting Africa and the Middle East. Each of these types of outages provides data that NetBlocks is using to train machine learning algorithms in hopes of better automating detection and analysis of different events.

“Each of the groups that’s currently monitoring Internet censorship uses a different technical approach and can observe different aspects of what’s happening,” says Zachary Weinberg, a postdoctoral researcher at the University of Massachusetts Amherst and member of the Information Controls Lab (ICLab) project. “We’re working with them on combining all of our data sets to get a more complete picture.”

ICLab relies heavily on a network of commercial virtual private networks (VPNs) to gain observation points that provide a window into Internet connectivity in each country, along with a handful of human volunteers based around the world. These VPN observation points can do bandwidth-intensive tests and collect lots of data on network traffic without endangering volunteers in certain countries. But one limitation of this approach is that VPN locations in commercial data centers are sometimes not subject to the same Internet censorship affecting residential networks and mobile networks.

If a check turns up possible evidence of a network shutdown, ICLab’s internal monitoring alerts the team. The researchers use manual confirmation checks to make sure it’s a government-ordered shutdown action and not something like a VPN service malfunction. “We have some ad-hoc rules in our code to try to distinguish these possibilities, and plans to dig into the data [collected] so far and come up with something more principled,” Weinberg says.

The Open Observatory of Network Interference (OONI) takes a more decentralized, human-reliant approach to measuring Internet censorship and outages. OONI’s six-person team has developed and refined a computer software tool called OONI probe that people can download and run to can check local Internet connectivity with a number of websites, including a global test list of internationally relevant websites (such as Facebook) and a country-specific test list.

The OONI project began when members of the Tor Project, the nonprofit organization that oversees the Tor network designed to enable people to use the Internet anonymously, began creating “ad hoc scripts” to investigate blocking of Tor software and other examples of Internet censorship, says Arturo Filasto, lead developer of OONI. Since 2012, that has evolved into the free and open-source OONI probe with an openly-documented methodology explaining how it measures Internet censorship, along with a frequently updated database that anyone can search.

“We eventually consolidated [that] into the software that now tens of thousands of people run all over the world to collect their own evidence of Internet censorship and contribute to this growing pool of open data that anybody can use to research and investigate various forms of information controls on the Internet,” Filasto says.

Beyond the tens of thousands of active monthly users, hundreds of thousands of people have downloaded the OONI probe. That probe is currently available as a mobile app and for desktop Linux and macOS users who don’t mind using the command-line interface, but the team aims to launch a more user-friendly desktop program for Windows and macOS users in April 2020.

Other groups have their own approaches. The CensoredPlanet lab at the University of Michigan uses echo servers that exist primarily to bounce messages back to senders as observation points. The Center for Applied Internet Data Analysis (CAIDA) at the University of California in San Diego monitors global online traffic involving the Border Gateway Protocol, which backbone routers use to communicate with each other.

On the low-tech side, news articles and word-of-mouth reports from ordinary people can also provide valuable internet outage data for websites such as the Internet Shutdown Tracker run by the Software Freedom Law Centre in New Delhi, India. But the Internet Shutdown Tracker website also invites mobile users to download and install the OONI probe tool as a way of helping gather more data on regional and city-level Internet shutdowns ordered by India’s government.

Whatever their approach, most of the groups tracking Internet shutdowns and online censorship still consist of small teams with budget constraints. For example, ICLab’s team would like to speed up and automate much of their process, but their budget is reliant in large part upon getting grants from the U.S. National Science Foundation. They also have limited data storage that restricts them to checking each country about two or three times a week on average to collect detailed cycles of measurements—amounting to about 500 megabytes of raw data per country.

Another challenge comes on the data collection side. People may face personal risk in downloading and using OONI probe or similar tools in some countries, especially if the government’s laws regard such actions as illegal or even akin to espionage. This is why the OONI team openly warns about the risk up front as part of what they consider their informed consent process, and even require mobile users to complete a quiz before starting to use the OONI probe app.

“Thanks to the fact that many people are running OONI probe in China and Iran, we’ve been able to uncover a lot of really interesting and important cases of Internet censorship that we wouldn’t otherwise have known,” Filasto says. “So we are very grateful to the brave users of OONI probe that have gathered these important measurements.”

Recent trends in both government information control strategies and the broader Internet landscape may also complicate the work of such groups. Governments in countries such as China, Russia, and Iran have begun moving away from network-level censorship toward embedding censorship policies within large social media platforms and chat systems such as Tencent’s WeChat in China. Detecting more subtle censorship within these platforms represents an even bigger challenge than collecting evidence of a region-wide Internet shutdown.

“We have to create accounts on all these systems, which in some cases requires proof of physical-space identity, and then we have to automate access to them, which the platforms intentionally make as difficult as possible,” Weinberg says. “And then we have to figure out whether someone’s post isn’t showing up because of censorship, or because the ‘algorithm’ decided our test account wouldn’t be interested in it.”

In 2019, large-scale Internet shutdowns affecting entire countries occurred alongside the shift toward “more nuanced Internet disruptions that happen on different layers,” Toker says. The NetBlocks team is refining its analytical capability to home in on different types of outages by learning more about the daily pattern of Internet traffic that reflects each country’s normal economic activities. But Toker is also hoping that his group and others can continue forging international cooperation to study these issues together. For now, NetBlocks relies upon community contributions, the technical community, and volunteers.

“There are bubbles of expertise in different parts of the world, and those haven’t necessarily combined, so from where we’ve been coming I think those bridges are just starting to be built,” Toker says. “And that means really getting engineers together from different fields and different backgrounds, whether it’s electrical engineering or Internet engineering.”

Editor’s Note: This story was updated on 20 March 2020 to reflect the fact that OONI probe has been downloaded hundreds of thousands of times and not hundreds of millions of times.

A correction to this article was made on 23 July 2020.