Anatomy of an Internet Hijacking
Did China Telecom really hijack 15 percent of the Internet's traffic in April?
Steven Cherry: Hi, this is Steven Cherry for IEEE Spectrum's This Week in Technology.
On November 17, the Web site TechNewsWorld broke some wild-sounding news. Here's a snippet: for a period of 18 minutes last April, China Telecom hijacked 15 percent of the world's Web traffic and sent it to servers in China, an accusation the state-run organization has denied. Whether the apparent reroute was intentional or accidental, it's exposed another weakness in the structure of the Web. Except they didn't exactly break the news. Back on the day in question, April 8, the company BGPmon, which makes an Internet monitoring and analysis tool, reported on their blog that one of the Data Centers operated by China Telecom normally only originates about 40 prefixes, however today for about 15 minutes they originated about 37 000 unique prefixes that are not assigned to them. This is what we typically call a prefix hijack. That's a lot of complicated stuff to understand. Fortunately, there are people who understand it. One of them is James Cowie. The day after the TechNewsWorld story, Cowie posted a remarkably readable 1700-word analysis explaining BGP, Internet routing, and prefix hijacking. Here's how his report began: "When the US-China Economic and Security Review Commission released its report to Congress this week, something slightly unusual happened: People read it." And there, buried on pages 236 to 247, a mystery was revealed, and the media have greedily amplified it. Did China's government really divert 15 percent of the Internet's traffic for 18 minutes in April, effortlessly intercepting sensitive traffic in flight, and generally creating a massively embarrassing man-in-the-middle attack on vulnerable global communications? Well, yes and no. Mostly no. My guest today is James Cowie. He's the chief technology officer and cofounder of Renesys, a network management company headquartered in Manchester, N.H. He's been working on things like Internet security, Web services, and network collaboration, for 15 years. Jim, welcome to the podcast.
James Cowie: thanks good to talk with you.
Steven Cherry: Jim should we start with the yes and then move on to the mostly no?
James Cowie: I guess that's a good way to handle it. So yes, a hijacking did take place. I think BGPmon listed some 30-something thousand. We saw more than 50 000. It really depends on where you look. The mechanisms are pretty straightforward. As you know from the blog, BGP routing is based on the honor system, something that most people just can't believe. It really comes down to asserting that you own networks, in this case someone who owned only about 40 asserted that they owned more than 50 000. And because of the way that network assertions propagate through the Internet, they were believed by a lot of people. Now they weren't really believed by enough people to take 15 percent of the world's traffic, but they did transiently affect about 15 percent of the world's routes. So maybe that's an important distinction.
So that gives you the "yes". In essence the routes were hijacked. If you were close enough on the network to one of the providers that was asserting these false ownerships, and you tried to talk to one of those networks, your traffic, your packets would flow to these sites that incorrectly asserted ownership.
Steven Cherry: Just to be clear, this was just one little corner of China Telecom's network, which is pretty massive, and it normally owned 40 prefixes?
James Cowie: That's right. That's about right. The way that Internet organizations represent themselves in routing is, any organization that sort of speaks BGP (the routing protocol) to each other across organization boundaries will have its' own autonomous system number kind of like a phone number for your organization. And these are numbers typically between 1 and 64 000 or so although now you can get them larger. But what happened is China Telecom is instantly recognizable to all Internet engineers as 4134. And in this case the leak didn't come from 4134 it came from 23 724 which all of us had to go look up. It's not a familiar number. It turns out to be a small part of China Telecom. I think it's registered as IDC China Telecommunications Corp., clearly in Beijing, looking at the routing you see that they received their Internet transit from China Telecom 4134. And so the problem here is that you have a small organization asserting this wild fantasy of ownership, and their parent 4134 picked up the ball and ran with it. And I guess people tend to trust 4134 in the sense that 4134 China Telecom usually says a lot of things like this. They have lots of networks, thousands and thousands, and so a lot of people won't filter what they hear from 4134. And it you don't filter, which is to say you doubt, you check each one of their assertions against known truths. If you don't filter you accept it, and if you accept it you'll propagate it, and if you propagate it then you're just making the problem worse. You just spread the rumor, and that's what happened here. This little tiny organization inside or underneath China Telecom made its fantasy statement, it was amplified by China Telecom, and then within just a minute or two most of the world believed it.
But that's the wrinkle, most of the world didn't actually believe it. Most of the world heard it and said, "Well, I've still got a closer route." If two organizations in the United States for example, maybe they both got hijacked and they were trying to talk to each other, they might not have even noticed anything was happening. That's because two organizations in the United States are almost always closer to each other than they are to China Telecom, and so they wouldn't necessarily have picked the suspicious route. They may have stuck with their perfectly good realistic route. And that's why it's important to say 15 percent of the world's routes were hijacked but nowhere near 15 percent of the traffic was hijacked.
Steven Cherry: I guess that's part of the "mostly no" and another part is this sort of thing happens every so often and it has for a decade or more. Is that right?
James Cowie: It has, I think. It's happened basically since the dawn of Internet time. Because it's based on the honor system, BGP is designed to make it possible to get around damage. It's based on the idea that if you have multiple ways to get to the Internet, and you lose one of them, it's just fine everybody else will pick the next best way to reach you. And so BGP and the Internet at large is based on having lots of ways to get there, and so in some sense it's set up for cases like this where "Oh look, I've got another way to get there." And the protocols are kind of designed that you will eagerly accept and consider, maybe as a back up, a new way to get to a place. And there really isn't a lot of trust mechanism built in to allow you to be discerning about this, especially if you hear something third hand—"Somebody told me that somebody told them"—you might just believe that. The resolution to this can only come when we come up with secure mechanisms for distributing these assertions. And there are efforts in place, of course, to allow people to cryptographically sign their assertions, to prove that "Yes, I'm the one who's allowed to assert this," but it's nowhere near deployment today. It would require an Internet flag day if you will, which basically pushes the likelihood of that happening off several years or an indefinite amount of time sadly.
Steven Cherry: If I understand your analysis, you came to believe that this April 8 hijacking was just an accidental error on the part of some network engineer probably.
James Cowie: That was pretty much my take on it. You know we've thought about these things for quite a while. We thought about, Could you use BGP route announcements, the deliberate falsehoods as an attack on somebody. And there's been quite a bit of thought about this and hand-wringing and attack analysis. Could you hijack somebody's traffic and take them off the Internet? Could you hijack somebody's traffic to yourself and then send it back to them, so you could man-in-the-middle attack them? It's also plausible. The difficult part here I guess is that BGP attacks are really visible. It's not like breaking into somebody's apartment secretly and stealing their money. It's more like breaking into their car while they're stopped at a stoplight in the middle of Times Square. It's very, very visible. By the nature of BGP to get your lie. To get your fantasy route propagated throughout the world, you have to tell everybody and it's pretty clear that it's you who did it. So it's a very blunt instrument if you wanted to use it that way.
Steven Cherry: You know you published some log data in your analysis and so it strikes me that it's also a little bit like breaking into somebody's house while having a bunch of wet paint on your shoes or something.
James Cowie: Yeah, exactly. You leave your tracks everywhere, because for the duration of the attack, the fact that you've changed the control plane of the Internet is out there for everybody to view. And there's a data trail left behind that shows when the routes changed, why the changed, who said that in the first place, where did it come from. You know we can pinpoint these things down to the second because we've got all of the BGP traffic sort of saved up. So it's not a very mysterious way to do these things, which is why you can't rule out the possibility that somebody did it intentionally although as I said it would be a very blunt instrument attack. Maybe a statement. But the far more likely scenario here, which is the sort of scenario that dominates the way that the Internet has historically worked, is that somebody messed up. Somebody fat-fingered something in a config, somebody mis-configured a router, somebody propagated a bad policy and probably got 10 minutes in and said, "Wait a minute. I've done something terribly wrong" and backed it out. And so nobody knows why those particular 50 000 networks are the ones that suffered this little injustice. There doesn't seem to be any particular pattern to them. There's really no rhyme or reason. It's not as if they're all United States networks or anything. I mean they came from all sorts of countries. They include large numbers of Chinese networks. If you were going to do this to someone deliberately, why would you hijack your own country's networks? I really think it's a pretty scattershot kind of impact and because there's so little focus to it, it just sort of smelled to us as if it must be a random table selection of some sort.
Steven Cherry: I'm curious though if it had been a deliberate hijacking of data, what are the possible consequences? I mean would somebody be able to actually read all of the data and understand it? How do those packets work?
James Cowie: When you send information over the Internet you're using TCP, right? You're establishing a conversation with somebody on the other side and you go back and forth. You ask for data; the data comes back. Every so often you pause and acknowledge your receipt of the data so it kind of flows in a conversation. The difficult thing here is that if you immediately hijack traffic the packets that are flying across the network will go to the hijacker lets say. But the hijacker doesn't know how to respond. He can't break into the conversation. It's sort of like going to a cocktail party full of quantum physicists, right? And you don't know quantum physics and you just sort of jump in and elbow somebody out of the way in front of his date, and start conversing about quantum physics. You're going to get found out very quickly. The conversation will be very short. And so it is with TCP sessions that get hijacked. You just can't emulate the state of the people that have been hijacked well enough, fast enough, to make believe that you're them, and keep up the conversation, and acknowledge the packets, and have the right sequence numbers and continue to receive the information. And so if they were sending you an e-mail you're not going to get it all. If they were sending you the contents of a Web page, you're not going to get the Web page, you're going to get what's in flight, and it's going to kind of stutter. Then it's going to kind of break down.
Steven Cherry: So this really isn't a good way of spying on somebody else's data.
James Cowie: Well, no. Generally not. Now there's one potential cause for concern here. Remember I said that if you simply steal one end of a conversation it doesn't function effectively because you can't emulate the state of the other guy. What if you stepped into a conversation as an intermediary? What if you were a man in the middle is the term of art? So I hijack your location, your friends all send me their traffic thinking I'm you. I take a look at it real quick, maybe I change a few things and I put it right back on the Internet and the packets continue flying to you. And the conversation might work perfectly well in that case. You might go for 10 minutes, an hour, a day, never realizing that somebody on the other side of the planet is intercepting every packet in every conversation. In practice that's really unlikely, because it would introduce huge delays first of all. When we did our trace route analysis in our blog, we saw delays of 400, 500 milliseconds (which in Internet terms is pretty glacial) in every round trip term for a packet that was going through China and back to its destination. So that seems like the sort of situation that could come about again by accident, but it is also potentially the most worrying scenario for data security in terms of a BGP hijack.
Steven Cherry: But network engineers aren't going to allow half a second delays to persist for weeks or even hours, right?
James Cowie: No, you would hope that somewhere in your service provider's data center alarm bells would be going off saying, "Why are packets taking so long? Why has throughput dropped so precipitously?" And anyone that does a trace route let alone looking at the routing tables will pretty quickly see the situation, so again it's not a great covert attack. There are techniques that can be used to hide the BGP hijack from the victim alone, but again everyone else around the planet will see it. You can't hide it from the planet.
Steven Cherry: So it sounds like the network can be kind of broken in this hijacking way, but really not for long. There are just too many mechanisms in place in the net and the way its managed to really allow it to persist for any length of time.
James Cowie: I think that's right. And there are a lot of people who are sort of keeping an eye out. People from all the large service providers, of course. Renesys is an example of a research organization and security company that keeps an eye on this sort of stuff for the rest of the Internet. It's the sort of thing where there are a lot of eyes watching.
Steven Cherry: So just to sum up: This attack affected a lot less data than people thought at first. It was almost certainly an accident and not deliberate. This sort of attack would be a very bad way to get information that you weren't entitled to, and the network is going to fix this problem pretty quickly even if it did happen deliberately. And finally, if you did this you would certainly be caught doing it.
James Cowie: I think that's a very good summary. It doesn't mean that people can be sanguine about it. You could think of it, I hesitate to say this, but you could think of it geopolitically as maybe a signal that such things are possible. It's much harder obviously to catch and diagnose a one-prefix, a one-network, hijack than it is to catch a 50 000-network hijack, which is like an explosion in a city. If a single network, maybe the network belonging to your service provider, that has your house inside it, gets hijacked, who's going to notice? That's the sort of thing that we look out for.
Steven Cherry: Well, thanks a lot, Jim.
James Cowie: Sure it's my pleasure.
Steven Cherry: We've been speaking with Jim Cowie cofounder of network management company Renesys about an April 8 hijacking of 15 percent of the Internet's routes, how it happened, and whether it's really a big deal.
For IEEE Spectrum's This Week In Technology, I'm Steven Cherry.
NOTE: Transcripts are created for the convenience of our readers and listeners and may not perfectly match their associated interviews and narratives. The authoritative record of IEEE Spectrum's audio programming is the audio version.