How New York City's oldest Internet service provider was hijacked and rescued
2 February 2005--It's 4:30 a.m. Do you know where your Internet address is? At about that time on 15 January, Alexis Rosen, owner of Public Access Networks Corp., New York City's oldest Internet service provider (ISP), and its flagship domain panix.com, certainly didn't. It was at that early hour that he first discovered that his company's Internet address and its entire business had been stolen in the night. Rosen's first thoughts were unprintable.
Panix.com's disappearance, while not the first cyberattack of its kind, was unusual, and different from the virus or denial-of-service attacks that have hit companies like eBay, Amazon, and Microsoft in years past. Those succeeded by overwhelming servers with data. In this case, Panix had complete control of its servers and systems. Instead, the domain name itself, panix.com, became dissociated from the four-part numerical address that is the actual means by which packets of data make their way to it on the Internet.
As a consequence, e-mail intended for Panix subscribers wound up in a random server in Canada. Panix officials believe none of the e-mail was opened, but have said they "cannot be absolutely sure" of that. Besides e-mail, there was the matter of the Internet domains that Panix hosts for its business customers, and the personal domains of its individual users. Sites with URLs like http://www.panix.com/~steven were unreachable, because their routing depends on the correct operation of the panix.com domain name.
Why would someone hijack a domain? A favored theory among Panix's customers is that the perpetrator was an unhappy former subscriber. "Panix has pissed off a lot of people over the years," was a common theme on a hyperactive private Panix message board called "panix.questions." With more pride than chagrin, Rosen agrees. "We've been around a long time, and we're pretty vocal about the way we think things should work," he says. "We've made our share of enemies--spammers, blackhat hackers, you name it."
Exactly when the hijacking began isn't known because such a change takes hours to propagate through the domain name system, or DNS. The DNS is a hierarchical structure of servers that each contain some domain name records, and know what other servers have additional relevant records. (See "Striking at the Internet's Heart," a December 2001 article in IEEE Spectrum, for a detailed explanation of how the DNS operates.) It takes a day for a change to be fully reflected in the domain name system, because servers routinely cache individual domain name records, such as Panix's for 24 hours.
It took about 36 hours of frantic work by a globe-spanning group of Internet specialists to finally regain control of the Panix domains. However, the e-mail misdeliveries were stopped well before that, through the heroic efforts of a lone Canadian network engineer, acting on his own authority. The person or persons responsible for the attack remain unknown and at large, and the success of the scheme has left ISPs on edge. Extensive interviews by Spectrum with key people involved showed the attack on Panix was less of a technological feat than an exploitation of human fallibility, and it was very human efforts that in the end rescued the stricken ISP.
Rosen Learned of the Hijacking when he was awakened by his pager. It was Panix's systems administrator. "When I turned on the computer, I saw we had a serious problem," he says. He realized that the hijacking threatened not only his customers' data but Panix's reputation--in the ethereal world of the Internet, nothing is more important--and therefore the business itself.
Rosen immediately began "reaching out to Panix's many customers and friends." And among them, Rosen says, are "certain relatively high-ranking people in law enforcement, who reached out to me," to see if they could be of any help. He also contacted fellow administrators on a small semi-private mailing list for key U.S. network operators.
The immediate need was to re-associate the various panix.com services, such as e-mail and Web pages, with Panix's block of Internet Protocol addresses. Those numerical addresses could in some cases be used directly. For example, the Web address http:// 18.104.22.168 would still take you to Panix's home page. But IP addresses are rarely used directly in that way, and if you typed "http://www.panix.com" last Saturday morning, your Web browser would have displayed a default page at a server, www.freeparking.co.uk, which is, despite its British suffix, located in Canada.
The hijacking exploited the fairly recent relaxation of some rules governing Web site ownership and a confusion among the entities that associate Internet domain names with IP addresses. These registrars--there are hundreds of them around the world--are authorized to make changes in a database of Internet names and numbers known as a registry. Confusingly, the owner of the database is also called a registry. In the case of Internet names that end with ".com," the registry is VeriSign Inc., of Mountain View, Calif.
Anyone can buy a domain name if it isn't already taken. An individual registration record--the record that matches the name to an IP address--can be modified only by the registrar listed in the record. Panix's registration is, or was, until Saturday 15 January, held by Dotster Inc., a Vancouver, Wash.-based registrar. Somehow, and for reasons that were still unknown a week after the attack, the registration was moved to Melbourne IT Ltd., in Melbourne, Australia.
The change in Panix's registration was made by a company known as Fibranet Services Ltd., which resells Melbourne IT's domain name registration service. Fibranet, which is officially registered in Douglas, on the Isle of Man, in the Irish Sea, operates the freeparking.co.uk site that showed up on Saturday morning instead of Panix's home page.
Someone--we still don't know who--used a stolen credit card account to sign up as a Fibranet customer. Claiming to be the rightful owner of the panix.com domain name, this party initiated a transfer of Panix's registration from Dotster to Melbourne IT. There are a number of reasons a domain name owner might legitimately initiate such a transfer. One registrar can be cheaper than another, or offer better customer support. "We've done hundreds of thousands of transfers," says Bruce Tonkin, Melbourne IT's chief technology officer. "This was the first genuine hijacking."
According to Bruce Tonkin, Melbourne IT has "a couple of hundred" resellers. A handful, "fewer than ten," he says, have agreements with Melbourne IT whereby they are responsible for obtaining the registrant's (Panix in this case) authorization, before initiating a transfer to Melbourne IT. In the case of its other resellers, Melbourne IT itself performs the authorization check.
"The reseller should look up the existing WHOIS record published by the original registrar," Tonkin says. Whois.net is an Internet-wide database whose records have complete contact information for the registrant, including names, street addresses, and an e-mail address. "The reseller is supposed to send a standardized e-mail to the authorized contact for the domain name saying, in effect, 'We've received a request for a transfer. Did you initiate it?' They're not supposed to make the transfer without this step."
According to Tonkin, Fibranet didn't take that step. "They sign a legal agreement that they will follow our procedures, and we audit them, to make sure they comply. As a result of this incident Melbourne IT is carrying out an immediate audit of all its resellers that authenticate transfers, and will implement improvements to its regular audit process."
If Fibranet enjoys privileges shared by few Melbourne IT resellers, it's an odd choice. Its ethereal structure hardly inspires confidence. The company is incorporated on the Isle of Man, a tiny island tax haven within the British Isles, but provides pay-per-minute customer telephone support from a remote location in Spain, says Richard Cox, an investigator for the London-based Internet volunteer organization, Spamhaus Project, Ltd. He notes that Fibranet's servers are in Canada, and some of its domains are owned by a company incorporated in Wilmington, Del. "So many entities on the Net nowadays feel they don't need a physical presence," Cox says, with some annoyance.
"Normally, where there is a dispute with respect to a transfer, the DNS information has not been changed," says Melbourne IT's Tonkin. That is, in the usual case, even if a registration record were erroneously moved from one registrar to another, the domain name would point to the same IP address, and the registrant's Internet services would function just as they did before. But when the stolen-credit-card-wielding hijacker initiated the transfer of Panix's registration to Fibranet, he or she changed the association between the domain name and the IP address. And other fields in the registration record were altered as well, including the names and affiliations of the individuals responsible for the domain.
This all presented a nearly insuperable problem for Panix. The company with which it has a business relationship, Dotster, no longer had any control over the registration record. The company now in control of the record had no knowledge of how it came to be in control, nor could it be sure who should have control. It didn't know Panix, from the man in the moon.
Since the transfer of Panix's registration was made erroneously--probably even illegally--and bypassed normal procedures, there were few log file entries that could help sort the situation out. After all, when something doesn't happen, there usually isn't an explicit record of it. The only other party that could have corrected the erroneous registration record was the registry--VeriSign. And yet, in the absence of any documentation that said who should own panix.com, and what IP address should be associated with it, VeriSign says it was helpless to act.
Further complicating Rosen's plight was the timing of the attack, and the time difference between New York and Melbourne--16 hours. Early Saturday morning in the United States is late Saturday night in Australia. While Melbourne IT has 24-hour phone support on weekdays, its support office is closed from Saturday afternoon to Monday morning--precisely when Rosen desperately needed to reach someone.
Meanwhile, phone lines and e-mail inboxes were abuzz , as various network administrators and other movers and shakers on the Internet frantically tried to connect Panix with people who might be able to help. For example, investigator Cox, of Spamhaus, helped put Rosen in touch with law enforcement people in the United Kingdom. "We have a great deal of experience dealing with hijackers," Cox says, "and we make that experience available, so that the right people can get in touch with one another."
Steven M. Bellovin, a former AT&T security researcher and the author of a definitive book on firewalls, was the first to alert the North American Network Operators Group e-mail list of the hijacking. The post from Bellovin, now a professor at Columbia University, in New York City, generated some invaluable connections. "It was a tremendous resource," says Rosen. Hearing about Panix's plight, many of the most important name servers on the Internet made manual changes, and so had the correct records with two or three hours, he says.
Steve Crocker, an independent computer consultant who has worked on Internet protocols dating back to the original Arpanet, delivered up a crucial connection, the cellphone number for Bruce Tonkin, chief technology officer at Melbourne IT.
Melbourne IT was in control of Panix's registration record, so it was in the best position to end Panix's pain by changing the record to reassociate the "panix.com" with Panix's IP address. But before Bruce Tonkin could be contacted, Rosen and his staff had found an obscure page on an even more obscure Web site that listed the cellphone number of Melbourne IT's chief executive officer. Unfortunately, by the time the Panix team found this number it was midmorning in New York but 2 a.m. in Melbourne.
"I waited until later," Rosen says. "As bad as things were, it was still better to wait for a more human hour to contact them." Rosen, still somewhat bitter days later, adds that "in retrospect they weren't deserving of that respect, but neither would it have mattered. I called the CEO [Theo Hnarakis] at around 8 a.m. their time. He seemed to be in his car, driving with his daughter. He said there was nothing he could do, that he wasn't even near a computer but would pass it on to his senior counsel, Moana Weir. Since diplomacy is an essential part of saving your own life, I let it go."
According to Tonkin, although Melbourne IT's customer support group is unavailable most of the weekend, it does have emergency staff. "We're closed to the public from Saturday 12:30 p.m. until Monday at 8:30 a.m., but we have customer service and computing people on call, and our customers have numbers for them. The numbers aren't advertised, but our resellers, like Fibranet, had them. In this case, though, it wasn't a customer--Panix wasn't a customer of ours," Tonkin says.
"Moreover," Tonkin says, "What our CEO heard was, 'Our domain's been hijacked.' Well, the fact is, we hear that all the time." Often, he says, it's just a misunderstanding among staff at the same company. "In hundreds of thousands of transfers, this was the first genuine hijacking." And, as he says, normally a putative hijacking doesn't actually change one's Internet service. "In this case," Tonkin says. "The person seems to have altered the records maliciously."
By Saturday afternoon, New York time, Panix's staff had come up with a workaround for its customers. It made another domain, panix.net, which it also owns, a virtual synonym for the panix.com name. The URL "http://www.panix.net" took viewers to the Panix home page. And customers could tell their friends and business associates to write them at, say, firstname.lastname@example.org, instead of email@example.com.
At around the same time, Rosen says, a network engineer working for Telecom Ottawa Ltd. in Ottawa, Canada, became aware of Panix's plight. The engineer was able to override the domain name system data that was causing Panix's mail to go to the wrong mail server, which was physically located in Canada. "He didn't check with anyone; on his own, he changed some of the DNS information so that it pointed to a nonexistent server," Rosen explains. Thus instead of mail being received by the Canadian mail server, it was undeliverable. "When that happens," Rosen says, "it's queued by the servers that sent the mail. So mail would be delayed but not bounced, lost, or read by hackers." Unsure whether Telecom Ottawa would reward or punish such initiative, Rosen declined to name this unsung Canadian hero.
Before Melbourne it would change Panix's registration record , it wanted to hear from Dotster, which had its own problems. An enormous ice storm in the Pacific Northwest had caused the company to close its own customer support lines early on Saturday. Nonetheless, Rosen says he was able to reach a weekend support technician at about noon Saturday, New York time, which was 9:00 a.m. for Dotster. "He didn't believe the domain was transferred away from us," Rosen says, "because Dotster showed no record of the transfer. So I knew the system had been broken in some way."
The current method of transferring registrations is supposed to get around a thorny business issue, says Columbia University's Bellovin. The new registrar is probably happy to get your business, but the old one doesn't want to lose it. "What if the existing registrar is unresponsive?" Bellovin asks. "For that reason, the procedures no longer require a positive acknowledgment from the old registrar, because there were reports of people having to pay more to get away from a registrar they didn't like." It was only a few months ago, in November 2004, that the process was altered by the Internet Corporation for Assigned Names and Numbers, or ICANN, in Marina del Rey, Calif., which has responsibility for some of the rules concerning registrars and registries.
Normally, when a registration transfer is requested by the new registrar, the registry will contact the old registrar. If the registry doesn't hear from the old registrar in five days, it allows the transfer to proceed. Bellovin says this was the change that was made in November--what to do when the old registrar doesn't respond. Previously, the registration transfer didn't go through, and the domain owner remained with the recalcitrant registrar. The original registrar can still deny a transfer if contacted by its customer within five days of the transfer request, or for other reasons, such as nonpayment.
There are other safeguards in the process. Some registries also require that the registrant, Panix in this case, be given a password, which then must be invoked when a transfer is requested. VeriSign, as the registry for .com, does not make that a requirement.
There's one other backstop as well. A registrant can instruct the registrar to lock the domain at the registry. This involves setting a field in the registration record to say "locked" or "unlocked." If the record is locked, the registry won't make the transfer. If a domain owner wants to change registrars, it's a simple matter to unlock the record before initiating the transfer. According to Melbourne IT, VeriSign has confirmed that the panix.com record was unlocked. Oddly, though, panix.net, and panix.org, which Panix also owns, and which are also registered with Dotster, are both locked, and were locked well before Saturday's events. Also according to Melbourne, Dotster was notified of the transfer, despite telling Panix otherwise in the early hours of the hijacking.
It took a failure of all these procedures to hijack Panix, and it would take a circumvention of them to get things right again. Of course, companies like Melbourne IT and VeriSign are on guard against such improper interventions. Hijackers could use just that kind of social engineering--sweet-talking people to go outside the normal procedures--to pull registrations out from under them.
Thus, for Melbourne IT, "The big issue was authenticating [Panix's] request," says Tonkin. Fortunately, whenever Melbourne IT or its resellers change a DNS record, it makes a copy of the existing one. "So we reverted the record back to what it said before the transfer."
Tonkin says that as soon as Melbourne IT verified the record, it was ready to transfer the name back. "But VeriSign advised us to get Dotster to initiate a transfer back, and this took a further 9 hours before Dotster responded to our request." Thus, Panix's service was fully restored, two full days from the start of the attack.
There's a pile of action items after Saturday's snafu that stretches around the globe. Any number of domain owners have already checked to see if their registrations are locked, and have locked them if they weren't. Melbourne IT plans to tighten its procedures, audit its resellers more rigorously, and publish better emergency contact information. Law enforcement officials in the United Kingdom, the United States, and Canada are looking into the credit card theft and what other crimes were committed by the hijacker.
Steve Crocker says that ICANN's Security and Stability Advisory Committee, which he chairs, will examine yet again the thorny balance between keeping it easy to transfer a domain registration and making it secure. And Alexis Rosen and his staff at Panix? They're catching up on lost sleep.
This story was corrected on 3 February 2005.