The Blackout of 2003

The initiating events appear to have happened under the lazy eyes of a mismanaged utility, but underlying conditions made a massive U.S. power failure almost inevitable

With additional reporting on sidebars by Willie D. Jones, Elizabeth A. Bretz, and Chris Lang

21 August 2003--What was by most measures the biggest electricity outage in history, surpassing the blackouts in the western United States in the summer of 1996, swept northeastern and Great Lakes states and the Canadian province of Ontario late Thursday afternoon, 14 August. Long before power had been restored to businesses and residences from New York City to Cleveland, Detroit, and Toronto, politicians and commentators on both sides of the border were pointing fingers. But, in fact, major difficulties in the electric power system had been predicted by three U.S. Department of Energy (DOE) studies going back to 1998, and had been duly reported in the press (including IEEE Spectrum), with plenty of blame for inaction to go all around.

At this writing, it appears that the 2003 blackout started in facilities owned and operated by FirstEnergy Corp., a large utility headquartered in Akron, Ohio. Early in the afternoon of 14 August, one of its coal-fired power plants began to behave oddly and had to be taken off-line, or "tripped" in the industry parlance [see Timeline]. An hour later and perhaps coincidentally, at 3:06 p.m., Eastern Daylight Time, one of the company's major transmission lines failed. Inexplicably, the alarm system meant to warn the utility of such problems did not operate properly, and so FirstEnergy did not give regional regulators and organizations in adjacent states any warning of the mishap.

Over the next 45 minutes, three more transmission lines failed--two owned by FirstEnergy, the other by American Electric Power (Columbus), the other big Ohio utility--at 3:32, 3:41, and 3:46 p.m. By 4:30 p.m., most people in Ohio, Michigan, Ontario, New York State, New Jersey, and Connecticut were without power.

In the coming weeks and months, as regulators and power engineers sift through thousands of event records, the focus initially will be on what exactly the initiating events were, why failures propagated so rapidly through the northeastern grid, and why the grid system operators established to prevent such disasters were unable to deliver. Initially, responsbility for the inquiry was taken by the North American Electric Reliability Council (NERC, Princeton, N.J.), the self-regulating utility organization meant to improve the trustworthiness of the grid system, and it will continue to provide data and analysis. NERC's leaders have let it be known that an objective will be to determine whether key players failed to follow rules properly or whether the rules themselves were defective.

"We were all just waiting for the big one."

Ultimately, however, the issues raised by the blackout go far beyond that, which may be why DOE took charge of the inquriy on 20 August. For more than five years, NERC has sought and failed to get legislative authority to make its rules mandatory. During the same period, its U.S. government counterpart, the Federal Energy Regulatory Commission (FERC, Washington, D.C.), has been struggling to impose stronger regional oversight over grid operations--often encountering impassioned opposition from big utilities and their political allies. (FERC deals mainly with state and local governments, NERC with utilities.) Meanwhile, as growth in demand for electricity has outstripped additions to transmission capacity by a factor of two, the grid itself has come to be ever more thinly stretched.

The net result, as virtually all experts on the nation's grid system came to agree, was a disaster waiting to happen. "We all knew something like this was coming along," a leader in a 1999 DOE study of the transmission grid told IEEE Spectrum, on condition of anonymity. "We were all just waiting for the big one."

"Look," agreed Karl Stahlkopf, an executive at Hawaii Electric Co. (Honolulu) who previously managed power system research at the Electric Power Research Institute (Palo Alto), "everybody in the business knew something like this was going to happen. It wasn't a question of whether but when."

One rotten apple?

If all the experts knew a major outage was going to happen, why couldn't anybody prevent it? The temptation, at first blush, may be to put the blame squarely on FirstEnergy, the Ohio utility whose operators and managers seem to have been sleeping at the switch. Could this be a mere case of one poorly run company bringing down an entire system--of one rotten apple corrupting the whole barrel?

As it happens, FirstEnergy owns and operates the Davis-Besse nuclear power plant east of Toledo. Barely more than a year ago, that plant was found to have suffered such severe corrosion in the cap where fuel and control rods enter the core that a catastrophic loss-of-coolant accident seemed possible. Perhaps because of that discovery, suspicions immediately centered after 14 August on the way FirstEnergy has been conducting its business generally. "Having come less than an inch from potential radiation leakage from Davis-Besse, they've now succeeded in blacking out eastern North America," a bond analyst with CreditSights (New York, N.Y.) commented in a report.

Reporters for The Wall Street Journal, The New York Times, and ABC-TV's "Nightline" news analysis show promptly uncovered a pattern of chronic difficulties FirstEnergy had had with regulators. These included penalties for violation of health, safety, and environmental rules, a requirement by auditors that it restate its profits over the last few years, and the cost of replacing electricity from the shut-down Davis-Besse plant, which ran into hundreds of millions of dollars. The company also is saddled with billions of dollars in debt, much of it associated with its acquisition in 2001 of the utility GPU Inc. (Morristown, N.J.), the former owner and operator of the Three Mile Island nuclear plant that suffered a near meltdown in 1979.

As a result of all those difficulties, FirstEnergy saw its bond rating cut to the lowest investment grade by Moody's, just hours before the incidents that precipitated the Northeast blackout. Four days after the blackout, Standard & Poor's followed suit, dropping FirstEnergy's bond rating to near-junk status.

The bad press continued as crusading Stanford University law professor Lawrence Lessig circulated allegations that the company had begun its ill-starred career as an electricity conglomerate by driving a municipal utility in Cleveland out of business.

Ironically, the company's activities in Cleveland propelled Democratic presidential candidate Dennis Kucinic into city hall in 1979, making him the youngest mayor in the country. Kucinic, now a U.S. representative from Ohio, wrote this on Lessig's Web site: "If there was ever an example of the failure of necessary regulation by the government of an investor-owned utility, it is found in the government's failure to regulate FirstEnergy."

The company points out, quite rightly but also narrowly, that it is too soon to tell whether its tripped power plant and lines were the cause or effects of the Northeast blackout. In a formal statement, it quoted NERC president Michehl Gent as saying that "any attempt on our part to identify the cause of the outages at this point would be premature."

The system fails

Even if the tripped lines and FirstEnergy's failure to detect and report them prove to be the exclusive initiating cause of the 2003 blackout, it still will be necessary to account for the failure of local and neighboring control authorities to prevent the cascading outages that ensued. As events unfolded, observers were struck at how unevenly various regulating organizations performed.

The neighboring PJM Interconnection (Valley Forge, Pa.), a regulatory body that first transmuted into a so-called independent system operator (ISO) and then regional transmission organization (RTO), was quick to detect problems on FirstEnergy's lines and to protect itself from them--quicker than FirstEnergy itself, evidently. (PJM's jurisdiction originally was Pennsylvania, New Jersey, and Maryland, but now also includes parts of Ohio, Virginia, West Virginia, the District of Columbia, and Delaware,) The New England ISO (Holyoke, Mass.) also largely protected its region. The New York ISO (Schenectedy, N.Y.), with operating rules that PJM and New England reportedly consider too weak, was not successful.

Least effective of all, it seems, was the performance of the Midwest ISO, based in Carmel, Ind. Lacking--in contrast to some of the other ISOs--the authority to impose rules on utilities in its area, the Midwest ISO was reduced to contacting them at times of trouble and advising them on what it thought they should do.

How did this crazy quilt of regulating authorities come about? Basically, the U.S. government sought from the early 1990s to create an open-access grid system, in which independent power producers and power brokers could sell and buy electricity. At the same time, it also encouraged states and regions to create ISOs to oversee or even directly manage their grids and sometimes to organize the local energy exchanges as well.

Long before most states had even begun to set up ISOs, however, it was becoming apparent to federal regulators that the supposedly independent organizations were held hostage by local utilities and other special interests. What's more, ISOs lacked adequate authority to get transmission lines built and transmission services properly priced.

In midstream, therefore, FERC started to encourage the parallel creation of RTOs, with broader jurisdictions and powers. That effort, and FERC's subsequent attempt to win general acceptance of a "standard market design" based on PJM Interconnect's successful model, ran into wide resistance. In regions like the Southeast and Northwest, large and influential utilities didn't like the whole idea of deregulation to begin with. And places like California, which had gone to a lot of trouble setting up an ISO, didn't feel, now, like going back to the drawing board again.

But it was in the Middle West, tellingly, where FERC's effort to get an RTO created turned out so badly. In Ohio and its immediately neighboring states, FERC faced a maze of local regulatory authorities, hugely complex conflicting interests, and two titanically powerful utilities, American Electric Power and FirstEnergy. Tortured negotiations to create an RTO dragged out over a couple of years, and finally, at the end of 2001, FERC rejected the proposed Alliance RTO as utterly failing to meet the specifications it had set out. Jurisdiction over the Ohio grid ended up divided between the weak Midwest ISO and PJM, with blurry lines of authority.

Ultimately, a human failure

Six months ago, NERC singled out the Middle West as the one part of the country at risk of a devastating grid breakdown, and put local utilities and regulators that they had to be on high alert this summer. Evidently the message did not get through. Why is that?

The process of electricity restructuring, besides leaving many parts of the country with gaping holes in their regulatory nets, was damaging in other ways as well. Because of uncertainties as to who ultimately was going to end up owning the major elements of the transmission system, nobody wanted to invest in it and build lines--a supremely frustrating business even under the best of circumstances.

On foot

New Yorkers (including four IEEE Spectrum editors) walked home over the Brooklyn Bridge, as the city was without power for subway trains or traffic lights.

At the same time, as utilities and power producers girded themselves for the new world of competition, they sought to streamline operations and shed staff. Linesmen often left the field to work for the booming telecommunications companies, which were laying optical-fiber cable as fast as they could buy it, while operations engineers migrated to the newly forming ISOs or RTOs--or just retired.

The baleful consequences of that process became apparent in the two huge outages that swept the western states in July and August 1996. The results were described in a report on grid reliability prepared for the DOE by an elite panel in December 1999: "All of the problems that the Western Systems Coordinating Council (WSCC) identified after the August 10 breakup had been progressively reported to it in earlier years…. Through a protracted decline in planning resources among the member utilities, the WSCC had lost its collective memory of these problems and much of the critical competency needed to resolve them."

"The market forces that caused this," the report went on to say, "pervade all of North America. Similar effects should be expected in other regions as well," including the eastern interconnection, where the same kind of migration of technical support from utilities to emergent ISOs, with attendant loss of memory, was apparent.

Fixing the system

For U.S. political conservatives, who traditionally are sympathetic to state's rights and hostile to expansion of federal powers, it has been tough to swallow the iron logic that making the electricity system work necessarily implies a massive transfer of regulatory authority from local public service commissions to FERC and NERC. Yet in the wake of the 2003 blackout, that bitter pill seems at last to be going down.

In an editorial published Monday, 18 August, four days after the blackout began, The Wall Street Journal noted that "some of our friends on the right have assailed [FERC's scheme for RTOs] as a federal takeover of state prerogatives. And yes, in a perfect world we would prefer if the states eliminated their franchise monopolies on power generation. While we wait for that utopia to arrive, however, the rest of us have to find some way to avoid blackouts that close down much of the country." Therefore, said the Journal , it now supported FERC's standard market design and hoped President George W. Bush and Congress would give FERC the authority to impose it.

In an accompanying column, William Hogan of Harvard University's John F. Kennedy School of Government (Cambridge, Mass.) noted that the Senate Commerce Committee had recently voted to bar FERC from implementing RTO market designs based on the PJM Interconnect model. But, he wrote hopefully, "the blackout should change the game."

Whether that will be so remains to be seen. Several weeks before the blackout, in the interest of getting a national energy bill passed after five years of haggling, Senate Republicans accepted a Democratic draft that contained language giving NERC and FERC more authority. Yet Senate and House members continue to argue now about whether a stand-alone electricity bill should be enacted immediately, or whether it should remain part of a comprehensive and controversial energy bill, which would continue to be hostage to every kind of horse trading. Longtime observers, such as Hawaii Electric's Stahlkopf, are not particularly optimistic.

But as long as the U.S. government fails to sort out the transmission mess, there will continue to be inadequate investment in needed technology, both old and new. Until then, observed John F. Hauer of the Pacific Northwest National Laboratory (Richland, Wash.), and an author of the DOE's 1999 grid study, "Nobody is in charge, and nobody is picking up the bills."

In the meantime, the country will continue to suffer the ill effects of what former Energy Secretary Bill Richardson, now governor of New Mexico, has called a "third-world" transmission system. Ilya Roytelman, a power control engineer in Minneapolis, Minnesota for Siemens AG, with experience working in Russia and Europe, says the U.S. gird system is 30 years behind the state of the art. "Every system in Europe works better," he told Spectrum .

Asked why the United States remains so far behind, Roytelman said the principal problem is that Americans just don't want to shed load--industry parlance for quickly killing power to some customers in order to lessen the chance of a regional blackout. But without shedding load promptly when the whole system is endangered, cascading blackouts cannot be prevented.

Second, says Roytelman, echoing a viewpoint that is ubiquitous among the leading U.S. experts on the grid system, "the best students are not going into power engineering."

Fernando Alvarado, an emeritus professor at the University of Wisconsin (Madison) who was deeply involved in the most recent DOE transmission studies, notes that many universities, including his, are suffering severe resource problems. Those problems need to be resolved "to attract a different kind of engineer, and [to] rethink the problems that have afflicted the U.S. grid systems," he says.

To get adequate brainpower into the pipeline, it may be necessary for the federal government, state governments, and universities to all do more to improve power engineering programs. In the final analysis, however, the problem is a chicken-and-egg one: the brightest students will not pick power engineering until emerging organizations like RTOs, ISOs, and reorganized utilities create the high-paying and challenging jobs that beckon them once their studies end.

To Probe Further

For backround on the crisis in power engineering education, see " Power Engineering at the Crossroads," by Adrul H. Chowdhury, IEEE Spectrum, October 2000, pp. 64–69

Related Stories