There was a great story over at Ars Technica this week regarding a recently published special audit report (pdf) by West Virginia’s Legislative Auditor regarding the state’s purchase three years ago of 1164 Cisco model 3945 routers at a price of US $24 million using federal stimulus funds (a tip of the hat to a Risk Factor reader for bringing this to our attention in a comment to a recent post). The auditor concluded that not only did the purchase bypass the state’s competitive purchasing rules for IT equipment; the state bought far more capability than it would ever need now or in the foreseeable future, and at non-competitive prices to boot.
The audit report, for example, gives as an example the “city of Clay in Clay County [which] received 7 total routers to serve a population of 491. Five of these routers are located within .44 miles of the each other.” The cost of those seven servers—each of which can support 200 simultaneous users—was around $20 000 apiece.
The auditor noted that over $6.6 million was spent on Cisco model 3945 router features that weren’t necessary to begin with. Furthermore, if the state had actually purchased the correctly sized routers, it could have saved at least another $8 million or so. I say at least, because that number is based on router prices quoted in a non-competitive bidding environment—holding a competition that included other router manufacturers (Alcatel-Lucent, Brocade, HP, Juniper, et al.) would have likely saved even more money. For each $5 million saved on routers, the state could have purchased 104 additional miles of needed broadband fiber, the auditor noted.
I name those manufacturers specifically because the West Virginia audit report points to “California State University, the largest four-year university in America, [which] used a competitive bidding purchase to purchase an eight-year refreshing of its 23-campus 10G network. The Director of Cyber Infrastructure of California State University provided documentation showing that Alcatel-Lucent won the project with a bid of $22 million. Cisco’s bid was $122.8 million. The other bids were Brocade at $24 million, Juniper at $31.6 million, and HP at $41 million. Furthermore in May of 2011, Purdue University bid out replacement components for its Hansen Computer Cluster. Cisco won the Purdue University competitive bid process by offering a 76 percent discount off the cost of its products.”
Why did this wasteful fiasco happen? The audit report basically says no one really knows for certain—or at least is willing to 'fess up to being the party who screwed up: stuff just sort of happened. The best that can be determined was that those receiving the federal stimulus funds wanted to spend as much of them as fast as possible, need be damned. Or in the auditor’s words, “Those making the decisions on how to spend the money did not consult individuals with technical knowledge on the best methods to utilize the funds.”
The audit report tried to place some of the blame on Cisco for selling routers that the state obviously didn’t need. Cisco, which admits its routers are expensive, says the blame doesn't rest with them: It merely sold equipment to West Virginia in the quantities and type the state asked for.
What happened with the routers does not bode well for West Virginia’s $98 million system modernization effort, which has been under fire for government project risk mismanagement.
If you have a chance, read over the Ars Technica article as well as the auditor’s report. There is also an article at WCHS Radio 58 concerning members of the U.S. House Subcommittee on Communications and Technology trying to find out why as part of the Cisco purchase a $22 500 router was bought for a West Virginia library with a single computer.
Head of the U.S. National Telecommunications & Information Administration, Lawrence Strickling, staunchly defended the purchase saying, “I think it's not at all clear from those reports that what West Virginia did was unreasonable in terms of its choice of a single platform Cisco router at the time they made it.” Strickling says that those making the purchase were smart to purchase equipment that would allow for future growth.
Strickling didn’t say when in the future he expected the population being served by the Marmet Public Library, where the lone router now sits and is open only three days a week, would grow to a level where the library would need to serve 220 simultaneous users. With the local population currently of about 1500, my guess it will be long after that Cisco router is sent to the recycling center.
After reading through the audit report and other articles, IT professionals will be highly amused, and American taxpayers, whose federal stimulus money was wantonly wasted, will be highly outraged. I was both.
Photo: West Virginia Legislative Auditor
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.