Risk Factor iconRisk Factor

French Hospital's Computer “Bug” Trashes a Fortune in Perfectly Good Drugs

IT Hiccups of the Week

Last week saw another wave of healthcare-related IT malfunctions, problems, and issues being reported. This time, we turn our focus to a controversy currently capturing the attention of the French press: the startling admission by administrators at the university hospital in Rennes that perfectly good drugs and other medical supplies are being trashed as a result of technical issues with its relatively new automated pharmacy system.

What first drew my attention to this story was an English-language story appearing at the International Business Times that claimed the Centre Hospitalier Universitaire de Rennes—better known as CHU—has sent a staggering €10 million worth of medicine and other medical paraphernalia to its incinerator because of a “computer bug” in the pharmacy distribution “robot” that was installed at the hospital in 2010. According to the IBT story—which claimed as its source an investigative story published in The Monthly Rennes—the “bug led to duplication and storage problems, which caused pallet-loads of medication to be destroyed.”

Read More
A man and a woman, each with a child in their lap, sit and wait in a crowded room.

900,000 Medi-Cal Applicants Stuck in California Healthcare Backlog Purgatory

IT Hiccups of the Week

Last week saw an emergency room full of healthcare-related IT problems, issues and challenges being reported.  We chose to concentrate this week’s edition of IT Hiccups on one which resembles a healthcare version of Hotel California where hundreds of thousands of California Medi-Cal health insurance applications have been checked in, but can’t seem to leave the confines of California government offices.

Read More

Fat Finger Flub Takes Down Cloud Computing Datacenter

IT Hiccups of the Week

A wide variety of IT-related blips, failures, and mistakes occurred last week. However, the most interesting IT Hiccups related story involved what was described as a “fat finger” error by an operator at the cloud computing service provider Joyent’s US-East-1 datacenter in Ashburn, Virginia. It disrupted operations for all of Joyent’s datacenter customers for at least twenty minutes last Tuesday. For a small number of unlucky Joyent customers, the outage lasted 2.5 hours. 

According to a post-mortem note by a clearly embarrassed Joyent, “Due to an operator error, all us-east-1 API [application programming interface] systems and customer instances were simultaneously rebooted at 2014-05-27T20:13Z (13:13PDT).” The reason for the reboot, Joyent explained, was that a system operator along with other Joyent team members were “performing upgrades of some new capacity in our fleet, and they were using the tooling that allows for remote updates of software. The command to reboot the select set of new systems that needed to be updated was mis-typed, and instead specified all servers in the datacenter. Unfortunately the tool in question does not have enough input validation to prevent this from happening without extra steps/confirmation, and went ahead and issued a reboot command to every server in us-east-1 availability zone without delay.”

Read More

1 Million Americans Likely Receiving Incorrect Federal Healthcare Subsidies

IT Hiccups of the Week

Last week saw an increase in the number and types of IT-related malfunctions, ooftas, and errors reported over the previous week. The most interesting IT Hiccups-related story of the lot was one from the Washington Post indicating that 3 million of the 8 million or so individuals enrolled for health insurance under the Affordable Care Act have a variety of “discrepancies” in their applications, including 1.1 million to 1.5 million applications with levels of income individuals are claiming that differ “significantly” with Internal Revenue Service (IRS) documents.

As a result, the Post estimates, the Obama Administration may be making improper subsidies payments for more than 1 million Americans to help pay for their health insurance plans, including hundreds of thousands likely receiving higher subsidy amounts than they are entitled. Furthermore, because of computer-related issues, it may be well into late summer before the situation can be rectified. As of today, the Administration has not lowered (or increased) anyone’s subsidy amount because of these discrepancies, the Post reports.  A person who is currently receiving too high a subsidy will be required to pay it back to the IRS by April 2015, however. Given that those receiving subsidies require low incomes in the first place, a person who is innocently getting too high a subsidy may be facing a massive, and potentially bankrupting, tax bill next year.

And for those found to be lying on their applications in an attempt to receive a higher subsidy than they deserve, not only are they going to get a nasty payment demand from the IRS,  but they also face a $250 000 civil fine. Even an “honest error” on an insurance enrollment application can result in a civil fine of $25 000, although I doubt that will ever be imposed, because it would be a surefire way to discourage lots of subsidy eligible individuals from signing up for health insurance.

Paying out health insurance subsidies (or even approving health insurance) without first fully verifying income (and other application information) was never supposed to happen. According to the original plan, the “back-end” office IT systems linking the different government agencies and departments such as the IRS, Department of Homeland Security, the Social Security Administration, the Veteran’s Administration,  and so on were to be in place by 1 October so that the income, immigration status, citizenship, age, other health insurances received, etc. could be verified almost immediately upon receipt of a person’s health insurance enrollment application.

However, because of all the troubles trying to get the ACA website itself working by 1 October and afterwards, the Administration decided early on to defer the back-end office systems development.  Henry Chao, deputy chief information officer at the Centers for Medicare and Medicaid Services testified in front of the US Congress last November that 30 to 40 percent of the back-end systems work still needed to be completed. While everyone knew the systems were behind schedule, to say that Chao's admission took Congress by surprise is an understatement.

Kathleen Sebelius, the Health and Human Services Secretary at the time, promised a startled Congress that the back-end systems would be in place by mid-January, so not to worry. That timetable, however, soon slipped to mid-March, and now, as the Post reports, to late summer. Some $121 million has been budgeted since the beginning of the year to try to get these back-end systems working, which comes on top of the unknown tens of millions of dollars previously spent on their development.

Interestingly, when the systems' development timetable slipped from mid-January to mid-March, the Administration admitted that if the back-end systems were not completed by the March date,  “the entire healthcare reform program [could] be jeopardized” because the “issuance of [incorrect] payments to health plans ... could seriously put them at financial risk; potentially leading to their default and disrupting continued services and coverage to consumers.”

Another reason why the lack of back-end systems could place ACA in jeopardy is that without them, the Federal government cannot accurately determine how many people have even paid their health insurance premiums. The lack of valid premium payment data along with improper subsidy payments could lead to wrongly predicting the “Risk Adjustment, Reinsurance, and Risk Corridor [pdf], potentially putting the entire health insurance industry at risk," Administration documents stated.

The Obama Administration naturally is trying to play down its own dire warnings about paying out subsidies without first verifying a person’s eligibility or knowing who has and hasn’t paid their premiums, which has angered many Republicans in Congress, some of whom who are calling for a suspension of unverified subsidy payments. The Administration is unlikely to do that, but if it can’t get those back-end systems up and running in the next few months, there will likely be an increasingly large political—not to mention huge financial—cost to be paid.

Speaking of cost, a story by Politico reports that $475 million has been spent so far on the failed Oregon, Massachusetts, Maryland, and Nevada health insurance exchanges alone. Both Maryland and Massachusetts, however, now want even more money to fix their broken exchanges, while the FBI is trying to determine whether there was fraud involving the Oregon exchange effort. Nevada announced last week that it has decided to throw in the towel on its own exchange, and will use the Federal one instead, at least for 2015. Rumor has it that Rhode Island, which has a working exchange, may soon decide to move to the Federal exchange as well. In Rhode Island’s case, the future costs of operating and maintaining its exchange is starting to look increasingly unaffordable.

The Politico story also says that according to the Kaiser Foundation’s calculation, the Federal government has given state governments some $4.698 billion since 2011 to support their exchange development efforts.  In addition, according to recently released figures, the Federal government has so far obligated some $834 million to create its exchange and back end support systems, and will need another $200 million in Fiscal Year 2015 to maintain it, which is about double what was thought to be required to develop the exchange in the first place.

Finally, last week the real story of the first day ACA enrollments came to light thanks to a Freedom of Information Act lawsuit by Judicial Watch. Whereas it was long thought that only 6 people were able to enroll that first day, in reality, only one person was ever able to enroll for health insurance coverage, even though 43 208 accounts were created, government records show.

In Other News ...

Oregon Secretary of State Website Error Delays Congressional Vote Reporting

eVoting Machines Malfunction in Pennsylvania’s Westmoreland County

Hotel Reservation System Error Overbooks Rooms for Tennessee Ironman Competition

Computer Malfunction Causes CSX Train to Block New York Road

IT Ticketing Problem Fixed for Glasgow 2014 Commonwealth Games

Massachusetts Still Trying to Fix New Unemployment System Issues

Tacoma Washington Schools Experience More Online Testing Woes

New York City Places “Fundamentally Wrong” 911 System Implementation on Hold

U.S. Immigration Court System Outage Enters Week Six

IT Hiccups of the Week

It’s been another slow news week in the world of IT-related errors, problems and failures. The most interesting IT Hiccup of the week involves a puzzling six-week outage affecting the computer system that supports U.S. Executive Office for Immigration Review (EOIR) administered courts across the country.

About a month ago, the New York Post ran a story describing a “tech meltdown” involving five servers located in Falls Church, Virginia, on 12 April. The servers are an integral part of the computer system supporting the 59 immigration courts administered by the EOIR. According to the Department of Justice, the EOIR “primarily decides whether foreign-born individuals, who are charged by the Department of Homeland Security (DHS) with violating immigration law, should be ordered removed from the United States or should be granted relief or protection from removal and be permitted to remain in this country.”

As a result of the outage, which also took out the mandatory electronic registry for accredited immigration attorneys and representatives, the 260-plus immigration court judges (and staff) have had to return to pencil-and-paper methods. As described at JDSUPRA.com, the problem has kept “court clerks from accessing court records, entering new ones in the system, and making audio recordings of hearings.”

Read More

British Columbia's Integrated Case Management System Falls Over

IT Hiccups of the Week

After the previous week’s deluge, last week saw a return to a more “normal” number of reported IT-related malfunctions, errors and problems. The most significant involves British Columbia’s controversial CAD $182 million Integrated Case Management (ICM) system. It has been plagued with so many operational issues that government social workers using the system complain that it has been essentially unusable for over a week, the Vancouver Sun reported.

The ICM (pdf) is a new  government IT system aimed at streamlining the management of computer files across the three British Columbia government ministries (Social Development and Social Innovation; Children and Family Development; and Technology, Innovation, and Citizens’ Services) that provide social services to “poor children, disabled people and troubled families racked by addiction, mental illness and violence.” The ICM system is being introduced in four phases. Phase 1 occurred in November 2010; Phase 4, which is the final phase, is scheduled for roll-out at the end of this year.  

I previously noted that with the introduction of Phase 2 in April 2012, social workers bitterly complained that the ICM system kept freezing, data routinely disappeared, and the system was extremely cumbersome to use. Government officials at the time tried to play down the problems, characterizing them as just the routine “challenges” accompanying any new IT system introduction. But independent assessment reports later confirmed the validity of the complaints. The assessment also indicated a rather dysfunctional project development that needed immediate correcting.

Phase 3 of the ICM system was introduced a year ago March, and the complaints about the system started to subside. That is, until a couple of weeks ago, when all the old performance problems seemed to reappear with a vengeance. Social workers were livid that they were once more having to deal “with blank screens, and the inability to access to their clients’ addresses and warnings that may have been part of the file” which forced them to revert to paper and pencil methods. Exacerbating their anger was the B.C. government’s decision not to disclose publicly for almost a week that there were new problems occurring with the ICM system.

That information blackout also raised the ire of Mary Ellen Turpel-Lafond, the province’s independent watchdog for children and youth, who said ICM's ongoing problems were placing vulnerable children and adults, especially those who might need protection, at risk. She also called for a new independent assessment of the ICM system's performance and development.

After the ICM problems became public early last week, British Columbia’s Technology Minister Andrew Wilkinson claimed they were nothing more than “intermittent slowdowns.” Further, despite all the warnings by Turpel-Lafond as well as the political opposition about vulnerable children, the problems encountered caused only minor impacts on the delivery of critical government services. B.C. Children’s Minister Stephanie Cadieux also strongly rejected the notion that children were being placed at risk, the Vancouver Sun reported.

On Wednesday, in a bit of delicious irony, the Times Colonist newspaper reported that Wilkinson confidently told reporters that all the issues with the system were now fixed, so, please would everyone just move along. However, within an hour of saying the ICM was up and smoothly running again, an opposition party official informed a very surprised Wilkenson (and apparently those same reporters) that the ICM had crashed and burned again.

Opposition party members had earlier in the week pointedly reminded Wilkenson that even while the ICM system was having its previously undisclosed troubles, he was boasting to the legislative assembly about the government’s “enviable record of deployment of top IT services.” The opposition members wanted to know whether Wilkinson was just out of touch with what was going on in his area of responsibility, or whether he had misled the assembly. Given the latest meltdown, opposition members were now inquisitive about his lack of up-to-date knowledge on ICM's status since they knew more about it than he did.

A highly chastened Wilkenson told the Times Colonist that he was “unhappy” about the latest ICM outage, and implied that he was misled about the system's true state.  He added that, “This is clearly a system that’s unstable. We’re going to get to the bottom of this, we’re going to report back to the house and to the people of British Columbia what’s gone on, and we’re going to seek our remedies.”

Wilkenson didn’t offer a timetable as to when the ICM would indeed be fixed, or when he would report on what was really going on with the ICM, however.

It now looks like the Auditor General of British Columbia, who is currently conducting an IT audit of the ICM system, will have one more issue to investigate. He may also want to ask Wilkenson why the government is claiming that the ICM system has been “available and functioning for the benefit of British Columbians nearly 100 percent of the time” since April 2012, when it demonstrably has not.

In Other News…

Bank of Montreal Apologizes over MasterCard Problems

U.S. State Department Reports New Green Card Lottery Glitch

Thousands of Northern Ireland Nurses Underpaid for Five Months

New Election Software Delays Voting Report in North Carolina

Maybank in Malaysia Apologizes for Multi-day Nationwide Operational Problems

Exposing the Roots of the Perpetual “STEM Crisis”

Okay, here are your choices: 1957, 1982, and 2014. Match each year to when the following statements were made:

a. “It is pretty generally realized that our country faces a serious scientific and engineering manpower shortage. We have at present about half the engineers which we need, and each year we are graduating only about half our annual needs.”

b. “Science, technology, engineering and math form the foundation of the global economy. Yet, … if educational trends continue, fewer qualified candidates will be available to support growth in these areas.”

c. “We appear to be raising a generation of Americans, many of whom lack the understanding and the skills necessary to participate fully in the technological world in which they live and work.”

Well, for the record, the order of the year when each statement was made is 1957, 2014 and 1982.  However, as explained in Michael Teitelbaum’s new book, Falling Behind?: Boom, Bust & the Global Race For Scientific Talent  (Princeton University Press:2014), whatever order you chose is as good as any other. Teitelbaum is a Senior Research Associate in the Labor and Worklife Program at Harvard Law School (and whom I interviewed for my IEEE Spectrum feature, The STEM Crisis is a Myth). According to him, since the end of the Second World War there have been regular proclamations of shortfalls in the graduation rates of engineers and scientists, as well as prophecies of the imminent loss of U.S. technical leadership caused by the abysmal education of U.S. students in math and sciences.  

Teitelbaum writes that these recurring concerns could well be cut and pasted into one sentence:

“The United States, long a leader in the number and quality of its scientists and engineers, has been falling behind its international competitors, and is thereby risking serious deterioration in its future prosperity and security.”

However, as Teitelbaum clearly demonstrates in his well-researched book, the past and current cries of an engineering and science crisis “are quite inconsistent with nearly all the available evidence,” going back to the early 1950s.

Teitelbaum begins Falling Behind? by examining the many hyperbolic claims of the current so-called science, technology, engineering, and mathematics (STEM) crisis. He expertly dissects these assertions and clearly demonstrates the weak assumptions and sloppy reasoning underlying each.

Teitelbaum next turns his attention to previous declarations of engineer and scientist shortages over the previous seventy years in a chapter aptly titled, “No Shortage of Shortages.” Teitelbaum has been debunking these claims since the 1980s. In Falling Behind?, he shows that the fear sown by the false alarms of impending doom did achieve their political goals of successfully spurring governmental actions to alleviate the supposed crisis by pouring more money into research and education. However, those crying wolf have also produced a series of (at least) five alarm boom-busts episodes that have predictably—if unintentionally—destabilized the U.S. science and engineering workforce. In addition, Teitelbaum argues, when the artificially created demand for engineers and scientist goes bust, it serves to discourage those very same sought-after students from pursuing engineering and science careers in the first place.

Teitelbaum's analysis shows that the basic problems in attracting students to science, technology, engineering, and mathematics are structural in origin, and “cannot be cured simply by providing additional resources.” He devotes a full chapter describing how the U.S. engineering and science “academic production process” operates, and how that process is insulated, especially at the graduate degree level, from the needs of the market. As Teitelbaum demonstrates, the current system of higher education has been geared for some time towards producing PhDs and postdoctoral students “irrespective of whether there is sufficient demand for such highly educated personnel in the market place.”

Especially useful is the light Teitelbaum shines on the many financial and political incentives that motivate industry, academia and government to proclaim an engineering and science crisis. He dedicates most of a chapter to, for example, exposing incessant lobbying efforts to increase the number H-1B visa workers to meet the supposed STEM shortage. Teitelbaum describes in detail the various lobbying organizations, interest groups and companies involved and how they are spending  tens, if not hundreds, of millions of dollars to change immigration policy.

Teitelbaum also distressingly, but accurately, points out how “advocates of these shortage claims have had a nearly open field in politics and the media,” and that while there have been decades of credible evidence debunking shortage claims, those making them are far less “organized, funded or politically connected” than those making those claims.  This disparity in resources, organization and political power make the fight for the truth, “really no contest.”

This was apparent in a recent “STEM Crisis Debate” sponsored by the Information Technology and Information Foundation (ITIF), a Washington, D.C. think tank. The ITIF insists there is a shortage of STEM students and workers, and is an advocate of increasing the number of H-1B visas. Robert Atkinson, ITIF president, for example, was quoted as saying, “If you don’t say there is a shortage, you don’t drive improvement.” That sort of admission (as do others) goes unnoticed in much of the press, as exemplified by the newspaper USA Today, which has been strongly beating the U.S. STEM crisis drum for quite some time.

Unfortunately, the outsized claims of a shortage in engineers and scientists will likely swamp Teitelbaum’s many sensible recommendations to start addressing the situation. One recommendation, for instance, is a call for the gathering of unbiased information on the STEM current and potential workforce. As he writes, “Given the importance of the health of the U.S. science and engineering enterprise and its talented workforce, it is striking that there continues to be no credible entity charged with regular, systematic, authoritative, and objective assessment of trends and prospects for education and careers in these occupations.” Until there is, one can confidently predict the past STEM alarm—boom—bust cycle to continue into the future unabated.

Another recommendation (or plea) Teitelbaum makes is for stability in government research and development funding. Past booms (and then busts) in funding have helped students and workers alike to pursue careers that just were, unsurprisingly, not sustainable. The recent boom and bust in the National Institutes of Health’s funding is one recent example Teitelbaum delves deeply into; the periodic increases and decreases in other government departments’ and agencies’ R&D provide fertile ground for others seeking examples.

Falling Behind? is a very useful addition to the science and engineering crisis literature, and fills in many of the areas not covered by Richard Freeman and Daniel Goroff’s Science and Engineering Careers in the United States (University of Chicago Press:2009) or Daniel Greenberg’s Science, Money and Politics: Political Triumph and Ethical Erosion (University of Chicago Press:2001). While the audience for Falling Behind? will tend towards the government and educational policy analyst crowd, the audience it deserves are the U.S. politicians and media editors and producers who are being bombarded daily with STEM crisis propaganda. Working scientists and engineers would also do well to give the book a read, though they may find it depressing.

However, while Falling Behind? is a comprehensive book about the origins of the science and engineering shortage mythology, it doesn’t tell the complete story. I think insufficient emphasis is given to how changes in market place demand have also contributed to the claims of science and engineering shortages. For instance, the petrochemical industry has been complaining of shortages of petrochemical engineering students for over a decade now. However, the industry’s leadership conveniently forgets about its previous massive and ruthless layoffs in the late 1980s and early 1990s that demolished the demand for petrochemical engineering students. Other industries that also like to moan about STEM shortages such as those in computing, aerospace and manufacturing, exhibit similar amnesia to their roles in creating them.

A number of years ago, the current Mayor of London Boris Johnson, then the Conservative Party spokesperson on higher education, bemoaned in the Guardian newspaper that Britain was failing to educate sufficient numbers of engineers and scientists such as nuclear physicists.

Sir John Rose, who recently retired as chief executive of Rolls-Royce, wrote a letter to Johnson about his article saying that, “If you don't have a nuclear industry, then anyone who is smart enough to be a nuclear physicist is not going to choose that career... People won’t become engineers just for the sake of it.”


Note: This post has been updated to reflect Michael Teitelbaum's current research affiliations.

Report Claims Spy Plane to Blame for Air-Traffic Outage on U.S. West Coast

IT Hiccups of the Week

Unlike the past month, last week saw an overflowing cornucopia of IT-related malfunctions, errors and complications. We start off this week’s edition of IT Hiccups with a strange story that has followed in the wake of a major air traffic control outage in the U.S.—one of several incidents that aggravated air travelers around the world.

The story starts off simply enough: On Wednesday afternoon, a little after 3:00 p.m. Pacific time, a computer problem occurred with the En Route Automation Modernization (ERAM) system at the Los Angeles Air Route Control Center. The snafu, a USA Today article reported, left, “controllers temporarily unable to track planes across Southern California and parts of Nevada, Arizona and Utah.” The FAA issued a ground stop on planes wanting to fly to Los Angeles for about an hour until the problem could be cleared up. However, that action caused the cancellation of some 50 flights arriving and departing Los Angeles International Airport and delayed another 455 flights across the country.

ERAM is part of a $2.2 billion Federal Aviation Administration (FAA) modernization effort, which as I have written about previously, has had its problems. So, while the computer problem was a major annoyance,  nothing there seemed out of the ordinary. The FAA issued its usual bland statement indicating that it was investigating the issue: “The FAA will fully analyze the event to resolve any underlying issues that contributed to the incident and prevent a recurrence.”

A spokesperson for the Professional Aviation Safety Specialists, the union that represents many FAA employees, hinted at the source of problem by telling USA Today that, “There was so much information coming into the system that it overloaded.”

That seemed to be the end of the incident—well, that is until the weekend, when NBC News ran a story online citing “sources familiar with the incident” who claimed that a U-2 spy plane flying in the area triggered the problems with the ERAM computers. (The article's title said the U-2 had "fried" the system.) The NBC story reported that:

“The computers at the L.A. Center are programmed to keep commercial airliners and other aircraft from colliding with each other. The U-2 was flying at 60,000 feet, but the computers were attempting to keep it from colliding with planes that were actually miles beneath it.”

“Though the exact technical causes are not known, the spy plane’s altitude and route apparently overloaded a computer system called ERAM, which generates display data for air-traffic controllers. Back-up computer systems also failed.”

NBC News contacted the FAA, which basically reissued its previous statement, but also would neither confirm nor deny that the U-2 was the cause of the outage. The U.S. Air Force also declined to comment directly on the story's details, and the Pentagon was not responsive to an inquiry from Reuters. The Wall Street Journal ran a story about the incident today, again, with government officials all deciding to remain mum.

When I saw the U-2 story at the NBC News website, I was more than a bit skeptical, especially over the "frying" statement. It is hard to believe that this is the first time military aircraft have passed through the LA ERAM control space at altitudes above normal commercial airline altitudes. On the other hand, I wouldn’t totally discount that there couldn’t have been some unique set of circumstances involving a military aircraft that just happened to be a U-2 triggering an unknown problem within the ERAM software. Still, my instinct is to attribute the problem to a more prosaic explanation.  If some official explanation comes out, I will update the post. However, feel free to speculate on what happened in the meantime.

Update 6 May 2014 : U-2 did indeed cause the LA-area ATC problem

The FAA admitted late yesterday that a U-2 did, in fact, trigger the problems last week with the LA-area ERAM system. Piecing together the FAA explanation from various news sources, e.g., NBC News, Reuters and CNN (no one seems to have published the FAA statement yet in its entirety):

“On April 30, 2014, an FAA air traffic system that processes flight plan information experienced problems while processing a flight plan filed for a U-2 aircraft that operates at very high altitudes under visual flight rules.”

“The computer system interpreted the flight as a more typical low altitude operation, and began processing it for a route below 10,000 feet.”

“The extensive number of routings that would have been required to de-conflict the aircraft with lower-altitude flights used a large amount of available memory and interrupted the computer’s other flight-processing functions.”

“The FAA resolved the issue within an hour, and then immediately adjusted the system to now require specific altitude information for each flight plan.”

“The FAA is confident these steps will prevent a reoccurrence of this specific problem and other potential similar issues going forward.”

The CNN story, which provides (so far) more detail than anyone else, indicates that the ERAM system was overtaxed because of the many waypoints the U-2 flight plan had filed. In addition, however, CNN reports that, “Simultaneously, there was an outage of the Federal Telecommunications Infrastructure, a primary conduit of information among FAA facilities.”  CNN did not indicate what caused the FTI to go out, however.

Altogether, the complex U-2 flight plan and the FTI outage added up to what one government official said on background was a “perfect storm” that took down the ERAM system. Well, sometimes truth is stranger than fiction.

U.K. Airline Travelers Unhappy over Passport Control Failure

Los Angeles-bound passengers weren’t the only ones unhappy last Wednesday. International airports and ports across the U.K. experienced unbelievably long queues at passport control as the U.K. Border Force computers went down at around 2:30 p.m. London time and the outage lasted for some 12 hours. Some passengers at Gatwick reported waiting in line for over four hours, and fights were said to have broken out among passengers waiting in line at both Gatwick and Luton Airports. While non-EU passengers waited the longest, UK citizens also had to wait for up to two hours as well, newspapers reported.

The Home Office put out a statement that read in part, “We apologise for the delays that some passengers experienced at passport controls yesterday, but security must remain our priority at all times.” The Home Office also said that its technical staff has been asked “to look into the incident to ensure lessons are learnt.”

However, whether those lessons to be learnt will ever be publicly disclosed is still to be determined. The Home Office is refusing to disclose what caused the massive computer meltdown, which seems to be its standard operating policy.

Completing the trifecta of airport problems that occurred last Wednesday was a report that a construction crew cut a fiber optic cable at Florida’s Fort Lauderdale-Hollywood International Airport at about 4:00 p.m. Eastern time. The loss of the cable resulted in in dozens of flights being canceled, delayed, or rerouted. No doubt passengers trying to travel to the U.S. West Coast from the airport thought they were snake-bit.

More General Motor Recalls for Software Issues

General Motors announced two more vehicle recalls to correct software issues in its vehicles. First, it is recalling some 56 400 Cadillac SRX crossover vehicles from the 2013 model year because of a software problem in the SRX transmission control module. According to the National Highway Traffic Safety Administration, “In certain driving situations, there may be a three to four second lag in acceleration due to the transmission control module programming.” No crashes have been attributed to the issue, GM reports.

In addition, General Motors said it is issuing recalls for 51 640 Buick Enclave, Chevrolet Traverse and GMC Acadia SUVs, from the 2014 model year that were built between 26 March and 15 August 2013. In these vehicles, GM says, the engine control software may cause the fuel gauge to read inaccurately the amount of fuel remaining, leading the vehicle unexpectedly running out of fuel. GM says it doesn’t know of any crashes that are linked to this problem, either.

In Other News of Interest….

New York City Accidentally Sends $298 475 644 in Duplicate Pension Payments to Retired Police and Fireman

U.S. Selective Service Sends Erroneous Military Draft Registration Letters to Marylanders

New Mexico’s Albuquerque Water Utility Authority Admits $9 Million Accounting Error

Washington State’s 911 Calls Routed to Colorado in Recent Outage

Virgin Mobile Australia Outage Hits 350 000 Customers

Virginia Tunnel E-Z Pass Toll Problems Continue Unabated

What’s the Wait for the Next London Tube Train? How About 939 Minutes?

Taipei Metro Suffers Fresh Problems

Some Indiana Schools Opt to Avoid Online ISTEP Tests

Michigan Hospital Still Struggling with New EHR System Installed in March

Is Australian Splendour Concert Ticketing Problem Glitch, Hack or Prank?

Computer Error Forces NY Stock Exchange to Cancel 20 000 Trades

Northern California Kaiser Permanente Says Insurance Billing Error to be Corrected Soon

Indiana’s Lebanon Utilities Trying to Correct Delayed Billing

Computer Issue Delays Paychecks for Flint Michigan Schools

Nova Scotia Power Overcharges Thousands of Customers

Computer Issue Causes Bank of Tokyo-Mitsubishi UFJ to Delay 23,000 Scheduled Wire Transfers

U.K.’s Norfolk County Council Staff Email Out for Over a Week

Computer Problems Delay Visitor Entry to Kentucky Derby

Erroneous Delinquent Tax Notices Sent to Michigan Residents

Kansas Farm Machinery Manufacturer Forced Shut by Computer Problems 

UK Farmers Still Struggling with Single Payment Scheme Online System

National Australia Bank Forced to Make More Compensation in Wake of 2012 Computer Problems

British Columbia’s Pharmanet Computer System Operating Once More

Online Testing Problems Strike Again

IT Hiccups of the Week

Last week was another relatively quiet week, with only a smattering of IT-related errors, malfunctions or problems being reported. But those that were gave one a definite sense of déjà vu.  Yet again, Oklahoma, Florida, and Indiana reported problems last week with their end-of-school-year online standardized testing.

On Monday, the AP reported that Oklahoma state education officials had to suspend online testing across the state for middle school and high school students; the tests either responded very slowly or quit working altogether. This comes after a similar occurrence last year, and promises from the testing vendor, CTB/McGraw-Hill, that the company would take steps to ensure that it wouldn’t happen again. CBT/McGraw-Hill apologized for what it called a “network service interruption” it said lasted only three hours.

Testing resumed on Tuesday and proceeded without further incident for the remainder of the week.

Last year's testing problems forced CBT/McGraw-Hill to forgo $1.2 million in order to settle damage claims filed by Oklahoma. However, this latest incident is likely to cost CBT/McGraw-Hill its $13.5 million per year online testing contract. State Superintendent of Schools Janet Barresi, who came under fire for last year’s testing fiasco and staked her reputation on ensuring that this year’s testing would be carried off without a hitch, said she would be recommending that Oklahoma not renew CBT/McGraw-Hill’s contract. If any other testing problems erupt this year, expect calls for Barresi’s contract not to be renewed as well.

Then on Tuesday, connectivity problems derailed online testing in many of Florida’s 67 school districts trying to administer the Florida Comprehensive Assessment Test (FCAT). The problems led state education officials to suspend online testing across Florida. Pearson, the company which holds Florida’s 5-year, $254 million online testing contract, apologized for the disruptions, which it blamed on its hosting service sub-contractor.

Florida Gov. Rick Scott called the testing problems “unacceptable” and said the state would “pursue all liquidated damages and other remedies that may be available as a result of Pearson’s failure to fulfill its duty under the contract with the department.” This is the second time in four years that there have been problems with the FCAT. In 2010, Florida fined Pearson $14 million for delivering the FCAT results late.  

FCAT testing resumed across the state on Wednesday, but some schools in the Miami-Dade area reported they were still having problems. Those, however, were attributed to a Microsoft security update issue.  No testing concerns were reported in Florida for the remainder of the week. 

Indiana, which, like Oklahoma, contracts with CTB/McGraw-Hill for its online standardized testing, also experienced massive problems last year. And it is reportedly very nervous about the start of its ISTEP tests this week. CTB/McGraw-Hill says that it is “confident about everything we have control over” and that it “stands behind all the measures” it took to ensure that nothing happens this year.  However, Indiana school officials are not as confident as CTB/McGraw-Hill given that some Indiana school districts were reporting trouble during their online ISTEP practice tests.

The four-year, $95 million ISTEP testing contract between the state and CTB/McGraw-Hill ends in four months, and the two still have not reached a final settlement over last year’s testing debacle.  

Ulster Bank Has another IT System Issue

In another case of déjà vu, Ulster Bank apologized to its ATM customers for being debited twice on each transaction over a 24-hour period spanning Monday and Tuesday of last week, the Belfast Telegraph reported. Ulster Bank has suffered numerous banking issues over the past few years, such as an outage in 2012 that lasted for weeks. The bank is still trying to repair its reputation over that failure.

The bank promises to refund all of its customers for the ATM error and says no customer will lose money as a result.

Oregon Surrenders and Calls in the Feds to Provide Healthcare Insurance Portal

Oregon announced as expected that after spending over $248 million, it is going to close down its Cover Oregon healthcare insurance website and use the federal Affordable Care Act website for the next enrollment period that starts in November. The cost of trying to fix Cover Oregon was not fiscally prudent, since the move to the federal website was estimated to cost $5 million while the estimated cost to repair the site was some $78 million—a cost that the federal government probably would not underwrite.

Also, as expected, Oregon state health officials refused to call the Cover Oregon debacle a waste of money. They did concede, however, that the IT failure was a “disappointment.” State officials still insist that Oracle was mostly, if not entirely at fault, for the failure, which Oracle vigorously denies—which reinforces the old adage that success has many parents, but failure is an orphan.

In Other News…

Homeland Security Experiences Immigration System Computer Meltdown

Some Galaxy S5s Shipped with Non-working Camera

New Zealand Exchange Experiences Trading Issue

Georgia Ports' Garden City Terminal Traffic Glitch Reappears

Vodafone Ghana Users Unhappy Over Outage

Astronauts Fix Faulty ISS Computer

Apple Offers to Replace iPhone 5 Faulty On-Off Button

Software Issue with Bluefin AUV Searching for Missing Malaysian Aircraft Search Fixed

Oracle Assails State’s “False Narrative” Explaining Cover Oregon Debacle

IT Hiccups of the Week

Last week was a very, very quiet week with only a few IT-related errors, malfunctions or problems being reported. So, for this week’s edition of IT Hiccups, we decided to revisit the ongoing and increasingly nasty public dispute surrounding Oregon’s health insurance exchange. Things took an interesting turn last week, courtesy of Oracle’s irate letter telling state government officials to quit lying about the company's role in the debacle.

As I wrote recently, Oregon’s attempt to implement its own health insurance exchange, called Cover Oregon, by 1 October 2013 to support the requirements of the Affordable Care Act (ACA) has not been exactly stellar. The state, after spending $200 million, is still trying to decide whether it will fix its website (no member of the public has ever directly enrolled for state health insurance coverage using it), use another state’s exchange software (like Maryland has decided to do), or default to using the Federal exchange. As I noted last week, news reports state that the number of “most serious programming bugs” discovered in Oregon's healthcare website implementation has reportedly grown from 13 in January to over 300 currently. State officials are supposed to decide any day now which alternative is most appealing.

Oregon state officials have not been shy about blaming its major software supplier, Oracle, for its problems. As a story at MSN last November documented, in a state hearing into the causes behind the Cover Oregon’s woes, Cover Oregon board member Ken Allen said:

“This is their [Oracle’s] failure. . . Their dates have shifted and shifted and shifted. . . The Cover Oregon staff are tremendously dedicated folks who have worked really hard. . . All of that good will and support from the business community is being frittered away because Oracle didn’t get it online. It’s 98% Oracle’s screw up.”

Democratic Oregon Senator Jeff Merkley also pointed the finger in November at Oracle on NBC’s Nightly News television program, saying that:

“Oracle was contracted to write the exchange. They promised it would be fully delivered on time, it would be beautiful and do more than any other exchange in the country, and it's in complete dysfunction.”

Merkley has continued placing the blame on Oracle in subsequent public appearances, as did Cover Oregon’s previous CIO Aaron Karjala, and Governor John Kitzhaber, who also claimed he was in the dark about how bad the problems really were. Kitzhaber did concede that the state may have been a bit at fault, but only in terms of having had an “unrealistically high sense of optimism that Oracle could deliver.”

Early in March, Oregon and Oracle reached an agreement to basically end the contract. The new deal calls for the state to pay Oracle $43.9 million of the $69.5 million the company claimed it was owed for its Cover Oregon work from November last year through February of 2014. Oracle had already been paid some $90 million for its previous project efforts. According to the agreement, at the end of sixty days, both sides could seek legal recourse to recover monies each thought due it.

The rapidly approaching end of the sixty-day period and the recent appointment of a new Cover Oregon executive director, Clyde Hamstreet, may well explain why Oracle President and Chief Financial Officer Safra Catz decided to end the company's silence about what it believed went wrong. Last week, Catz sent what can only be termed a provocative letter (pdf) to Hamstreet demanding the state to quit spreading a “false narrative” that seeks to blame the company for everything wrong with Cover Oregon.

Catz says in her letter that “contrary” to what Oregon officials have been telling the press, Oracle was never the lead in charge of the Oregon Health Exchange effort. In fact, Oregon “by choice and by contract, were squarely in charge of the project,” she writes. And in spite of repeatedly being told by outside experts to hire a system integrator and promises that it would do so, Catz said, Oregon declined to hire one.

Oracle’s only role was to “assist” the state with various tasks, Catz states. In addition, Oracle “provided clear and repeated warnings” for months that the effort was in trouble. A person “with even minimal IT expertise would have known that the system would not, and could not, go live on October 1.” Furthermore, some “critical specifications” were not given to Oracle until November 2013.

Catz goes on to say that the Cover Oregon system is actually working and has been for several weeks, despite the state saying it is not. System availability now usually is “exceeding 99%” and “the current error rate has dropped to 0.7%.” Catz says individuals could indeed sign up using the Cover Oregon website, but for some unexplained reason, the state refuses to let that happen.

Catz finishes her letter with a parting shot telling the state to do itself a favor and hire a systems integrator, as it promised it would do “at the beginning of the project.”

Cover Oregon’s Hamstreet wrote back (pdf) to Catz saying he was still getting up to speed, and couldn’t yet comment directly on the “factual assertions” contained in her letter.  However, he was forwarding her letter to Cover Oregon’s lawyers.

Hamstreet apparently has a great deal of experience in troubled projects, which might be one reason Catz decided to write to him as soon as he was hired. He probably knows that rarely is one party fully to blame for an IT debacle. In addition, Catz probably wanted to try to encourage Hamstreet to look deeper into the history of Cover Oregon, as well as put the state on warning that if a lawsuit ensues, Oracle will be looking to embarrass Cover Oregon and other state officials as much as possible.

In Oracle’s favor is that even a simple Internet search immediately turns up warnings of deep trouble (pdf) with the project dating back to 2012. Also, the Centers for Medicare & Medicaid Services released an independent report (pdf) about Cover Oregon in February of this year that described Oregon’s management of the effort, in not so many words, as abysmally amateurish.

This is not to absolve Oracle in any way, however. The company didn’t seem to have any problems taking the state’s money and running, as it were. If things were really so bad, and Oregon reneged on its promise to hire a systems integrator as Catz claimed, why didn’t Oracle immediately ask to be let out of the contract? In addition, for Catz to claim that Oracle was there to merely “assist” the state is clearly disingenuous. To say that Catz’s letter shows more than a little hubris and hypocrisy is putting it mildly.

It will be interesting to see whether Oregon or Oracle decide to take legal action against the other in the next few weeks. Although it would be fun to watch, my guess is no, since it looks like a real lose-lose situation. Oracle risks more damage to its reputation, as does Gov. Kitzhaber, who is running for reelection.

Amid this contretemps is even more drama: the former CIO for the Oregon Health Authority is contemplating suing Oregon for wrongful discharge and defamation—among other charges—based on what she claims is retribution for refusing to go along with a “cover-up” of the problems with Cover Oregon. Other state officials are probably not anxious to get hauled into court to testify about what they knew and when they knew it.  

So after a bit more public sword play among all the parties to leave a perception of really caring about wasting taxpayer monies, I expect this debacle will eventually be buried and forgotten, like so many other government IT projects before it.

In Other News…

Virgin Media Says Sorry for All the E-mails

Virginia E-Z Pass Suffers More Tunnel Software Problems

German Shepard Gets Jury Duty Call

GLONASS Systems Has New Problem

Australia's Commonwealth Bank Systems Go on Fritz Once More

Manila’s Ninoy Aquino International Airport Immigration Computer System Goes Down

Pennsylvanian Surprised by $43 000 Water Bill


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Robert Charette
Spotsylvania, Va.
Willie D. Jones
New York City
Load More