Risk Factor iconRisk Factor

Nest Labs Suspends Sale of Smoke and Carbon Monoxide Detector until Software Fixed

IT Hiccups of the Week

Last week saw several interesting IT-related bugs, ooftas and malfunctions being reported. We start this week’s edition of IT Hiccups with a potentially serious software issue in the Nest Labs’ Wi-Fi connected smoke and carbon monoxide detector.

Nest Labs announced on its website last Thursday that it was going to suspend sales of its Nest Protect: Smoke + Carbon Monoxide alarm to fix a “unique combination of circumstances that caused us to question whether the Nest Wave (a feature that enables you to turn off your alarm with a wave of the hand) could be unintentionally activated. This could delay an alarm going off if there was a real fire.” In addition to suspending sales of the product, the company is remotely deactivating (where possible) the Next Wave feature until a software fix is found. The company notes that even with the Wave feature disabled, the detector will work.

According to the New York Times, Nest Labs discovered that during laboratory testing the alarm’s software algorithm could misinterpret movement near the detector as being an intentional “wave” command and shut the unit off. The feature, the ability to easily turn off the smoke detector with a wave of the hand from a distance of 0.6 to 2.4 meters away when there was a false alarm, is a major selling point of Nest Protect. But in its current instantiation, the feature may be a life-threatening bug.

Nest Labs stated that it may take two to three months before a software fix is designed, tested and approved by safety agencies in the U.S., Canada, and the U.K. Once complete, the company will send out a software update to reactivate the Wave feature. Nest Labs also emphasized that no one has reported that a detector failed to sound an alarm in event of a fire.

Nest Labs, which is better known for its elegantly-designed Learning Thermostat, was purchased earlier this year by Google for $3.2 billion in cash. There have also been reports of a small number of Nest thermostat customers having wiring-related problems.

The issue with Nest Protect nicely highlights, as IEEE Spectrum’s Automaton editor Erico Guizzo noted to me, the paradox of how software can make things more complicated and worse (i.e., a flawed hand-waving feature), as well as how software can make things more complicated and better (i.e., a remote software update to fix the bug).

The Financial Times of London reemphasized Guizzo’s point in its article on the Nest Protect software bug. It noted that customers and companies can expect these types of incidents to occur more often as “smart” technology is added to simple devices in the emerging rush to embrace the “Internet of Things.” The article also noted that companies’ customer service better be ready to respond when a problem is found. The FT noted, for instance, that just in January, Nest Lab was “forced to increase its customer-support hours after several customers complained about persistent false alarms” that were unrelated to the current software issue.

Eircom Says We’re “Embarrassed” Over the Error, But We Still Want Our Money Now

There are a lot of customers who are angry with Eircom, the largest telecom provider in the Republic of Ireland. Apparently, some 30 000 of its customers will be receiving bills of as much as €500 in their next billing cycle. Eircom is attempting to recoup funds it previously failed to collect because of what it is calling a “system error,” the company told Dublin’s Herald newspaper. That error relates to its implementation of the new Single European Payments Area (SEPA)—a project that is meant to harmonize how retail payments are made and processed across the 34 European countries. The affected customers, which the Dublin Herald describes as “a mix of phone and internet users, mainly those with bundle packages, who make monthly direct debit payments...did not have some or all of their monthly direct debit payments taken from their bank accounts for phone, broadband and TV service since January.” This was despite the fact that customers’ bills showed that the bills had indeed been paid.

Eircom’s director of corporate affairs said that the incident was not only “regrettable,” but that “it’s embarrassing and we're very sorry that it's happened.”

However, Eircom wasn’t too embarrassed or sorry to insist that although it plans to reimburse the failed direct debit fee charge of €18.45, those 30 000 customers would still have to pay all the monies owed the telecom immediately.

Eircon’s hard stand has not endeared it with Ireland’s telecom regulator, which publicly stated that it is demanding a report into the billing error and how it happened. The regulator also wants to know why Eircom failed to inform it that a billing problem had occurred.

In Other News…

New Illinois Teacher Licensing System a Buggy Mess

Western Digital Finally Fixes Multi-day Cloud Outage

One of Melbourne’s Largest Hospital Suffering Booking System Chaos

Georgia Seeks Food Stamp System Fixes

Kansas Municipal Court Mistakenly Mails Arrest Warrant Notices

Maine Homeless Man “Given” $37 000 in ATM Glitch

UK Supermarket Chain Asda Charges £50 for Cabbage

Russia’s Glasnost GPS Experiences Problems

Software Problems Hit French Soldiers’ Pay

Springdale, Arkansas, Residents Receive Tsunami Warning

Nissan Recalls Nearly 1 Million Cars for Air Bag Software Fix

IT Hiccups of the WeekLast week saw a marked increase in the number and types of IT-related errors, bugs and malfunctions being reported.  However, as we have the past few weeks, we again begin this week’s IT Hiccups edition with an auto-related IT issue.

According to the Washington Post, 989 701 Nissan and Infiniti 2013 and 2014 model year vehicles are being recalled to fix a problem in the software that controls air bag deployment for the front seat passenger. They include: 544 000 Altima sedans; 29 000 Leaf electric vehicles; 124 000 Pathfinder SUVs; 183 000 Sentra compact cars; 6700 NV2000 taxis; and 104 000 Infiniti JX35, Q50 and QX60 vehicles.

The Post states that, “Unfortunately, the software installed on the vehicles…may incorrectly determine that the passenger seat is empty when it is, in fact, occupied. If that were to happen, and if the vehicle were subsequently involved in an accident, the passenger‐seat airbags would fail to deploy, increasing the possibility of injury or death.”

A New York Times article says that, “The automaker blamed the sensitivity of the software calibration, particularly when ‘a combination of factors such as high engine vibration at idle when the seat is initially empty and then becomes occupied’ or an ‘unusual’ seating posture are factors.”

Nissan indicates that it was aware of three such accidents, although no fatalities were reported, as a result of the collisions where airbags failed to activate. The company is working on a patch for the software, which should be available in the next few weeks.

In another software-related automobile recall, General Motors is recalling 656 of its 2014 Cadillac ELR vehicles in order to recalibrate software in the electronic control brake module that is part of its electronic stability control system.  GM also announced last week that it was recalling another 824 000 vehicles for issues with defective ignition switches I discussed a few weeks ago. It also announced recalls of another 662 000 other vehicles for various other mechanical issues. GM has now recalled over 4.8 million vehicles since the beginning of year. This week GM and the National Highway Transport Safety Administration will face Congressional hearings on their roles in regard to the delayed ignition switch recall.

Iowa Mayor Unhappy About Disclosure of Crime Reporting Software Problem

Matt Walsh, the mayor of Council Bluffs, Iowa, is reportedly very unhappy that the state government was told by a local county board member that the new software system the local police use to record crime statistics was flawed. According to a World‐Herald News Service article, the Council Bluffs police department began using software last year provided by the Iowa Department of Transportation to enter local crime statistics. However, a software programming error “upgraded” many of the crimes entered. For instance, a simple assault reported by the police instead got changed by the software into a more serious aggravated assault.

The problem with the new software helped explain why Council Bluffs was recently listed as No. 56 on the most 100 most dangerous places to live in the U.S., the World-Herald said. Now, one would think the city’s mayor would be happy about the flaw being discovered and his city’s reputation as a criminal haven being rehabilitated. Yet, the Des Moines Register states that Mayor Welsh was incensed. Why? Well, the state provides crime enforcement grant funding to the city based on crime statistics, and the mayor is now worried the city may have to return some of the state grant money it received.

The Register quotes the mayor as saying, “What kind of individual runs to the state and tattles? …This money is to fight crime.” The mayor also claims that since the police department originally reported its crimes correctly, it isn’t the city’s fault that the software system provided by the state screwed up, so the money it previously received from the state is rightfully the city’s.

Hmm… maybe the teenager who found $31 000 mistakenly deposited in his account by the First Citizens Bank in Hull, Georgia a few weeks ago and who decided to spend it should have used the same ethical reasoning, instead of lying and pretending that the money was deposited on purpose as part his share of an inheritance from his grandmother’s estate.

Maryland Throws in the Towel on its Affordable Care Act Website

Today is the last day to sign up for health insurance under the Affordable Care Act (with some exceptions) until the next open enrollment season. To say the least, the introduction of the ACA on 1 October 2013 has been interesting to watch from an IT system risk mismanagement perspective, both at the federal and state level. Even now, the federal site still is reporting access issues.

However, five states— Maryland, Massachusetts, Nevada, Oregon and Vermont— have given star performances in how not to create a state ACA website and supporting infrastructure systems. An Associated Press article provides a decent overview of the health insurance exchange implementation problems encountered in each state, as does another story at VTDigger.org that examines the issues confronting the Massachusetts and Vermont exchanges in greater depth.

However, for sheer incompetence, Maryland’s ACA implementation debacle really stands out (although Oregon's comes in a close second). After spending at least $125.5 million, Maryland has decided to basically abandon its exchange, the Washington Post reported. The state reportedly will be using the exchange system Connecticut has developed and is eager to sell to other states, which seems to work better than most.

Earlier this month, the U.S. Department of Health and Human Services launched an investigation into what went wrong with the Maryland's health insurance exchange. However, it is unlikely that any results will be published before the state primary elections in June. The reason is that Democratic Lt. Gov. Anthony Brown, who once proudly proclaimed that he was in charge of the Maryland ACA implementation, is running for governor, and I doubt the federal government wants to be seen as possibly interfering with the election. To say that Brown was asleep at the health exchange switch would be to assume, given his role in the unfolding debacle and his very recent claims that the exchange implementation is a “success,” that he knew where the switch was in the first place.

In Other News…

FAA Instructs Boeing to Fix Critical 747-8 Software Flaw

System Issues Delay Bombardier Learjet 85 First Flight

Illinois Demands DUI Offenders’ Pay Fines Years after Computer Error

Soyuz Spacecraft Suffers Software Issue on Trip to Space Station

Northern Ireland Hospital Staff hit by £400k Payroll Shortfall

Hundreds of Irish Motorists Receive Fines after M50 Toll Glitch

Price Glitch Charges £450 for Loaf of Bread in Wolverhampton, England

Denver-based Public Service Credit Union Experiences Four Days of Computer Problems

Allied Irish Bank Customers Double‐charged


Software Testing Problems Continue to Plague F-35 Joint Strike Fighter Program

The U.S. General Accountability Office (GAO) earlier this week released its fifth annual report on the state of the F-35 Lightning II, aka the Joint Strike Fighter (JSF), aka the “most costly and ambitious” acquisition program ever. What the GAO found was foretold by a report earlier this year by the Department of Defense’s Director of Operational Test and Evaluation. The upshot: the F-35 operational and support software development continues to be the major obstacle to the program's success.

In addition, the GAO report states that the projected cost of acquiring the planned 2443 F-35 aircraft (which comes in three flavors) threatens to consume some 20- to 25 percent of annual defense program acquisition funds for the next twenty years or so. The GAO doesn’t explicitly say so, but the operations and maintenance costs of the program—currently estimated to be between $800 billion and $1 trillion dollars or more over the next 50 years—will also consume a significant chunk of DoD’s annual weapon-system related O&M budget as well.

The GAO report states that, “Challenges in development and testing of mission systems software continued through 2013, due largely to delays in software delivery, limited capability in the software when delivered, and the need to fix problems and retest multiple software versions.”  Further, the GAO notes that the F-35 program continues to “encounter slower than expected progress in developing the Autonomic Logistics Information System (ALIS),” which is the F-35’s advanced integrated maintenance and support system (pdf). In the latter case, Lt. Gen. Christopher Bogdan, the F-35 Program Executive Officer, conceded last month that the ALIS system was “way behind” where it should be and was “in catch-up mode.” This, the GAO indicates, was apparently at least partly because of a lack of testing facilities that remains a problem years after ALIS development began.

The GAO notes that as a result of the on-going software problems with the aircraft's mission and support systems, F-35 program officials and contractors alike believe that software development will continue to be the F-35 program’s “most significant risk area.”

Software-testing related issues involving the development and fielding mission systems were the main thrust of this year’s GAO report.  The F-35, you may recall, is delivering its mission capabilities in a series of  incremental “software blocks,” designated as Block 1A/B, Block 2A, Block 2B, Block 3i, and Block 3F.  Each block builds on the mission capability developed in the preceding block. As described by the report, “Blocks 1 and 2A provide training capabilities and are essentially complete, with some final development and testing still underway. Blocks 2B and 3i provide initial warfighting capabilities and are needed by the Marine Corps and Air Force, respectively, to achieve initial operational capability. Block 3F is expected to provide the full suite of warfighting capabilities, and is the block the Navy expects to have to achieve its initial operational capability.” According to Flightglobal, a software Block 4 is being planned as an eventual mission capability upgrade for which development will begin late this year or more likely early next.

However, the GAO report states that, “Developmental testing of Block 2B software is behind schedule and will likely delay the delivery of expected warfighting capabilities,” required by the Marines for their variant of the F-35  (the F-35B) that is scheduled for delivery by July 2015. As of January of this year, “the program planned to have verified the functionality of 27 percent of the software’s capability on-board the aircraft, but had only been able to verify 13 percent,” says the GAO report. In more than a bit of an understatement, the GAO says that, “This leaves a significant amount of work to be done before October 2014, which is when the program expects to complete developmental flight testing of this software block.”

The GAO notes—and seems to agree with—the Operational Test and Evaluation Director's view that a more realistic estimate for when Block 2B’s software functional verification will be completed is sometime closer to November 2015. The report also notes that such a delay would create a knock-on effect to the subsequent F-35 software blocks as well, increasing the cost of the acquisition, not to mention delaying the planned initial operational capability (IOC) of the aircraft (2016 for the Air Force F-35As, and 2018 for the Navy F-35Cs).

Yet, despite everything it saw, the GAO indicates that the F-35 program office and contractors, and especially the Marines, seem to be all whistling along to Bobby McFerrin’s song, “Don’t Worry, Be Happy.” The GAO states that, “Program and contractor officials have stated that while they recognize that the program faces software risks, they still expect to deliver all of the planned F-35 software capabilities to the military services as currently scheduled.” Why do they think so? Why, they are now going to introduce new approaches to gain “testing efficiency.” The plan: mainly by using “test results from one F-35 variant to close out test points for the other two variants in instances in which the variants have common functions.”  However, Bloomberg News quoted a recent RAND assessment of the F-35 program as stating that, “As of this writing, it is not clear how common the mission systems, avionics, software and engine will be among the three service variants,” so how much efficiency will in reality be gained remains to be seen.

In fact, in testimony before Congress yesterday, Lt. Gen Bogdan was reported by Reuters as saying he was “pretty confident” that Block 2B software would be delivered within 30 days of its current target date to allow the Marines to get to initial operational capability by July next year, as the software is “80 percent complete.” However, Bogdan also indicated that he was not as confident that even ten Marine F-35Bs would be IOC ready given that most of the 40-plus Marine F-35Bs will require some 96 engineering modifications by then.

Lt. Gen. Bogdan also disclosed at the hearing that “Block 3F [software] is dependent upon the timely release of Block 2B and 3I, and at present, 3F is tracking approximately four to six months late without taking steps to mitigate that delay.”

One does hope the program’s Block 2B software testing efficiency strategy is successful, since the GAO indicates the F-35 is scheduled to undergo operational testing in June of next year, “to determine that the aircraft variants can effectively perform their intended missions in a realistic threat environment.” If the new testing strategy is not successful, the GAO's view is that the cost of the F-35 acquisition and its future sustainment costs will just keep on escalating.

In response to the GAO report, the F-35 program office has agreed to deliver to Congress an assessment of the “risks of delivering required capabilities within the stated initial operational capability windows for each military service.” The GAO wants that assessment completed and the risks reported by July 2015, but the program hasn’t committed itself to any specific timetable to deliver a detailed assessment. As a Marine Corps Times article seems to suggest, future disclosures on the part of the program office concerning the risks of possible program schedule slips or cost increases will more than likely happen only in piecemeal fashion and by accident.

Of course, even if the F-35 Block 2B software is late—or one or more of the other software blocks are delayed for that matter—it really presages very little change in the general future direction of the program. Why? Well, in a CBS News 60 Minutes interview in February, Lt. Gen. Bogdan was asked, “Has the F-35 program passed the point of no return?” to which he replied, “I don't see any scenario where we're walking back away from this program.”

The GAO is officially scheduled to conduct one more annual review of the F-35 acquisition. The only purpose of it that I can see is merely to warn current and future U.S. taxpayers, many who are not yet born, how much more money they will have to shell out for the next 50 years or more.

Photo: U.S. Department of Defense

U.S. Fines Toyota $1.2 Billion but Defers Criminal Prosecution Over Vehicle Safety Deceit

IT Hiccups of the Week Last week was another slow week concerning the number of IT-related malfunctions, bugs and kinks reported. So we decided to devote this week’s IT Hiccups to the U.S. Department of Justice's (DoJ's) controversial announcement last week of a massive fine and deferred criminal prosecution as punishment for Toyota having misled the National Highway Traffic Safety Administration (NHTSA) and the public about two safety issues. The deception was part of the automaker's attempt to beat back claims that “sticky pedal” and “floor mat entrapment” in its vehicles could lead to sudden unintended acceleration (SUA). We have been following the Toyota SUA saga in the Risk Factor for several years.

The DoJ announcement states that Toyota “defrauded consumers in the fall of 2009 and early 2010 by issuing misleading statements about safety issues in Toyota and Lexus vehicles.” In addition, Toyota “misled U.S. consumers by concealing and making deceptive statements about two safety issues affecting its vehicles, each of which caused a type of unintended acceleration.”  

Toyota admitted to the DoJ’s criminal charges in a statement of fact (pdf) which read in part that:

“Contrary to public statements that Toyota made in late 2009 saying it had ‘addressed’ the ‘root cause’ of unintended acceleration through a limited safety recall addressing floor mat entrapment, Toyota had actually conducted internal tests revealing that certain of its unrecalled vehicles bore design features rendering them just as susceptible to floor mat entrapment as some of the recalled vehicles. And only weeks before these statements were made, individuals within Toyota had taken steps to hide from its regulator another type of unintended acceleration in its vehicles, separate and apart from floor mat entrapment: the sticky pedal problem.”

The statement of fact also highlighted a January 2010 report of an apparent “smoking gun” discussion following a meeting between Toyota and NHTSA in which “one Toyota employee was said to exclaim, ‘Idiots! Someone will go to jail if lies are repeatedly told. I can’t support this.’”

The agreement between the DoJ and Toyota requires Toyota “to pay a $1.2 billion financial penalty—the largest penalty of its kind ever imposed on an automotive company, and imposes on Toyota an independent monitor to review and assess policies, practices and procedures relating to Toyota’s safety-related public statements and reporting obligations.”  If Toyota abides by the terms of the agreement and continues to cooperate with the U.S. government for the next three years, the criminal prosecution will be dismissed.  The Toyota fine is not tax-deductible, in case you are wondering.

U.S. District Judge William H. Pauley, who signed off on the DoJ-Toyota agreement last week, was quoted in a Wall Street Journal report as saying that Toyota’s activities “painted a reprehensible picture of corporate misconduct.” In addition, the judge said that, “I sincerely hope that this is not the end but rather the beginning to seek to hold those individuals responsible for making these decisions accountable.”

Toyota declined to make a comment in regard to the judge’s comments, preferring to let its published statement about the agreement say it all:

“At the time of these recalls, we took full responsibility for any concerns our actions may have caused customers, and we rededicated ourselves to earning their trust. In the more than four years since these recalls, we have gone back to basics at Toyota to put our customers first. … Importantly, Toyota addressed the sticky pedal and floor mat entrapment issues with effective and durable solutions, and we stand behind the safety and quality of our vehicles.”

You will no doubt notice that there is nothing in this settlement that addresses the other high-profile sudden unintended acceleration issue: a supposed hardware and/or software defect in Toyota’s electronic throttle system. As I noted at the end of last year, Toyota and the myriad lawyers suing it over that allegation are still in court-ordered negotiations, even as Toyota continues to maintain that there isn’t any electronics/software-related SUA defect but only human error involved.

Several commentators at the Wall Street Journal and Washington Times have called the agreement an “unjust” “government shake-down” of Toyota. Their claim is that Toyota did nothing wrong other than perhaps, as one commentator put it, a “few incidences” where its employees “dithered about whether to report data to [NHTSA] right away and took pains to present facts to the public in the most flattering light.” In other words, Toyota didn’t violate any important safety procedures in its recall approach, just government paperwork requirements, despite Judge Pauley’s comments to the contrary. Toyota, they go on to maintain, only agreed to the $1.2 billion fine and deferred prosecution to put an end to what they see as the totally media-inspired public frenzy and besmirchment of Toyota's reputation.  

These commentators see GM as the next likely “victim” of government overreach when the Justice Department assesses penalties related to the case of the automaker's belated 3-million-vehicle ignition-switch recall. They point to the warning that Attorney General Eric Holder made in the news conference announcing the department's agreement with Toyota: “Other car companies should not repeat Toyota’s mistake: A recall may damage a company’s reputation, but deceiving your customers makes that damage far more lasting.”

Of course, the “Toyota got railroaded” crowd don’t say what would rise to fraudulent and deceptive practices on Toyota’s part, or why GM should receive a pass from the DoJ, especially in light of GM’s own admissions. Their lines of reasoning might be more convincing if they had.

In Other News…

Electronic Voting Problems Concern Ontario Town

Computer Issue Slows Illinois County Vote Count

Louisiana Experiences Income Tax Refund Programming Glitch

Error Duplicates and Delays Fidelity Online Trading Orders

Google Hangouts, Talk, and Sheets Crash

Caltrain Suffers Dispatch Problems

Manila Metro Rail Transit Experiences Two Communication Problems in Three Days

NASA's Mars Reconnaissance Orbiter Fixes Glitch

Auckland District Health Board Hospitals Have Computer Problems Again

Northern Ireland Water Getting Refunds for Billing Errors

CMS Scrambles to Fix Healthcare.gov Programming Error

GM CEO: “We Admit It. Somebody Messed Up”

It’s late in the evening. The sun has just dipped below the horizon. You’re cruising along the local interstate at highway speed, listening to your favorite tune on the car stereo. Just as you’re about to go around a curve, the engine loses power. Making your state of panic worse is the fact that the steering wheel seems to have seized up. That’s because the electric motor that provides the power-assist steering has turned off. You’re not sure you have the strength to safely navigate the curve, so you tap the brake to slow down. But after the second tap, the engine-supplied vacuum that provides power-assist braking is gone. Technically, the brakes still work, but bringing the vehicle to a halt before you leave your lane and collide with another car—or run off the road completely—will require a stomp instead of a tap.

That set of circumstances actually occurred many times in GM vehicles. How often is still unknown. Millions of cars were equipped with a part that didn’t provide enough resistance to, say, a key ring swinging and rotating the car key so that the ignition was suddenly turned from the on (run) position to the off (accessory) position. There’s nothing to prevent that turn from happening except the tension provided by the spring in the part, known as a detent plunger. A bigger part with a longer, stronger spring was included beginning in 2008. But it replaces one that has been in vehicles since 2003.

It seems executives at General Motors, manufacturer of the Chevy Cobalt, Chevy HHR, Pontiac G5, Pontiac Pursuit, Pontiac Solstice, the Saturn Ion, and the Saturn Sky—vehicles that have been linked to a dozen deaths caused by such sudden shut-offs—knew about the problem for more than a decade but failed to act.

Yesterday, Mary Barra, GM’s new CEO, said that the company will recall 1.5 million cars that include the part that has since been redesigned. The announcement comes just weeks after a February announcement that it planned to recall 1.6 million of the cars with the ignition switch problem.

Channeling her inner politician, Barra told GM employees in a video message that was posted online that, "Something went wrong with our process in this instance, and terrible things happened." Yes. Mistakes were made. But some all-important pronouns were missing from the message. Who made the mistakes? As federal regulators look into whether the ignition problem caused several hundred deaths instead of just a dozen and whether the problem involves more GM vehicles than those currently recalled, GM will soon find itself in the U.S. Congress’ crosshairs. The House of Representatives and the Senate have already said they want to schedule separate hearings to discuss possible criminal penalties related to the automaker so belatedly disclosing the issue

Barra said the company is changing how it handles defect investigations and recalls, and has done her best to distance GM’s current set of executives from the inevitable fallout. The damage control has gone so far as to include the creation of a new executive position dedicated to vehicle safety. But she and her colleagues will still have to provide reasonable answers to a simple question: If the ignition recall would have sent drivers in for a quick fix with a relatively cheap part—or GM could have avoided it altogether by putting the reengineered part on the assembly line—what was the calculus behind the initial decision to ignore the problem and subsequent inaction even after hundreds of complaints about sudden engine shut-offs?

“I don’t know how long you’ve been covering this business, but I’ve been in it for 50 years,” says Michael E. Bresnock, head of Transportation Technology, Inc., a Marietta, Ga.-based firm that does accident investigations and technical analysis of vehicle faults and failures. “A difference of a dollar will determine whether a car is going to roll down the assembly line.”

Another vehicle expert who actually worked for GM for decades and agreed to speak on the condition of anonymity, tried to imagine what the company’s executives were thinking when they repeatedly declined to make the fix. “I guess they told themselves that it is the operator’s responsibility to ensure that he or she doesn’t sit in a way that causes them to bump the ignition switch or to avoid putting too many keys on a keychain.” Asked whether they could still adhere to that reasoning knowing that most drivers were completely unaware that this could happen—and had happened, with deadly results—he imagined the executives taking morbid comfort in the fact that when a car suddenly shuts off, “You’ll still have brakes and steering; you just lose power assist.”

GM is going to need some serious legal power assist in order to step out of the hot water it’s gotten itself into.

Chrysler: "I Brake for Electronics Recalls"

IT Hiccups of the Week

Last week was an extremely slow week with regard to the number of IT-related problems, snafus, and bugs reported. So we decided to dedicate this week’s IT Hiccups to the increasingly common occurrence of an automotive recall to fix a vehicle’s electronics or software.

As reported by Reuters, Chrysler announced last week that it was recalling a total of 25 250 Jeep Grand Cherokee and Dodge Durango SUVs from model years 2012 and 2013 to improve anti-lock brake pedal feel “during certain aggressive braking maneuvers.”  According to the Chrysler, a supplier raised an issue with a part that supports Ready Alert Braking, a safety system that primes a car’s brakes in anticipation of a driver making an emergency stop.

Chrysler literature says that Ready Alert Braking “anticipates situations when the driver may initiate an emergency brake stop and uses the electronic stability control (ESC) pump to set brake pads against rotors in order to decrease the time required for full brake application.”

The LA Times reports that while the brake worked as designed and was in compliance with safety requirements, customers were complaining about an “odd feeling in the brake pedal” that was traced to factory settings that overly restricted the flow of brake fluid. A software update was being issued, Chrysler stated, “to improve the flow and restore appropriate pedal feel.”

Chrysler also announced that it is recalling 19 500 Fiat 500L cars from the 2014 model year to fix an issue affecting their automated dual‐clutch transmission. The company says that car owners complained that the vehicles would not shift out of park quickly or readily into the gear selected by the driver. Apparently, the problem is tied to a specific microcontroller component in the transmission that fails to operate correctly in extreme temperatures. A software update fixes the problem in all but about 200 vehicles; in those vehicles, a shift-module replacement may be necessary to ensure hardware-software compatibility, the company said.

Neither of the recalls were in response to reported accidents or injuries.

There was word of another vehicle software issue last week, but it affected only one rather special vehicle. According to ESPN, an engine software programming problem affected Sebastian Vettel's Red Bull Formula 1 racing car during qualifying for yesterday’s Australian Grand Prix. The team said that the problem “meant he was down on power with extremely poor drivability,” and was the reason Vettel only qualified for 12th place on the starting grid, his poorest starting position since 2012.

Apparently, the software issue wasn’t completely fixed before the starting flag was waved; Vettel—who had won nine consecutive F1 races—dejectedly retired from the race after only a few laps when he continued to sense a lack of engine power.

Last week also saw a story published by the London Telegraph claiming that “increasingly complicated electronics in cars have prompted a surge in expensive breakdowns.” Based on data gleaned from 50 000 insurance policies from Warranty Direct for vehicles three years or older, the Telegraph says “the number of electrical faults in cars has risen by two thirds over the past five years, with repair bills rising by one third.” On average, nearly one in four drivers will experience an electronics-related problem over the course of a year, compared with only one in ten drivers five years ago, the newspaper said. The average cost for fixing a problem has also climbed from £221 to £291 over the same period.

The Telegraph data indicate that the Subaru had the fewest electronic issues while Renault had the most. However, the next least reliable manufacturers listed were Bentley and Porsche, apparently because their vehicles are typically fitted with the latest electronic systems. The average repair cost for an electronic problem was reported as £670 in a Bentley and £757 in a Porsche, but I suspect the owners of these luxury cars are far less concerned about the cost of repairs as much as their frequency.

Finally, the GM recall of 1.6 million Chevrolet Cobalt and Pontiac G5 vehicles for an ignition switch problem that has resulted in at least a dozen deaths continues to make headlines.  Regulators are now investigating whether the issue has in fact resulted in several hundred deaths, and there is concern that the problem involves more GM vehicles than those currently recalled.  The U.S. Congress is looking to schedule hearings into the recall while the U.S. Department of Justice is investigating whether GM is criminally liable for not disclosing the ignition switch problem earlier.  No doubt, pointed questions will also be raised during the congressional hearings into the excuses the U.S. National Highway Transportation Administration is giving for failing to launch a full scale investigation even after receiving 260 complaints of vehicles suddenly shutting down.

In Other News …

California DMV Suffers Computer Problems Two Consecutive Days

Food Stamp “Glitch” Affects 27 000 Floridians

Mars Orbiter Goes Into Sleep Mode Again as NASA Readies Fix

UK House of Parliament Experiencing Frustrating IT Outages

London Transport Refunds £11 000 after Oyster Card Bus Overpayment Error

Twitter Crashes for Second Time in Two Weeks Due to Software Update Problem

Healthcare.gov Problems Stymie 84 000 Medicaid Requests in Michigan

College Exam Papers Lost Due to Software Error

Tesco Computer Identifies Teaspoons as Age Restricted Purchase Item

Photo: Chrysler Group

UK Coroner Fingers NHS Computer System in Toddler’s Death

IT Hiccups of the WeekThe number of IT-related errors, ooftas, and deficiencies reported last week reverted back towards the mean from the previous week's overabundance. We start off this edition of IT Hiccups with a sad case of a child’s death in the UK. The tragedy is being attributed in part to the past effort to fully computerize the UK’s National Health Service.

According to the Bristol Post, a coroner in charge of the inquest into the death of Samuel Starr, aged three, indicated in a narrative verdict that, “Due to the failure of the [Royal United] hospital's outpatient booking system, there was a five month delay in Samuel being seen and receiving necessary treatment.”  It is very rare for a coroner to criticize a hospital IT system so directly.

Samuel Starr was born with “complex congenital heart disease” in 2009. His parents were told at the time of his birth that Samuel would need several operations before he was five, and in fact, Samuel underwent an operation when he was nine months old. The Post reported that he made a good recovery, and was due to have regular checkups and further treatment at the Pediatric Cardiac Clinic at the Royal United Hospital (RUH) in Bath. Samuel received a checkup in October 2010 and one in April 2011, at which time his parents were told by his doctor to schedule another in about nine months for a more extensive examination of his heart.

However, a new electronic health record system, called Cerner Millennium, was being installed in 2011 at the hospital as part of the NHS’s National Program for IT (NPfIT), which was shortly thereafter cancelled. Though the main program was cancelled, certain elements, such as its national Choose and Book system for patient scheduling, remained. (Hospitals, like at Royal United, that were already installing electronic health record systems were given the go-ahead to proceed if they wished).

According to the Daily Mirror, “glitches” in the Royal United patient booking system caused Samuel not to receive his scheduled appointment with heart specialists as required, despite pleas for an appointment by his parents and a primary care specialist. The Mirror stated that medical secretary for Samuel's doctor insisted that she had taken down the appointment details and forwarded them on to a dedicated appointments team, but they were apparently not logged in. “While Samuel's medical records had been created on the new Millennium computer program, no appointments had been transferred across [from the old scheduling system],” the Mirror explained.

By the time Samuel was eventually seen, his heart condition had taken a turn for the worse, and he required immediate surgery. Unfortunately, the child died after enduring a series of cardiac arrests a few weeks after his surgery.

While it is not certain that Samuel would have lived if he had been seen earlier, the inquest did highlight that he was not the only patient who didn't receive timely medical care due to problems with that same hospital’s appointments system. BBC News reported that, “Minutes of board meetings in RUH a year before Samuel's death show the hospital was fully aware of the problems with their new computer system. They reveal ‘there were significant issues with...data that had not been migrated which affected...long-term follow-up appointments.’” Some 63 overdue pediatric cardiac appointments in all were uncovered, “with some taking nearly two years to discover,” the BBC story stated.

The Royal United Hospital has since corrected its booking/EHR system problems, and has apologized to Samuel Starr’s family. However, the episode does lead one to wonder why hospital administrators didn’t work harder to look for missing patient appointments after the issue was identified not only at their hospital, but also at other NHS hospitals implementing the same booking and EHR system during the same time frame.

Votes Go Missing for Two Years

An AP story last week reported that 3971 early votes cast in Warrick County, Indiana, during the 2012 general election went uncounted until recently “because of an error by an electronic voting machine technician.”

The AP story said that the missing votes were discovered “by a Democratic precinct leader who recently was cross referencing precinct summary reports with a state voter history report.” The precinct leader was surprised to find out that the summary reports of tallied machine and paper votes for the county had a discrepancy of more than 3700 votes.

Further investigation revealed that “Indianapolis‐based MicroVote General Corp., which services the county's electronic voting machines, found that one of their technicians incorrectly uploaded early votes,” the AP story said.

Luckily, the missing votes would not have changed the outcome of the election. There was no word as to what would have happened if they had.

In a similar vote-delay incident, the Port Arthur News in Texas reported that “a misplaced flash drive and, later, a software glitch, delayed election results in Jefferson County by more than four hours on Tuesday.” The flash drive, the News stated, was found in a locked-up early voting station polling machine, while “a glitch in the software deleted information containing the number of registered voters in each precinct” and necessitated manually inputting the precinct voter registration numbers.  

Delhi Police “Lost” Password for Eight Years

The Indian Express reported last week that some 667 complaints by the public regarding the conduct of the Delhi Police that were forwarded by the Central Vigilance Commission (CVC)—which investigates the complaints and either addresses them itself or sends them via an online portal to the police to deal with—have been awaiting resolution for the past eight years.

The reason? The Express stated that the “Delhi Police didn’t know the password to access the portal or how to operate it, a lapse that went undetected since 2006.”

It wasn’t until January of this year that two high ranking Delhi police officers were given the needed training by the CVC to access and operate the portal.

The Express reported that the CVC hosts meetings every year with government departments to review the complaints it receives about those agencies. However, since 2006, “the CVC had got no feedback on complaints pending with the police.” For an unexplained reason, the CVC finally became curious over the lack of feedback early this year, and discovered the reason behind it.

Delhi police officials indicate that they are now addressing the backlog of complaints against the department.

Coroner Says NHS Computer System Partly to Blame for Toddler’s Death

Coroner Blames “Failure” of NHS Computer System for Boy's Death

Three-year-old Dies after New NHS Computer System Delays Heart Treatment Appointment

Hospital Booking System “May Have Contributed to Death”

Coroner Criticizes Hospital’s IT System After Boy’s Death

Votes Found Two Years after Election

Jefferson County Finds 3700 Untallied Votes from 2012 Election

Software Glitch Delays Port Arthur Election Count

Delhi Police Finally Address Complaints Against It, Some From Eight Years Ago

667 Vigilance Complaints Pile Up as Delhi Police Claim Password Ignorance

Delhi Police Complaints Mount as Police Lose Password for Eight Years

In Other News…

RTÉ Primetime Team Oblivious to 13 Minute Technical Glitch

Software Failure Causes Power Outage at Notre Dame

Blackberry Experiences 5-Hour Outage

Computer Problem Shuts Down California DMV License Applications

Twitter Mass User Password Reset Blamed on Error

Despite Pledges, Npower Still Refusing to Fix UK Utility Billing Errors

Feds and Florida Still at Odds Over New Unemployment System

Photo: Alamy

Julian Assange's Virtual Address at South By Southwest

In a wide-ranging talk Saturday, Wikileaks founder Julian Assange said that despite the efforts of his organization and the revelations of Edward Snowden, "We are all actually living in a world we don't understand."  Assange, who is still confined to the Ecuadorian Embassy in London because he faces arrest on espionage charges in the United States and Great Britain, addressed a crowd of about a thousand at the South By Southwest Interactive conference via a live-video Skype connection.

His and Snowden's revelations showed that "the true nature of human institutions" such as the national-security, defense, and diplomatic agencies of major governments and their contractors, "are all obscured by fog. Every now and then, there is a clearing of the fog, when there is one of these disclosures," he added. He identified his bete noir as a "fluid, postmodern amalgam of agencies and contractors," including the National Security Agency in the United States, as well as the U.S. Department of Defense, the Central Intelligence Agency, the State Department, and Britain's GHCQ.

After asserting that the Western bloc led by the United States was responsible for 75 to 80 percent of the world's aggregate expenditures, Assange assailed what he said was a new kind of totalitarianism. "We're moving into a new totalitarian world," he said. "Not in the sense of Stalin or Pol Pot...but in the sense that anyone can be surveilled.

"The ability to surveil everyone on the planet is almost there," he claimed, "and arguably will be there in a few years. And to store all the information."

Because of the efforts of Wikileaks and a few individuals, he continued, public perception has grown that the Internet, "the greatest tool of human emancipation, had been coopted" by agencies using it to gather information surreptitiously to further their agendas. He referred to the use of the Internet by powerful government agencies as "a militarization of our civilian space. A military intrusion into our civilian space."

But he also saw a brighter side to the Internet's recent evolution. The Internet had been transformed over the last four years, he said, because of revelations such as his and Snowden's and also by events such as the "Arab Spring." The Internet "four years ago was a politically apathetic space," he said. But the conflicts involving him and Snowden against the United States and other powers have played out in public, "and everyone could see what was going on," he declared. "The Internet became a political space, and that is an important development."

He further noted that the conflict has made de facto refugees not only of him—he has been confined to the Ecuadorian Embassy in London for 650 days—and Snowden, who is in Russia, but also of several others. He cited three journalists, including Glenn Greenwald, who brought the Snowden revelations to light for the British newspaper The Guardian, and is now living in Brazil. The others were American Laura Poitras, who reported on Snowden's revelations for the Washington Post and Der Spegel, and British citizen Sarah Harrison, who helped Snowden get from Hong Kong to Russia. Assange also mentioned the American Jacob Applebaum, who has been identified as a hacker and Wikileaks supporter. Poitras, Applebaum, and Harrison are all now living in Berlin.

"National security reporters are a new kind of refugee," Assange asserted. But at least they are "not in a situation where they have to be terrified all the time," he went on. "I see this as quite a positive phenomenon: Where, once, people would have been completely crushed, they can use these basic tenets and rights to confine nations, and in restraint of the powerful countries.

"We are all part of what we would traditionally call the State," Assange said. "So we have no choice but to attempt to manage the behavior of the State."

At one point, in defending Wikileaks, Assange seemed to compare himself to Robin Hood. Major government security agencies are "stealing information from all of us," he began. "Knowledge is power. Wikileaks specializes in going in the opposite direction. Reversing the process--taking knowledge about how this process works and putting it back in the public record. And that empowers us."

But Assange's spirited defense of liberty and openness stumbled badly when he was asked about recent events in the Ukraine, including the Russian military's apparent invasion of Crimea. Assange seemed to accept and even endorse the incursion. "Geopolitically, it is utterly intolerable for Sevastopol to fall into the hands of NATO," he declared. Such an occurrence would be "an existential threat to Russia."

"Russia will reclaim Crimea," Assange stated flatly. "And the United States will prop up the rest of Ukraine."

Asked if he is fears for his own safety, Assange began by saying "I am a normal person." Then he added: "Courage is not the absence of fear, it is seeing fear and proceeding anyway."

He also said, apparently in reference to intelligence and national security agencies in the United States and Britain: "They don't need to kill you. They just need make you believe they will kill you" to make you give up a quest such as his.

On a lighter note, he joked about Wikileaks that "we're one of the few media organizations that's in the black." And he said that his organization has proved an important principle, which is that "with a little help from your friends, with hard work, and with dedication, yes, you can stand up to these awful, fearful, great powers. Yes, you can outmaneuver them."

U.S. Navy: “I Thought the Other Guy Was Doing Security”

The Wall Street Journal reported this week that security holes exploited by suspected Iranian hackers  existed because of “a poorly written contract with computer-services provider Hewlett-Packard.” Under the terms of the contract, H-P wasn’t required to secure the Navy Department databases. But the Navy, under the mistaken assumption that the computer company was the sentry at the gates, didn’t assign personnel to oversee security for the databases. The lapse made the computer network ripe for the picking. How ripe? So much so, an unnamed source told WSJ that restoring the Navy network took four months and cost about US $10 million. The source, a senior defense department official said that “after the Iranian hack, the Navy took stock of its security efforts and drew up a list of 62 security issues…Some [will] cost more than $100 million and may require asking Congress for permission to redirect funding.”

Though it’s clear that, in the parlance of politicians, mistakes were made, everyone currently or recently in charge is being spared the indignity of being blamed for this massive screw-up. The unnamed defense official said the comedy of errors was based on “decisions made years ago as to what the Navy network structure should be and what kind of risk it was comfortable taking.” Because the contract was first awarded in 2000 and last updated in 2010, Vice Admiral Michael Rogers, who served as the Navy's cyber chief in 2011 and oversaw the cleanup, has been able to sidestep blame for the cock-up as Congress prepares to vet him for the role of director of the National Security Agency.

300 000 Routers Hijacked

Security researchers at Team Cymru in Lake Mary, Fla., published a report this week revealing that more than 300 000 small office and home office routers located across Europe and Asia have been compromised during a rash of attacks that began in mid-December. Team Cymru says hackers began overwriting the DNS settings on routers from a number of manufacturers, including TP-Link, D-Link, Micronet, and Tenda, and rerouting traffic to attacker-controlled sites. The victims, say the researchers, have been located mainly in Vietnam, Thailand, India, and Italy.

The attacks were first detected in January. The Cymru researchers noticed that several TP-Link routers were redirecting victims to two IP addresses that were unrelated to the sites unwitting computer users were trying to reach.

The attackers took advantage of a cross-site request forgery vulnerability on the devices that gave them admin privileges without them having to provide even so much as the default authentication password.

Team Cymru said it immediately notified the affected vendors, but when none responded, it shared the information with law enforcement.

Though there are similarities between this set of attacks and those suffered by several Polish banks in recent weeks, the Cymru report notes that, “The scale of this attack suggests a more traditional criminal intent, such as search result redirection, replacing advertisements, or installing drive-by downloads; all activities that need to be done on a large scale for profitability. The more manually-intensive bank account transfers seen in Poland would be difficult to conduct against such a large and geographically-disparate victim group.”

Cisco Issues Internet of Things Grand Challenge

There’s good reason for concern over the prospect that the “Internet of Things,” the name given to the point when just about every electronic device will be connected to the Internet, will create innumerable points of vulnerability that can be exploited by hackers. Few of these gadgets were originally created with security in mind. With that in mind, Cisco Systems has created the Internet of Things Security Grand Challenge, a $300 000 global competition whose winners will be the people who come up with innovative yet practical ways of securing millions of gadgets and the networks to which they connect. The entries will be judged on four criteria: feasibility, scalability, performance and ease-of-use; applicability to multiple industries and applications; technical maturity and viability of the proposed approach; and the proposers’ expertise and ability to feasibly create a successful outcome.

According to Cisco Security Group Senior VP Chris Young, as many as six recipients will be awarded between $50 000 and $75 000 at the company’s second annual Internet of Things World Forum in Barcelona, Spain later this year. The deadline for submissions is 17 June.

Report Suggests How to Secure the Grid from Cyberattacks

Another potential point of vulnerability to cyberattacks is the electric grid. And so although it’s not technically appropriate for a blog called This Week in Cybercrime, we’d be remiss if we didn’t report that a group of current and former U.S. government officials and representatives from the entities that operate the grid did last Friday. They convened a panel at the Bipartisan Policy Center in Washington, D.C., and presented a new report (PDF) containing guidelines for protecting North America’s grid. The report also included recommendations for what to do if the grid is ever compromised.

Matthew Wald, an energy reporter with the New York Times and the moderator of the panel session, noted that of the more than 250 cybercrime incidents reported to the U.S. Department of Homeland Security last year, two-thirds of them targeted the energy sector in general and the grid in particular.

“What permeates the report is that you can’t win this just defending the perimeter, you can’t win this with just prevention and defense,” former National Security Agency and C.I.A. Director General Michael Hayden said. “It’s the concept of resilience, what happens after things start to go wrong?”

Among the proposals in the paper, whose authors include Hayden, is the creation of a new Institute for Electric Grid Cybersecurity modeled after the Institute of Nuclear Power Operations. That group was formed in 1979, in the aftermath of the nuclear accident at Three Mile Island.

Cyberthreats: Assessing the Enemy Within

Clear your mind. Now quickly conjure the image of a group of hackers breaking into a corporate database. Did your mental image include the corrupt middle manager acting as the team’s inside man? How about the middle manager who violates security protocols and unwittingly opens the door to a cyberattack? A just-released report from PricewaterhouseCoopers (PwC) focuses attention on all aspects of global economic crime, not just cybercrime, but one of the things that stood out is how frequently the enemy lurks within an organization. “Many times those who are colluding [with hackers] are individuals inside these companies who have administrative access to the corporate computer system," Steve Skalak, a partner in PwC's forensic service practice told Investors Business Daily. Skalak coauthored the Global Economic Crime Survey.

The report notes that an increasing share of internal fraud of all types is being committed by middle managers—54 percent in 2012 versus 45 percent in 2011. PwC has even developed a profile of the average middle management fraudster: a man whose tenure with the organization is six years or longer.

"Because they have more intimate knowledge of internal processes and infrastructure, better access and higher trust, they can be a much more challenging 'enemy within' compared to junior employees or external fraudsters," Amir Orad, CEO of Nice Actimize, a New York-based unit of financial security software firm Nice Systems (NICE), told Investors Business Daily. He adds that middle managers who are actively involved in cybercrime tend to “feel they haven't been properly appreciated or compensated.”

Other middle managers, says Orad, may just be duped into helping hackers. "Because of [their] access, middle managers may unknowingly be accomplices to cybercrime and fraud by having their credentials and accounts taken over by cybercriminals. Cybercriminals know that middle management has access to key systems and therefore target this layer within the organization."

Meetup Website Suffers DDoS Attack After Spurning Ransom Demand

For five days, many groups that make connections via the social media site Meetup were unable to. The website was felled by several massive DDoS attacks that began last week and resulted in a protracted battle against the cybercriminals to keep the site up and running.

Last Thursday, Meetup CEO Scott Heiferman received an e-mail that said, "A competitor asked me to perform a DDoS attack on your website." Heiferman revealed in a blog post that the sender said the attack wouldn’t happen if the company forked over a measly $300. When the company refused, reasoning that to negotiate with criminals would make the site a target for further extortion—demanding much greater sums—the series of attacks began. The site was quickly overwhelmed, but service was restored by Friday morning. The battle didn’t end there, however. Another attack brought the site down again, and as of Monday, Meetup was reporting that it was working urgently to restore functionality. By Tuesday, it was back online, with a link on its homepage to some FAQs related to the outage. The company was quick to reassure customers that none of their personal data, including credit card information, was accessed during the cyberattack.

Cybercrime Hits the Airwaves

CBS is planning to air a spinoff of the hit TV series "CSI" (short for Crime Scene Investigation) that will focus on the agents in the FBI's cybercrime division.




GM’s “Heavy Keychain Recall” Reaches 1.6 Million Vehicles

IT Hiccups of the WeekThere was a plethora of IT-related faults, errors and defects reported last week. We start off this week’s edition of IT Hiccups with what arguably is not a classic IT problem but a more general electronically-related one that affects over 1.6 million GM vehicles.

A few weeks ago, a New York Times article reported a world-wide GM recall (pdf) of 778 000 or so 2007-model-year Pontiac G5 and the 2005-7 Chevrolet Cobalt (619 000 in the U.S.) because a “jarring event” such as a crash, bumping the ignition, or a heavy key chain could inadvertently cause the cars’ ignition switches to move from the run position to the accessory position. Switching into that mode would disable the cars’ engines and prevent their air bags from deploying.

At the time, the Times reported that GM “knew of six deaths in five crashes in which the front air bags did not deploy” as well as 17 additional crashes “involving some type of frontal impact and nonfatal injuries where the air bags did not deploy.”

A GM spokesperson, having insisted that, “Safety of our consumers is paramount to G.M,” also tried to minimize the recall by saying that, “All of these crashes occurred off-road and at high speeds, where the probability of serious or fatal injuries was high regardless of air bag deployment. In addition, failure to wear seat belts and alcohol use were factors in some of these cases.”

GM recommended using only the key issued with the affected vehicles until the ignition switch is repaired.

In addition, GM disclosed the news that it had issued a service bulletin in 2005 about the issue to its dealers, although the car manufacturer was not exactly forthcoming in saying whether dealers felt obligated to inform vehicle owners of the potential problem before (or after) purchasing their GM vehicles.  GM did say that “the ignition switch torque performance may not meet General Motors’ specification.” In other words, dealers were going to tighten the switch on the recalled vehicles.

A Detroit News story quoted several auto analysts who said “because the company took quick action” the reputation risk impact of the recall to GM was minimal, rating it as only a “6” on a scale of 10.

However, last week, GM expanded the recall by another 748 000 cars in the United States as it disclosed that four more of its vehicles—the 2003-07 Saturn Ion and the 2006-07 Chevrolet HHR, Pontiac Solstice, and Saturn Sky—also used the same ignition switch. The additions brought the total to nearly 1.37 million vehicles in the U.S., and 1.6 million vehicles worldwide. What’s more, GM revealed that the number of related deaths has reached thirteen and the number of reported crashes due to the defect rose from 22 to 31.

GM declined to explain to various news media inquiries why it did not include those vehicles in its original recall, why the additional crashes and deaths were not reported or linked to the others, or why it had taken so long for the company to issue a recall since it admitted that it knew of and had been studying the problem since 2004.

In fact, the Detroit News reported, GM “spent nearly a decade studying the issue and repeatedly opted not to recall the vehicles or pay for potentially expensive fixes.” GM, the Detroit News stated, “downplayed the ignition switch issue in prior years, including canceling in 2005 an approved redesign of the ignition key head. By the end of 2007, GM said it knew of 10 frontal crashes in which air bags didn’t deploy—linked to the ignition problem—but the automaker opted not to recall the cars.”

GM North America President Alan Batey, sensing that the recall issue had moved considerably higher on GM’s reputation risk meter, issued a “rare apology.” He said that GM was “deeply sorry” for the problem, and that the chronology of its actions, reported to the National Highway Traffic Safety Administration (NHTSA), showed that “the process employed to examine this phenomenon was not as robust as it should have been.”

NHTSA shortly thereafter announced it was launching an investigation into GM’s recall delay. GM faces up to a $35 million fine for not reporting problem in a timely manner to NHTSA. However, NHTSA was itself under heavy criticism from at least one U.S. senator for not acting earlier when it became aware of the problem with the ignition switch back in 2007. GM, in light of the NHTSA’s announcement, issued an unheard of second apology saying that, “We deeply regret the events that led to the recall and this investigation. We intend to fully cooperate with NHTSA and we welcome the opportunity to help the agency have a full understanding of the facts. Today’s GM is committed to learning from the past while embracing the highest standards now and in the future.”

A story in yesterday’s New York Times provides a bit more information about the chronology of the ignition switch issue. Among the damning details is the fact that back in 2004, GM engineers were able to replicate the problem and suggested a fix, but GM executives decided against it after “consideration of the lead time required, cost and effectiveness.”  Another example of company executives playing “pay me now or pay me later” roulette.

[Update 06 March 2014:  GM’s new CEO Mary Barra announced in an email to GM employees this week that she had put into place a “working group of senior executives, which I lead, to direct our response, monitor our progress and make adjustments as necessary.” In addition, Barra stated there is now “an internal review to give us an unvarnished report on what happened. We will hold ourselves accountable and improve our processes so our customers do not experience this again.” She added that, “We sincerely apologized to our customers and others who have a stake in GM’s success.”

To “help” GM with its inquiry, NHSTA has now sent a 27-page list of 107-questions seeking all GM information about the recall and why it wasn’t initiated earlier.

GM says that replacement parts for the defective ignition switches will begin to become available early next month.]

GM Chevy Silverado and GMC Sierra Truck Steering Control Malfunction

GM was also the source of an IT-related hiccup last week. According to Edmunds.com, GM is trying to convince NHTSA that a “glitch” affecting its steering wheel controls  is “inconsequential to motor vehicle safety” and doesn’t warrant the recall of more than 200 000 GMC Sierra and Chevy Silverado trucks manufactured between 29 January and 28 October 2013. Another recall would be quite embarrassing considering that the Silverado was recently named 2014 North American Truck of the Year. You may remember that just a few weeks ago, GM recalled 370 000 of those same two vehicles for a software update in order to reduce the likelihood that their exhaust systems would overheat and catch fire.

In this latest problem, GM says that, “under certain circumstances when an owner uses the steering wheel controls to browse and select songs to play from an external device (i.e., MP3 player) that is plugged into one of the vehicle's USB ports, the instrument cluster may reset. When the instrument cluster resets the analog gauges and identifications, the PRNDM [shift position] indicator, and the cruise control telltale will briefly turn off. In addition, some of the instrument cluster telltales may also illuminate briefly without the condition the telltale is designed to indicate being present.”

It doesn’t sound like a major safety issue, but it is one that could annoy or even distract a driver. However, given aforementioned GM ignition switch recall debacle, the previous software recall on the same vehicles, and the interesting interaction of what would seem to be at least on the surface disparate vehicle systems, NHTSA may want a bit more information from GM before granting it a waiver.

New Zealand Hospital EHR Outage Sparks Political Row

While not nearly on the same technical level as the Affordable Care Act health insurance exchange problems in the United States, a relatively minor electronic health record system outage in New Zealand has created much the same political hue and cry, apparently.

Last week, Dunedin Hospital, which serves the Otago catchment of New Zealand, suffered an electronic health record outage that lasted a little more than a day due to an apparent hardware problem. Staff reportedly resorted to paper records during the outage and the hospital said patients were not put at any risk by the outage. Coincidentally, New Zealand Prime Minister John Key, who leads a National-led government, was visiting the hospital the day after the EHR system crash. Key, who was on hand to open the hospital’s new neonatal intensive care unit, reiterated to the press that the incident was not a big deal, saying that anyone working with computers “will know that at some point they break down.”

Others, however, such as out of power Labour Associate Health spokesman David Clark, saw something sinister in the outage. Clark vigorously proclaimed that, “Patient care has been compromised, there's no doubt; radiation treatment didn't happen yesterday; there have been other monitoring mechanisms that are in place that just simply weren't working.” Clark said the EHR outage was obviously the result of the ruling government’s “cost cutting pressures.”

While Clark was clearly trying to score political points, the hospital apparently has no robust back-up systems in place to handle equipment or software outages. Given that the hospital recently admitted that a different IT failure that resulted in the loss of 4000 mammogram images taken in 2012, it is probably fair that some IT professionals are calling for a review of the hospital’s IT systems.

DMV Headaches Abound

We close this week’s edition of IT Hiccups with a couple of stories of motorists having trouble with their local department of motor vehicles, something all of us can readily relate with. The first is about a software error in the computer systems of the Washington, D.C., DMV that has existed for at least 5 years (and possibly as long as 15). The bug had made it extremely difficult if not impossible for motorists to get refunds for incorrectly issued traffic citations. At least 450 paid traffic tickets that were later voided were not refunded because of the error—a problem the DMV says is now fixed, but was addressed only after television stations started reporting on motorists fighting for years to get their refunds.

Next, news reports emanating from North Carolina indicate that problems with that state’s new DMV computer system have resulted in its offices “losing or delaying up to 35,000 vehicle inspections per month.” The computer system, the DMV says, “has had logic and code issues” that frequently keep the record of a successful vehicle inspection from being successfully communicated to the DMV from the state’s 7500 certified inspection stations.

Typically, the DMV sends out a bill to the motorist once it is notified of a passed car inspection. As a result of the computer problems, a motorist can’t pay for their car inspection nor can they pay their annual vehicle fee when it is due since the state doesn’t know that the vehicle has been successfully inspected as required by state law. According to media reports, there is no timetable for when the problem will be resolved.

Finally, news reports from across New York State indicate that local DMV offices have been experiencing repeated IT outages that are driving both DMV employees and motorists crazy. As in North Carolina, no one in charge seems to have a date in mind as to when the outage problem will be fixed.

Alas, would you expect anything different from the DMV?

GM Recalls 1.6 Million Cars for Ignition Switch Fix

GM Recalls 778 000 Small Cars for Ignition Switch Problem

GM Ignition Switch Recall Not a Big Deal Auto Analysts Claim

GM Adds 750 000 Cars to Recall Bringing Total to 1.6 Million

GM Delayed Ignition Switch Fix for Years

GM Issues Rare Public Apology Over Recall

NHTSA Launches Probe into GM Cobalt Recall

NHTSA Aware of Ignition Switch Issue in 2007

 Massachusetts Senator Markey Asks Where Was NHTSA?

Trail of Inaction in Ignition Switch Recall Debacle

Truck Glitch is Inconsequential GM Tells NHTSA

New Zealand EHR Meltdown Sparks Political Row

New Zealand Hospital Suffered EHR Outage

Labour Claims Patients Placed at Risk During EHR Outage

Review of Hospital IT Systems Called For

Motorists Suffer Because of State DMV Computer Woes

DC DMV Finally Fixes Software Error That Stopped Ticket Refunds

NC DMV Computer Error Affects Vehicle Inspections

NY DMV Offices Suffer Plague of Computer Problems

In Other News…

California Healthcare Insurance Website Software Problem Forces 14 500 Applicants to Start Over

Maryland Healthcare Insurance Website Flaws Cause $30 Million in Unnecessary Medicaid Payments

Washington State Healthcare Insurance Exchange Keeps 15 000 Applications in Limbo

Bomb Threat Sent in Error

Software Issue Delays Payment for Indian Eye Surgeries

Social Security Numbers Printed on Banner Health Address Labels

Milwaukee Fire Department’s New CAD System Has Flaws

New Digital Water Meters in LaVergne, Tennesse, Cause Billing Problems

Louisiana Disqualifies Food Stamp Recipients for Exploiting Benefits Glitch

Florida Says New Unemployment System Fixed, But Many Still Express Doubts

NPower Utility Billing Errors Power On

Swedish Company Accidentally Invites 61 000 to Job Interview

Pizzeria Unwanted Recipient of American Idol Votes

DeGeneres' Selfie Causes Oscar Twitter Crash

ABC’s Live Oscar Internet Stream Goes Out

Photo: GM/AP Photo


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More