Risk Factor iconRisk Factor

Obamacare Data Hub Security Faces Scrutiny

We already know that the HealthCare.gov website and many of the state-run healthcare insurance exchanges created as part of the Obamacare rollout hit the ground crawling on 1 October. There is much left to be said about the players in that still-unfolding debacle. But there’s an element of the drama—namely data security—that will likely get increasing attention as fixes aimed at letting the uninsured sign up for coverage allows millions of people to finally input their personal information. But Congress is wasting no time: Members of Congress want answers about Obamacare Data Hub security measures


In Other Cybercrime News…

  • Hackers broke into a database at PR Newswire that contained login credentials and contact information for the press release distribution service’s customers—tens of thousands of companies and public relations agencies

Image: iStock

California’s EDD Unemployment System Disaster: Predictable Fiasco?

The unemployment insurance payment system fiasco presided over by California’s Employment Development Department (EDD) just keeps getting more bizarre by the week. California has the largest unemployment system in the U.S., distributing about $33 million a day in unemployment checks to some 800 000 claimants when everything is working properly. This week it was revealed that EDD management knew its new unemployment system contained potentially major operational flaws, but decided to roll it out anyway.

As I noted last month, the EDD has spent US $157.8 million upgrading the state’s 30-year-old unemployment payment processing system. The upgraded system was originally supposed to cost $35 million and go live in 2009 (pdf), but a series of ridiculously incomplete (if not incompetent) definitions of the system’s requirements by EDD officials significantly pushed up the project’s cost and delayed its delivery schedule.

The EDD finally transitioned to its upgraded unemployment payment system over this year's Labor Day Holiday weekend (31 August to 2 September). There were a few reported hiccups with the transition at that time, but nothing seemed too far amiss during the first week after the switch. In fact, EDD officials reportedly were congratulating themselves on a job well done.

However, by the second week after the transition, it became increasingly clear that there were the proverbial "technical issues" cropping up within the unemployment system. The EDD announced that while the new “system is working as designed,” there were “some processing delays in [the] transition” from the old to new benefits system.  Nonetheless, the EDD said that it was “working around the clock to catch up on unemployment claims.” A San Francisco Chronicle story quoted an EDD spokesperson who claimed only about 5 percent of the unemployment benefit claims—or about 20 000—were negatively affected by the upgrade's technical issues. She added that, “We apologize for the inconvenience to those affected.”

A few days later, though, the EDD started to hint to the press that the problem might be a bit bigger than it first let on. The EDD said that the new system was misreading legacy beneficiary data, and that it didn’t realize the issue would be so widespread. Rule number one in any transition from a legacy system to its replacement is to ensure that existing data is clean, consistent and complete so that it can be imported into the new system with little difficulty as possible. So that admission of wide-spread data quality issues was not an especially good sign. Word was also filtering out from EDD employees that the scope of the problem was in fact huge, but the EDD wouldn’t dignify those claims by making a comment.

Eventually, the EDD admitted that the number of affected claimants was closer to 50 000 instead of its previously estimated 20 000. The agency reiterated that its employees were “working around the clock and through the weekends to try and get these payments issued for the customers eligible and waiting for benefits.”

A few days later, the EDD revised its numbers upwards again, suddenly acknowledging that 185 000 unemployed Californians had been impacted by the system’s operational problems and that 80 000 claimants still hadn’t received their checks three weeks after the upgraded system went live.

As the debacle entered its fourth week, Marty Morgenstern, California's Secretary of Labor and Workforce Development, finally weighed in, saying that the continuing delays were “unacceptable.” With Gov. Jerry Brown’s support, an order was given “to immediately begin the process of paying backlogged claims for continued UI [unemployment insurance] benefits prior to a final determination of eligibility.”

Morgenstern’s order helped whittle down the backlog, and by the middle of the next week, the EDD officially declared victory (pdf). The agency proudly proclaimed during the first week of October that its “aggressive efforts succeed[ed] in eliminating our backlog of certifications.” That is an interesting choice of words, since the backlog was mostly eliminated by Morgensten allowing a bypass around a core system requirement.  Even so, EDD’s claims to have eliminated the backlog were immediately challenged by many unemployed Californians complaining that they still hadn’t received payments due them, a story at the LA Times stated.

Then another problem surfaced: apparently the claim forms needed to keep unemployment benefits flowing to beneficiaries went missing, the LA Times reported in a separate story. The Times reported that when queried about the missing forms, an EDD spokesperson initially “side-stepped” the question, then gave what amounted to a non-answer, and then wouldn’t respond to further questions from the newspaper about the missing forms.

The ongoing problems had, by early October, spurred a number of state legislators to call for an investigation into the system’s contractor, Deloitte Consulting. As I have previously described in more detail, EDD gave the contract to Deloitte despite the company having a long history of troubled IT projects across California state and local governments.

The most recent bombshell hit this Monday, when the Sacramento Bee followed up on a story first reported by television station KXTV in Sacramento last Friday. The station along with the Bee reported that the new unemployment system was "broken from the start," that EDD officials knew it was broken, and that they moved forward with the system’s roll out anyway. Furthermore, said the reports, when EDD realized that there was indeed a major problem with the system after it went live, officials tried to blame the press for magnifying the severity of the issue. And when that didn’t work, they turned to blaming budget cuts for the lack of testing performed on the system. Just as damning were the reports' revelations about insider e-mails indicating that as many as 300 000 Californians had been affected by issues with the system, not the 185 000 the EDD was publicly claiming.

EDD officials claimed that tests conducted before the new system was rolled out indicated that all the problems would be manageable. And, furthermore, if the tests had suggested otherwise, EDD officials insisted, the agency would have delayed the system's debut. Yet the KXTV story indicates that the errors found during unit tests in early system builds were routinely being passed onto future system builds without ever being corrected. (In some quarters, this is considered poor software development practice, while in others, it is seen as a good business strategy for ensuring future software maintenance work.) Internal EDD IT programmers who sounded the alarm about this practice of kicking the can down the road were said to have been ignored.

Massachusetts, whose own unemployment system has been having problems as well, recently admitted that its original contract with Deloitte to modernize that state’s $46 million unemployment system was “flawed” and allowed Deloitte  “to miss deadlines and still charge the state some $6 million more than originally planned,” the Boston Globe reported in September. State legislators in Massachusetts announced earlier this month that hearings are going to be held to look into the problems with the Deloitte-built unemployment system's roll out.

Last week, Florida began its move to a new unemployment system also developed (late and over-budget) by Deloitte, and, you guessed it: there are reported problems there as well. It is too soon to tell if Deloitte will face another embarrassing government hearing in Florida, but I wouldn’t bet heavily against it.

These latest disclosures seemed to finally force reluctant senior California politicians this week to agree to hold hearings on the EDD development next month or in December. Hopefully when they do, they’ll remember to ask Deloitte why it keeps insisting that the unemployment system is working and why it believes that none of the problems experienced to date are because of a “breakdown or flaw in the software Deloitte developed.”

Photo: Damian Dovarganes/AP Photo

Talking about the STEM Crisis Myth

Last month’s article “The STEM Crisis Is a Myth,” by IEEE Spectrum contributing editor Robert N. Charette, triggered a hearty response from readers. Many commenters shared his view—that there is no shortage of scientists and engineers—and quite a few were against it. It seemed clear that a discussion of the issue should continue.

And so, on 7 October, IEEE and Arizona State University’s Consortium for Science, Policy and Outcomes, convened a conversation between Charette and CSPO co-director Dan Sarewitz at CSPO’s Washington, D.C., office, just north of Dupont Circle. For those of you held back from attending by the government shutdown, the torrential rain, or the fact that you live nowhere near D.C., we’re posting a video of the hour-long event.

Radio fans can listen to Charette’s recent interview on NPR’s “Here and Now”. And Spectrum’s expanded coverage of the STEM crisis can be found here.

IT Hiccups of the Week: Electronics Benefits System Outage Hits 17 States

Last week saw computer-related problems still plaguing California’s modernized unemployment insurance system, and warnings of potential problems with Florida’s new unemployment system that is now being rolled out. There were also widespread problems with the IT systems supporting the Affordable Care Act, but these will be discussed in detail in a Risk Factor post later this week.  We start our review of IT Hiccups from last week with an outage over the weekend affecting the food assistance card payment system used in 17 states.

Problems with Electronic Benefits Debit Cards in 17 States on Saturday

Customers Experiencing Problems with EBT Cards

EBT Goes Down For Hours in 17 States during Routine Test of Backup Systems

EBT Glitch Exploited in Louisiana after Showing No Limits

Xerox: EBT Problems Resolved

Software Problem Takes Down Canada’s Rogers Communications Wireless Service

Software Glitch Blamed for Massive Rogers Outage

Rogers Apologizes for Canada-Wide Outage

Rogers Outage Highlights Canadian 911 System Issue

California Unemployment System Still Not Properly Fixed

California EDD Can’t Keep Pace with Unemployment Claims

EDD’s “Fully Functional” System Suffers More Problems

EDD $100 Million System: “Broken from the Start”?

Legislators Call for Investigation into EDD System

Fonterra Dairy’s New Software Systems Hides the Cheese in New Zealand

Software Issues Hits Milk and Cheese Supplies

Glitch Cheeses off Customers

Of Other Interest…

New Florida Unemployment System Rollout May Be Bring Benefits Delay

North Carolina Healthcare Providers Still Waiting to Get Paid by State’s New Medicaid System

NASA Jupiter Probe Now Fully Operational After Anomaly

Asda Supermarket’s Software Error Allows Free UK Shopping

Computer Problem Strands Universal Studios Roller Coaster Riders

Sydney’s Central Business District hit by Optus and Virgin Mobile Outage

Major College Online Application System Malfunctioning

Photo: Rich Pedroncelli/AP Photo

New Report Says Cyberthreats Multiplying Like Tribbles

Hackers have proven time and time again that they’ll eventually find a way to defeat any single digital security method. Their motivation to do so is evident in the fact that, on average, more than 150 000 new, unique malware strains are unleashed each day. That’s one of the startling conclusions drawn by analysts from the Aite Group in the report “Cyberthreats: Multiplying Like Tribbles” that was released earlier this week.

Tribbles were fictional creatures featured on the TV series Star Trek. They multiplied so rapidly that their consumption of resources grew exponentially. The same appears to be true of cybercrime. Julie Conroy, research director at Aite’s banking division and coauthor of the report, told IEEE Spectrum that last year, hackers were pumping out 72 000 new malware strains per day, less than half of the current level of cybercrime activity.

So, what’s the upshot? According to the report, “The username/password combination as an authenticator is officially broken…the sole relevant use of this combination is now that of a database look-up mechanism.” More than half of computer users don’t follow security experts’ advice to choose different, strong passwords for each of their online sign-ups—which allows a blaze in a small thicket to engulf a person’s entire online forest, so to speak. But what if you do follow best practices? “Nobody is ever 100 percent secure,” is the report’s sobering conclusion.

It does, however, point out steps that businesses such as banks, which are the primary targets of cybercrime, are taking to make a hacker’s job harder.

Among them are new ways to prevent a hacker from pretending to be an actual customer. Technology is available that will allow your bank to generate a “device fingerprint” for the computer, tablet, or smartphone you regularly use to conduct transactions. Business conducted from an unknown device automatically triggers more authentication steps.

Firms are also looking to use behavioral analytics. The vendor would collect data about how the customer interacts with, say, his or her smartphone. If the person using the handset owned by John Q. Smith (confirmed by the device fingerprint) doesn’t press the keys or swipe the touch screen the way Mr. Smith usually does, red flags would be raised.

Asked whether these security measures might be considered too intrusive, Conroy says they’re built into the process so that the average customer doesn’t even know it’s happening. “The aim is to perform a balancing act,” she says. “Businesses are asking themselves: How do we enable a secure environment without appearing to be Big Brother?”

Striking that balance may be impossible—especially in light of the fact that the U.S. government has and continues to force companies to turn over customer data. Conroy,whose research focuses on fraud, data security, and preventing money laundering, acknowledges that these new strategies may be implemented at the cost of a little privacy. But, she says, the alternative may be the loss of online and mobile channels for conducting business as the benefits of e-commerce are devoured by the rising tide of Tribbles. How much is being consumed? The report predicts that businesses worldwide will suffer more than half a billion dollars in losses from corporate account takeovers. Cyberthieves will take nearly US $800 million in 2016, say the analysts.

Image: Paramount Pictures

IT Hiccups of the Week

IT Hiccups of the Week The format of IT Hiccups of the Week is changing. It will now be more an aggregation of stories of IT-related system troubles from around the Web. This week saw a wide-range of IT snafus and snarls affecting millions of people, starting with the sign-up troubles involving the public health exchanges being created under the U.S. Patient Protection and Affordable Care Act (pdf) and more issues with U.S state IT projects.

U.S. State IT System Problems Piling Up

California EDD Department Says Backlog Cleared, Many Unemployed Say Not True

Nevada  Blames Feds for Recent New Unemployment System  Woes

Michigan’s New Unemployment Insurance System Stumbles Out of the Gate

Kansas Hospitals Bitterly Complain about State’s New Medicaid System 10 Months after Going Live

Massachusetts Senate Panel to Hold Hearings on Troubled New Unemployment Insurance System

North Carolina Lawmakers to Investigate Poor Medicaid and Food Stamp Systems Rollouts

Scotland’s Largest Health Board Suffers Major System Crash

NHS Greater Glasgow and Clyde Health Board Says IT System Affecting 11 Hospitals Finally Fixed

 “Unique” Active Directory Glitch Blamed for IT Failure at Scottish Hospitals

Minister Orders Investigation into Scotland NHS Computer Chaos

Of Other Interest…

Bank Error Makes World’s First Multi-Trillionaire

Tesco Pricing Glitch Sells 12-Piece Dinnerware Set for 56p

Weis Markets Charges Customers Credit Cards Multiple Times across Its 165 Stores in 5 States

Chrysler to Fix Software Flaw in 140 000 Pickups and SUVs Worldwide

Telstra in Australia Email Outage Angers Users

France Blames Phone Company “Malfunction” for Wrong August Unemployment Numbers

Photo: iStockphoto

Obamacare Exchange Sign-ups Hobbled by IT Systems Not Ready for Prime Time

I don’t need to tell anyone about the controversy surrounding the Affordable Care Act (more commonly known as Obamacare). It was the central issue in the game of brinksmanship that led to the U.S. government shutdown last week. But mirroring that mind-blowing dysfunction was the less-than-stellar 1 October rollout of the federal website healthcare.gov. The Obamacare-mandated Web portal lets consumers who don’t have employer-sponsored medical insurance meet the legal requirement to sign up for health coverage through the states where they reside. (To be precise, healthcare.gov is for residents of 36 states whose governments opted not to set up independent healthcare exchanges.) Online exchanges for some of the other 14 states and the District of Columbia also debuted with disappointing results.

So, what happened? Well, it’s no secret that governments are terrible at IT project implementation. Examples abound—as regular readers of The Risk Factor are well aware. (Some of our reporting on recent foul-ups is here, here, and here.) There’s been little evidence so far that these projects are any different.

The Department of Health and Human Services (HHS) reported Wednesday that there were 6.1 million unique visitors to healthcare.gov on the first day and a half after the site opened on Tuesday. By Friday, that number had surpassed the 8 million mark. That’s a good indicator of the level of interest in getting signed up for health coverage. But it’s only part of the picture. What HHS purposely left out (and left to our imagination) is the actual number of enrollments. Officials said they would probably release enrollment numbers next month after tabulating totals from, online, call centers, and paper enrollments. But the picture that’s forming based on anecdotal evidence is not pretty.

Most attempts to reach the federal website resulted in this:

“We have a lot of visitors on our site right now, and we're working to make your experience here better. Please wait here until we send you to the log-in page. Thank you for your patience.”

Or this:

“Important: Your account couldnt (sic) be created at this time. The system is unavailable."

According to a Los Angeles Times story, community groups aiming to help people sign up have been frustrated in their attempts to do so. Even large insurance companies, which have a vested interest in getting people enrolled in the exchanges, were unsuccessful in the early going. For example, a spokesman for Blue Cross Blue Shield of Louisiana, that state's largest insurer, told the Los Angeles Times that, as of Wednesday, the company hadn't been able to enroll anyone through the federal website. Others who left in frustration included reporters including one for the Huffington Post, who said: “Though officials from the Centers for Medicare and Medicaid Services said they'd made strides correcting the federal exchanges' problems, The Huffington Post made dozens of attempts and still couldn't sign into the website late Tuesday afternoon.”

"We have had a few slowdowns, a few glitches, but it's sort of a great problem to have. It's based on the fact that the volume has been so high and the interest is so high," Health and Human Services Secretary Kathleen Sebelius said on MSNBC Tuesday. "We're working quickly to fix that."

U.S. Chief Technology Officer Todd Park explained further, pointing out that the government expected HealthCare.gov to draw only as many as 60 000 simultaneous users. That estimate was apparently based on a projection from the volume experienced nearly a decade ago on a site for Medicare Part D. But at peak, the Obamacare site was being deluged by up to 250 000 people at a time.

"These bugs were functions of volume,'' Park told USA Today. "Take away the volume and it works.''  Right. Take away widespread interest in signing up for health insurance, and the portal through which people are supposed to sign up for health insurance will work as intended.

The system’s performance invited a swipe from an IT official from the previous administration. “Whoever thought it would draw 60,000 people wasn't reading the administration's press releases,” David Brailer, former national coordinator of health care information technology under George W. Bush, told USA Today. “The Medicare Part D site [launched in 2006] was supposed to have 20,000 simultaneous users and was [built to accommodate] 150,000, and that was back when computing was done on an abacus. It isn't that hard.”

The news wasn’t any better with the state-run exchanges. California residents were stuck in traffic along both routes to enrollment there: computer glitches stymied attempts to sign up online, while hold times at telephone call centers topped 30 minutes. The computer system created to, among other things, log a consumer’s data and determine whether he or she is eligible for government subsidies to cover part of the premiums, responded so poorly that its operators were forced to shut down the online enrollment system twice. According to the L.A. Times story, “Officials were pleased with the strong consumer interest and vowed to fix the problems.”

On the opposite coast, officials in the second most populous state fielding its own exchange reported what could generously be described as an anomaly. State of Health, the healthcare portal serving New York State, which has a population of roughly 18 million, had reportedly received 30 million hits by late Wednesday, prompting some observers to suspect that hackers may plotting a break-in or an out and out takeover. Whether that’s true or not, Donna Frescatore, director of the state’s exchange, confirmed that despite all that activity, only about 12 000 people had managed to enroll by Wednesday evening.

Responding to questions about the extraordinarily high volume, Frescatore told the Wall Street Journal that, “We have no evidence that this is anything but people learning more about [the site].” Furthermore, said Frescatore, state officials are not looking into the possibility that cybercrime was a contributing factor.

Ahhh…the power of positive thinking.

We can all keep our fingers crossed, but the issue of security will likely pop up again. As we recently reported on this blog, privacy safeguards have likely taken a backseat to getting the exchanges open on time. Another IEEE Spectrum post focusing on the exchanges’ security issues is here.

There’s no question that the overwhelming interest caught New York flat footed. Officials took the Web portal offline Tuesday night. Once the smoke cleared, they doubled its capacity and implemented some fixes aimed at keeping it from getting hung up as it did throughout its first day of real-world operation. What happened on day two? The same thing, more or less.

California and New York weren’t alone in their misery. According to a Huffington Post article, at “Maryland Health Connection, Kynect [Kentucky’s exchange], Connect for Health Colorado, Rhode Island's Health Source RI and others, consumers faced obstacles to setting up accounts or comparing plans—or even viewing the websites at various points in the day.” The Chicago Tribune reported that a glitch affecting Illinois’ exchange—missing fields in an online form—left people attempting to enroll in on the first day unable to figure out whether they were eligible for the federal subsidy for premiums. Though that problem was remedied by the middle of the day, sailing still wasn’t smooth, said the Chicago Tribune article. Illinois Gov. Pat Quinn’s advice for those who had trouble accessing the site? "Just keep trying."

Criticism of the portals’ bumpy first week has come from all quarters. But the rollout still has its apologists. In an e-mail sent to the Huffington Post, Jonathan Gruber, a Massachusetts Institute of Technology economist who was an architect of the 2006 Massachusetts health care regime after which Obamacare was modeled, says,

"Hours or even days is not the relevant timeframe for evaluating exchanges. The question is simply whether there are ways that folks can sign up to get insurance by Jan. 1. That is a question for late November, not early October. If things are really buggy in six weeks, that could be more of an issue."

The best “Keep hope alive” message had to be the widely reported one delivered by HHS Secretary Sebelius.

“I clearly have an iPad and I also have an iPhone and about 10 days ago I got the prompt that the operating system had changed,” Sebelius said. Noting that the experience wasn’t great, she added that, “everyone just assumes ‘well there’s a problem, [Apple will] fix it.’” Here’s the good part: “We’re building a complicated piece of technology, and hopefully you’ll give us the same slack you give Apple.”

Matthew Yglesias, writing for Slate, deftly picks the Sebelius comment apart:

“Apple, like any private business, is customer-driven. Apple knows that if it doesn’t provide good products and services, the public will exercise its options, and go to Samsung and Android, or Windows, or even Blackberry…Apple, the world’s most-valuable brand, has a reputation for producing quality products that work. The government has exactly the opposite track record. There is no public confidence in government programs, whether they be in veterans’ affairs, the postal service, the stability of Social Security, containing spending, managing contracts, rooting out fraud, the IRS, the NSA, the EPA, immigration, self-investigating, protecting our Embassies and personnel — you name it.”

As the federal and state governments have repeatedly reminded us, the more than 40 million U.S. residents without employer-sponsored health insurance have until 15 December to enroll in order to get coverage on 1 January, and until 31 March to avoid being assessed a penalty. Will the sites’ managers get their respective acts together in time? I won’t call Sebelius’ and Gruber’s optimistic takes on the situation into question. I’ll simply direct readers’ attention once again to the Risk Factor links in the second paragraph of this post. They’re concrete illustrations of the points Yglesias makes.

Here are links to several other related articles:

Photo: Mike Segar/Reuters

IT Hiccups of the Week: California’s Unemployment System Upgrade Saga Continues

Last week was a good week for IT project spotters to add to their snafu collection. We review our collection’s additions this week with a system snafu in California that just keeps chugging along with no real end in sight.

California’s Unemployment Insurance Upgrade Problems Rumble On

California’s Employment Development Department (EDD) put out a press release last Friday that tried to put another heavy layer of lipstick on its modernized unemployment insurance system’s pig. The EDD stated that it is “wrapping up work on new upgrades to our 30-year-old payment processing system” and that one month after the upgrade went live, it has reached the milestone of processing “about 83% of all certifications from all claimants within the week in which the certification was received.” Of course, 17 percent out of 800 000 who are on unemployment in California is still a pretty big number.

Also, notice the use of the word "about" used in the press release. As I noted earlier this month, EDD has had repeated trouble figuring out how many unemployment claims it has failed to process correctly. The number has grown steadily; first it was only 5000, then it grew to 20 000, then 50 000, and finally, at least 185 000. What also is unclear from the EDD press release is what it means by “claims processing.” Because of problems with the system upgrade, EDD personnel have been working lots of overtime to process unemployment claims by hand. It would helpful if EDD would break out its “about 83%” figure into those performed correctly by the new upgraded system, and those it still has to process by hand.

At one point early last week, some 124 000 Californians had yet to receive their unemployment checks—many many of whom who had been waiting since early September. Now that number has been whittled down to about 40 000. However, a big reason for that drop was Gov. Jerry Brown ordering the EDD last week “to start issuing the delayed checks immediately and verify the recipient’s eligibility afterwards.” EDD says it “hopes” to send out more checks later today.

Somehow I doubt the cost (or time) of verifying eligibility after the fact was budgeted into the upgraded system’s original project budget, although it probably should have been. Why? Glad you asked.

First, state unemployment insurance upgrade projects don’t have a good development track record: just ask Colorado or Pennsylvania, both of which canceled their projects after not being able to get them to work even after massive cost overruns.

Second, even when the systems go live, they aren’t very reliable, as Nevada and Massachusetts unemployed residents have found out over the past few weeks. Unemployment insurance officials always admit that they expected problems, but they are inevitably surprised that there are many, many more of them than they expected. For instance, an EDD spokeswoman told the LA Times that the department expected some “hiccups” with the system upgrade, “but we didn't know it'd be to this magnitude.”

Third, even when a system’s performance is admittedly “unacceptable” as California’s Secretary of Labor and Workforce Development Marty Morgenstern stated last week, it doesn’t prevent other government officials in the same department from claiming that problems are really not that bad. They only appear bad, they say. In Massachusetts, for example, Labor Department Secretary Joanne Goldstein insisted last week that the state’s new $46 million unemployment insurance system was operating with minimal problems even as a flood of complaints about the system came out after its rollout three months ago, the Boston Globe reported. The state had to negotiate a contract warranty extension with Deloitte Consulting, the prime contractor, a move made necessary in order to fix those supposedly minimal problems.

Finally, government IT program managers like to live dangerously. For instance, EDD gave the contract for its unemployment system to Deloitte “despite a string of problem projects” across California state and local governments that littered the company's track record, the LA Times reported late last year. These included: a project canceled after four years by California’s Department of Developmental Services after it figured out the system it paid $5.7 million to Deloitte to develop didn’t work; a project to link California's court system computers, originally slated to cost $33 million, but canceled because it couldn’t be made to work even after Deloitte pocketed $330 million; a botched payroll system Deloitte developed for the L.A. Unified School System; and a Deloitte-led, botched ERP system development for Marin County, California, among others. California government IT has been very, very good to Deloitte's bottom line.

Massachusetts admitted a few weeks ago that its original contract with Deloitte to modernize the state's unemployment system was “flawed” and allowed Deloitte  “to miss deadlines and still charge the state some $6 million more than originally planned,” the Boston Globe reported earlier in the month. Suzanne Bump, the Labor Department Secretary in charge of the contract at the time—who is currently State Auditor—told the Globe she has “no recollection of what was in that contract language.”

It is always amazing how selective amnesia seems to strike government officials whenever they are asked to explain their unexplainable decisions.

Jeep’s Transmission Problem Fixed

Usually, U.S. auto manufacturers work overtime to ship new model year vehicles to their dealers in August in order to take advantage of the big September Labor Day sales weekend. This typically means a ramp up in production during the summer months at the automobile production plants. So it was a surprise when early last week, Chrysler announced that it was temporarily laying off the several hundred workers it added just in mid-August at its Toledo Assembly Complex where its new and highly anticipated 2014 Jeep Cherokee was being produced. Chrysler told Automotive News that it had produced a “critical number” of vehicles, and it didn’t want to place too much of a strain on its logistic partners by shipping them out.

Chrysler’s explanation didn’t make any sense to many observers because Chrysler had been insisting that it would be selling the Cherokees in volume by now. Automotive News did a check and found that “no Cherokees were listed in dealer stocks anywhere in the United States.” The only thing that Chrysler did say to add clarity to its nonsensical statement was that the Cherokees required a “software fix” before the vehicles would be sent to its dealers.

Well, on Thursday, the head scratching ended when Chrysler reversed course and announced that it would soon begin delivering the new Jeep Cherokees to dealers after all. The Wall Street Journal reported that Chrysler had been having “problems tuning [the new vehicle's] nine-speed transmission” which caused the car not to shift as smoothly as expected at different speeds and temperatures.

The Journal reported that earlier this summer, “Chrysler delayed test drives for the media on the Cherokee because it was having trouble working out all the bugs in mating the new nine-speed transmission to the vehicle's engines.” According to Chrysler, the 2014 Cherokee has “the world's first application of a highly technical nine-speed transmission” and with it being “mated to two new engines and three complex 4x4 systems,” it wasn’t surprising that technical issues arose.

The company is updating the Cherokee’s powertrain software, and hopes that dealers will be able to sell the new Cherokees very soon. Those with a long memory may remember that there was a flaw in the transmission software for Chrysler’s new 2008 Jeep Grand Cherokees and Commander SUVs that caused engine stalling. The vehicles were subject to an embarrassing recall in May 2008, something that Chrysler wants to avoid this time around.

Obamacare Public Health Exchanges Cross Their Fingers

Well, tomorrow is when Americans can start to sign up for the public health exchanges being created under the U.S. Patient Protection and Affordable Care Act (pdf). As I have noted previously, a number of states are warning that problems may await those signing up. The list got a little longer last week, when the District of Columbia announced that computer issues have caused its exchange’s opening to be delayed for an unspecified amount of time.

A story today at the New York Times provides a good overview of the situation at several state exchanges, which, in a word, can be described as “tense.” Another word that could be applied is “uncertain.” For instance, Rocky King, the executive director of Oregon’s new health insurance exchange, which has delayed the full operations of its exchange to engage in more testing, told the Times that even when the exchange goes live with limited functionality tomorrow, “We could crash and burn and have to close it down.”

No one really will know until tomorrow how many of the exchanges are ready for prime time. We’ll report on the good, bad and ugly in next week’s IT Hiccups review.

Of Other Interest ...

New York City Targets Risky Taxi Drivers after Computer Glitch

Heathrow Passengers Stranded by Baggage Glitch

Apple Maps Glitch Sends Drivers onto Alaskan Airport Runway

Google Gmail Glitch Hits Millions

Wendy's Restaurant Glitch Charges Customers Multiple Times

UK Nationwide Bank Shows Customer Accounts as Bring Empty

Orbital’s Cygnus Docks With ISS After Software Fixed

Photo: Photo: Rich Pedroncelli/AP Photo

This Week in Cybercrime: NSA Wants More Info from Firms

You have to give it to Gen. Keith Alexander, head of the U.S. National Security Agency (NSA). The man can stand up to abuse. He’s faced the ire of attendees at public events ever since the spy agency's monitoring U.S. citizens’ electronic communications was leaked earlier this year. The aftermath of Wednesday’s keynote address at the Billington Cybersecurity Summit, where he called upon the private sector to partner with the NSA, the FBI, the Department of Homeland Security, and the CIA to prevent or limit cybercrime was no different. He couldn’t possibly have expected any different after he said, "We need the authority for us to share [cyberattack information] with [private businesses] and them to share with us."

Despite revelations about the NSA’s activities—some that directly contradict previous government assurances about the limits of the surveillance programs—Alexander insisted that the NSA hasn’t done anything illegal. Furthermore, he said, the calls from some members of Congress to limit the reach of the NSA and the nation’s other spy and law enforcement agencies are based on what he calls sensationalized reporting. Alexander pushed for even more data access from U.S. companies. The more information companies shared with NSA the more cyberattack warnings it could supply to them.

But many observers now see that rationale as threadbare and view Alexander and his ilk with a jaundiced eye. Jerry Brito, a researcher who heads the Technology Policy Program at the Mercatus Center at George Mason University, in Virginia, told CSO that the NSA already has the authority to share data with companies. It could simply declassify information, allowing companies to use it to protect themselves. But that’s not what the agency is interested in, Brito insists. "What they really want is more information about the communications of Americans under the rubric of cybersecurity information sharing," he told CSO.

Stolen Data Clearinghouse Gets Info from Its Above-Board Counterparts

Willie Sutton’s famous response to being asked why he robbed banks—“Because that’s where the money is”—could certainly be the rationale behind a recently discovered cybercrime program targeting data brokerage firms. According to an investigative report [pdf] from security reporter Brian Krebs, an online identity theft service that specializes in selling Social Security numbers, credit and background check reports, and other information, gained access to the data by hacking into the networks of companies such as LexisNexis, Dun & Bradstreet, and an employment background screening company called Kroll Background America Inc. Botnets in the companies’ systems continually siphoned off information and passed it to servers controlled by the cybercrooks.

The criminal clearinghouse, whose website was at SSNDOB[dot]MS, had served some 1300 customers who paid hundreds of thousands of dollars to get their hands on the SSNs, birth dates, drivers license records, and the credit and background check information of more than four million U.S. residents.

Researchers Identify Source of Hit and Run Cyberattacks

Security researchers at Kaspersky Lab say they have uncovered details related to a series of “hit and run” attacks against very specific targets. In a blog post on Kaspersky’s Securelist blog, the researchers said, “We believe this is a relatively small group of attackers that are going after the supply chain—targeting government institutions, military contractors, maritime and ship-building groups, telecom operators, satellite operators, industrial and high technology companies and mass media, mainly in South Korea and Japan.”

What’s most unique about the data theft campaign, which Kaspersky calls “Icefog,” is that after the attackers get what they want, they don’t hang around, using the backdoors installed on the victims’ networks to continue exfiltrating data. They go in knowing exactly what they’re after, take only the target information, then sweep up, turn off the lights, and close the door behind them.

Kaspersky Lab said that it has observed more than 4000 unique infected IPs and hundreds of victims. Some of the companies targeted during the operation, which began in 2011, include defense industry contractors Lig Nex1 and Selectron Industrial Company, shipbuilders such as DSME Tech, and Hanjin Heavy Industries, telecommunications firms such as Korea Telecom, and even the Japanese House of Representatives and the House of Councillors.

Kaspersky has since published a full report (pdf) with a detailed description of the backdoors and other malicious tools used in Icefog, along with a list of ways to tell whether your system has been compromised. The researchers have also put up an FAQ page.

iPhone Break-ins and Countermeasures

Someone tinkering with his Apple iPhone figured out a way to bypass its lock screen, the first line of security for the gadget other than keeping it in your pocket. This week, Apple released its latest countermeasure: an iOS 7 software update that fixes the security hole that allowed an unauthorized user to access information including the handset owner’s e-mail, Twitter, Facebook, and Flickr accounts.

According to Forbes' Andy Greenberg, “swiping upwards on the lockscreen to bring up the iOS Control Center, then opening the alarm clock app, then holding down the power button to show the ‘power off’ and ‘cancel’ options, then tapping ‘cancel,’ and finally quickly double-clicking the home button to bring up the multitasking screen for various apps,” made those apps accessible.

It’s amazing what people with loads of time on their hands eventually stumble upon.

That news came the same week it was revealed that someone had found an even more involved method for fooling the iPhone 5’s fingerprint sensor. According to Marc Rogers, a researcher at the mobile security firm Lookout, it’s possible but highly unlikely that you’ll be the victim of his hack, which he detailed in a blog post (“Why I Hacked Apple’s TouchID, And Still Think It Is Awesome.”). To give you an idea of just how remote the possibility of your phone being duped using his technique, here are a few of the steps Rogers mentions: “You take the cleaned print image and without inverting it, print it to transparency film. Next, you take the transparency film and use it to expose some thick copper clad photosensitive PCB board that’s commonly used in amateur electrical projects. After developing the image on the PCB using special chemicals, you put the PCB through a process called ‘etching’ which washes away all of the exposed copper leaving behind a fingerprint mold.”

In other words, you can rest easy.

Photo:Charles Dharapak/Associated Press

IT Hiccups of the Week: S&P Warns Stock Exchanges Shape Up Their IT or Else

Last week was much like the week before, with U.S. state governments still struggling to make their newly modernized unemployment insurance systems work correctly. We’ll get to them shortly, but we'll start off with a warning by the financial rating services company Standard & Poor’s (S&P) to the world’s stock exchanges over their continuing “technical snafus.”

S&P Tells Exchanges Downgrades in Store if Technical Problems Continue

In a warning shot across the bows of all the world’s stock exchanges, the rating services company S&P published a report last week with the very unsubtle title, “Exchanges' Technical Glitches Reveal Growing Operational Risk—And Could Trigger Downgrades.” The report outlined S&P’s concern that the continuing and seemingly increasing operational problems the exchanges are experiencing—including yet another one, almost on cue last week, at the Options Price Reporting Authority—would mean that the ratings agency might have to factor in a higher level of operational risk in its assessment of an exchange’s credit worthiness. That, in turn, could trigger a downgrade in an exchange’s credit rating, especially if it appears the exchange doesn’t maintain sufficient liquid capital to cover a major operational meltdown, including so-called black swan events.

The S&P report includes a nice table of 25 operational "snafus," as it likes to call them, that happened between March 2012 and August 2013. There have also been at least three more this month (including the aforementioned one at the Options Price Reporting Authority) that could have made the list. We have covered most of the technical problems here at the Risk Factor, including many that occurred before 2012, and were therefore not included in the list. Among these were the multiple technical problems that plagued the London Stock Exchange and the New York Stock Exchange.

The S&P warning came in the wake of the U.S. Security and Exchange Commission telling the exchanges basically the same thing in not so many words. The SEC gave the exchanges 60 days to come up with an approach that would increase their resilience and improve their management of operational risks that turn into operational problems or crises. The government didn't overtly threaten the exchanges, but it was definitely implied that penalties might be forthcoming if the exchanges don't get their act together.

The S&P concluded its report in this way: “We haven't lowered any ratings as a result of these issues so far, but exchanges whose operational risk exceeds industry averages could see rating actions down the road.”

Well, neither the exchanges nor their CIOs can say they weren’t given sufficient warning.

Obamacare Health Insurance Exchanges Experiencing Tech Issues

Next week,  on 1 October, the public health exchanges being created under the U.S. Patient Protection and Affordable Care Act (pdf) are slated to open. As I recently mentioned, concerns are being voiced over the security, or presumed lack thereof, of the federal data services hub as well as the state exchanges. In August, a Forbes article claimed that a major reason behind the security concerns is that, “The administration knows that if the exchanges don’t open on time, there will be a ton of bad press. So they are much more likely to attempt to launch the exchanges without adequate privacy safeguards.”

Whether IT security is lacking or not may be debatable (the Federal government claims that security hasn’t been comprised, at least at its end), the intense pressure to go live next Tuesday is not. For instance, April Todd-Malmlov, the head of Minnesota’s new health exchange, admitted last week to the MinnPost that, “The schedule that we have is essentially putting a five- to 10-year IT project within a two- to three-year timeframe.”

An 11 September risk assessment of the Minnesota Department of Human Services' efforts to implement the exchange stated that, “The schedule will remain [urgent] due to tight timelines with no slack… There is little to no room for vendors to miss code drop dates, and we are severely lacking in adequate testing time.” The assessment lists the project as “red” or “at risk” of not being able to go live next week, but Todd-Malmlov brushed that assessment aside, telling the MinnPost that, “The entirety of the project scope has always been red — at least one part of it has always been red… It’s an indicator for us as to where to focus efforts.” She added that, “I would guess that every state has that same ‘red’ on [their risk assessments].”

Iowa’s Gov.Terry Branstead recently said his state’s health insurance exchange would be open on-time, but added that state officials “are going to try our level best to try to make it work.” There is nothing like a strong vote of confidence to reassure everyone.

The Wall Street Journal added to Branstead’s vote of confidence in an article last week. The WSJ reported that “the government's software can't reliably determine how much people need to pay for coverage” in the health exchanges in the 36 states where the federal government is running all or part of the exchanges. It quoted one senior health-insurance executive as saying, “There's a blanket acknowledgment that rates are being calculated incorrectly… Our tech and operations people are very concerned about the problems they're seeing and the potential of them to stick around.”

Government officials, however, downplayed the issue, stating that, “We continue working with [insurers] and we are confident that on Oct. 1, consumers will see accurate premium costs, including tax credits.” They did admit to the WSJ that, “We may encounter some bumps when open enrollment begins,” but also added that, “we’ll solve them.”

You could almost see those government officials making that last statement while crossing their fingers on both hands.

Oregon, which is one of 14 states that are running their own health insurance exchanges, announced in early August that it was pushing back the 1 October initial enrollment date by two to four weeks in order to give it time to ensure that any issues with its “technology, customer service and other internal processes” could be “ironed out” before going live. That was a move I bet many state health exchanges wish they could also take, but can't politically.

As the health insurance executive stated earlier, I expect this story will not be going away anytime soon. Next Tuesday ought to be pretty interesting from both technical and political perspectives.

California’s Unemployment Insurance System Goes from Bad to Worse

I wrote last week about the problems that Nevada and California were having with their new unemployment insurance systems. While Nevada claims to have finally started to get a handle on the worst of its technical problems (although not everyone agrees), the problems in California have apparently worsened. California’s Employment Development Department (EDD) had originally stated that only 5000 Californians were having trouble receiving their unemployment checks on time because of problems with its new system. However, the EDD admitted to the Associated Press that the actual number—which it revealed early last week had climbed to 50 000, a staggering increase in itself—had reached 185 000 as of Friday. Some 80 000 residents, the EDD acknowledges, still haven’t received the unemployment checks due them.

In related news, California’s unemployment insurance system problems affected, for the second week in a row, the accuracy of the U.S. Department of Labor's report on the number of Americans receiving unemployment benefits. If this keeps up, all of September's unemployment numbers may have to be discarded. The EDD says it’s working hard to fix the problem and promises a major fix over the weekend. But even if the work is successful, it may be weeks before everything, including the federal employment data, is finally straightened out. The EDD did offer its apologies (again) for the delays, which did little to pay the bills of those desperately awaiting their unemployment checks.

Minnesota’s Department of Human Services, which, as I mentioned earlier, has its hands full dealing with the implementation of the state’s Obamacare health insurance exchange, has also had to deal with a multi-day outage of the system used by health care providers to file their Medicaid claims. It isn’t clear how many healthcare providers have been affected, but angry healthcare providers are probably the last thing the agency needs to be dealing with at the moment.

Of Other Interest…

NASA Says Deep Impact Probe Feared Lost After “Software Glitch”

Orbital Sciences' Cygnus Capsule Needs Software Update

Honda Recalls 405 000 Vehicles for Airbag Microprocessor Repair

Arizona Lottery Replaces Tickets After Lottery System Doesn’t Allow Numbers to be Drawn

New DFW Airport Parking System Wildly Overcharges Thousands of Customers

Several Airlines Hit by Navitaire Reservation System Outage

Photo: Scott Eells/Bloomberg/Getty Images


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More