Risk Factor iconRisk Factor

Have Heads Begun to Roll Over Obamacare Imbroglio?

Wednesday morning, as U.S. Health and Human Services (HHS) Secretary Kathleen Sebelius testified about the ongoing problems with healthcare.gov at a Senate Finance Committee hearing, the first head rolled as a result of the Obamacare website debacle. It was announced that Tony Trenkle, the CIO of the Centers for Medicare and Medicaid Services, the HHS entity responsible for creating the healthcare insurance portal, would be leaving his post as of 15 November. According to an e-mail sent to agency employees on Wednesday, Trenkle is “leaving for the private sector”; the agency’s chief operating officer, Michelle Snyder, announced a reshuffling meant to temporarily fill Trenkle’s role.

As for Sebelius, she told senators that, “We’re not where we need to be” with regard to meeting a 30 November deadline set by the administration for making the HealthCare.gov site functional. The HHS secretary revealed that “a couple of hundred functional fixes” would still need be made so the website would not get hung up, display error messages, or make other errors such as displaying blank drop-down boxes. (In this week’s IEEE Spectrum Q&A with risk management expert Robert Charette, he imagined that tackling a punch list containing only 30 items strained credulity.)

Read More

The Obamacare Rollout: What Really Happened?

By now, everyone with access to a newspaper or an Internet connection knows just how badly the first few weeks have gone for the U.S. healthcare insurance portal HealthCare.gov and some of the independent state-run insurance marketplaces. To say we are experiencing technical difficulties is an understatement. Despite testimony last week from the main contractors for the federal website and from Health and Human Services Secretary Kathleen Sebelius, we still have few details about what precipitated such a colossal failure and why no alarms went off before the site's ill-fated 1 October debut. IEEE Spectrum Assistant Editor Willie D. Jones talks with risk management expert Robert Charette about what likely happened and what we can expect going forward.

Jones: You wrote an article for IEEE Spectrum in 2005 that could have been used as the playbook for the HealthCare.gov debacle. I guess you’re somewhat of a prophet.

Charette: You can go back to the first book I wrote on software systems risk management back in 1989. It’s easy to make any project fail. You just don’t give it enough time, enough money, or requirements that are understandable.

Jones: Last week, Health and Human Services Secretary Kathleen Sebelius assured her inquisitors at a congressional hearing that her department has brought in experts that have a handle on the problems the site is facing. How confident should we be in Sebelius’ assurances?

Charette: Not very. They’re talking about dozens and dozens of items on their punch list—both in terms of functionality and performance issues. They’ve got just over 30 days to get through the list. Let’s just say that there are 30 items on it. What do you think is the actual probability of getting through testing them, making sure that the system works end to end and that there are no security holes all in a single month? How do you expect to get that done, knowing that every time you make a fix, there’s a high probability that you’re going to introduce an error somewhere else?

Jones: Let’s spin this forward a bit. How do you think this next month will actually go?

Charette: They said that they needed five weeks at the minimum to test it, and they’re still making all these changes. Where will that five-week window fit? If they had stopped right then and tested it for five weeks, they wouldn’t have been able to finish on time. And five weeks was probably the absolute minimum they needed, assuming everything worked. They’re patching the system as they go along and as Sebelius admitted, they’re doing very local unit tests (which, by the way, is what got them into this mess in the first place, with each contractor saying, "Well, my stuff works"). If they discover something major, they may have to run the whole system test again.

Read More

Nasdaq Experiences A Double Whammy

IT Hiccups of the WeekThe problems with the Affordable Care Act website and some of the ACA’s fine print again dominated the IT-related snafus, hitches and glitches news cycle last week. The list of ACA issues is too long to fully enumerate here, but it included the snarls that led to a miniscule sign-up on the first days of the website going live as well as worrisome security-cum-privacy issues, strange data errors, and politically-driven decisions that continue to haunt the IT implementation.

However, while the ACA brouhaha was last week’s headline IT comedy act, there were several other acts on the IT hiccups playbill that this reviewer found just as amusing, including Nasdaq’s double failure act, Dell’s Latitude laptop imitation of a cat’s litter box, and the Honda Odyssey minivan’s unexpected self-braking routine.

Stock Markets Receive Yet Another Reputation Hit  

Nasdaq Blames Human Error for 45-minute Trading Snafu

Second Error of Week Shuts Down Nasdaq Option Market For Nearly All Day

Nasdaq Problems Earlier in Year Sparked Risk Discussion, But Zero Action

Nairobi Securities Exchange Suffers “Just a Technical Hitch” For Second Time in Month

Dell Says It Will Solve Cat Pee Problem

Owners of Dell’s Latitude 6430u Laptops Say They 'Smell of Cat Urine'

Dell Assures Cat Urine Smell is Non-Hazardous and Not from a Cat

Dell Says It Has Fix for Cat Urine Smell

Bad Week for Car Electronics

Honda Recalls 344 000 Odyssey Minivans in U.S. Because Brakes Might Engage On Own

Honda 2007-2008 Odyssey Recall Notice (pdf)

Nissan Recalls 251 000 in Japan Due to Faulty Engine Control Unit

Nissan Reshuffles Management over Paltry Profit Partly Due to Big Recalls

Ford Recalls Ford Focus EV Due to Powertrain Control Module Software Issue

Consumer Reports Dings Ford over Continuing MyFord Touch Problems

Of Other Interest …

Gov. Jerry Brown Blames “Screw-ups” for California EDD System Problems

California EDD System Leaders Leave before Deloitte’s System Problems Emerge (video)

Florida’s New Deloitte-Built Unemployment System Continues to Have Problems

Florida’s Sen. Nelson Calls for Federal Investigation into Malfunctioning Unemployment System

Deloitte Tries to Defend Botched Massachusetts Unemployment System Implementation

Victoria’s Auditor-General Blames Medication Errors on Computer System Problems

US Cellular Loses Customers over Billing Issues

Network Issues Delay Flights at India’s Kolkata Airport

Network Problems Affect Saipan’s Bureau of Motor Vehicle

Barclay’s UK Bank Suffers “Technical Difficulties”

ANZ Bank Fixes Payment Problem

F-35 Helmet Contracted Ended Because of Dangerous Technical Glitches

Photo: Seth Wenig/AP Photo

US State Governments Can’t Shake IT Woes

IT Hiccups of the Week Most of the news involving last week’s IT-related problems, snarls and snags were once again drowned out by media stories concerning the recognition if not admission of major management blunders that led to the “glitches” in the Affordable Care Act website and its supporting systems. The Obama Administration now says that the two dozen or so major items on its IT “punch list” will be fixed and thoroughly tested within the next 33 days or so, but more than a few folks are willing to bet against that happening, especially in regard to the issue of data security.

Residents of California, Florida, Alaska, Pennsylvania, New Mexico, Kentucky, Michigan and Mississippi are among those fervently hoping that their states' ongoing government computer woes will be over by the end of November. However, I suspect that for many of those state residents, it will prove to be a forlorn hope as well.

US State Governments’ IT System Problems Persist

Unemployed Californian’s Continue to Complain of EDD Unemployment Computer System Problems

California Legislators Set Hearings into EDD System Troubles for Next Week

Florida’s New Connect Unemployment System Woes

Florida Trying to Fix New Connect Unemployment System

Alaska’s New Medicaid System Undergoing “Growing Pains”

Pennsylvania’s Worker’s Compensation System Problems Linger

New Mexico’s New Unemployment System Ops Improved, but Must Get Better

Kentucky Medicaid Billing System Problems Finally Clearing Up After Two Years

Call Volume Unusually High after Michigan Upgraded Unemployment System

Mississippi’s New Online Tax System Causing Headaches

Microsoft’s Less Than Smooth Windows RT 8.1 Update

Windows RT 8.1 Update Has Some “Show-Stopping” Installation Problems

Microsoft Pulls Windows RT 8.1 Update From Store

Microsoft  Fixes Windows RT 8.1 Update and Is Back in Store

What Does “Glitch” Exactly Mean?

A Brief Linguistic History of the Term “Glitch

Of Other Interest…

Facebook Suffers Worldwide Outage

Network Solution Experiences Outage for Third Time

US Customer and Border Protection Computer System Goes Offline

Cincinnati Bell Internet Goes Down

System Glitch Interrupts 911 Emergency Calls in Douglass County, Nebraska

Dallas Ft. Worth International Airport Traffic and Parking System Malfunctions Again

Nissan Recalling 152 000 SUVs for Anti-Lock Brake Software Upgrade

Credit Card System Problem Stiffing Washington, D.C. Taxi Drivers

Los Angeles Department of Works Computer Error Leads to Higher Water Bills

Schwab Trading Platform Experiences Problems after Integration with Third-Party System

Incompatible Hardware and Software Bring Beijing’s Subway Line 10 to a Halt

Photo: Ugurhan Betin/iStockphoto

DARPA Seeks Self-Healing Networks

This Week in Cybercrime DARPA, the U.S. military’s R&D arm, announced this week that it will pay US $2 million to the winner of its Cyber Grand Challenge, a contest aimed at developing an automated network defense system that actively searches for and identifies vulnerabilities and patches them on the fly. “Today, our time to patch a newly discovered security flaw is measured in days,” Mike Walker, DARPA program manager, told Kaspersky Threatpost. “Through automatic recognition and remediation of software flaws, the term for a new cyberattack may change from zero-day to zero-second,” says Walker.

Read More

Corporate Recruiters Insist There Really Is a STEM Worker Shortage

The Wall Street Journal published a story yesterday titled,  “More Businesses Want Workers With Math or Science Degrees” that highlights a new STEM skills shortage study. The article states that:

Bayer Corp., the U.S. arm of the German chemical and pharmaceutical giant Bayer AG, is due to release a report this week showing that half of the recruiters from large U.S. companies surveyed couldn't find enough job candidates with four-year STEM degrees in a timely manner; some said that had led to more recruitment of foreigners.

The shortages were most acute in engineering and computer-related fields, the recruiters said. The survey, completed in August, included 150 recruiters from 117 companies, all on the Fortune 1000 list of large companies.

About two-thirds of the recruiters surveyed said their companies were creating more STEM positions than other types of jobs.

So, to be clear, half the recruiters in 117 companies (assuming no double counting of recruiters) say that they have trouble hiring STEM workers quickly enough. Oh my, that is terrible! Of course, left unsaid, half of the recruiters surveyed apparently aren't having any such difficulty in their hiring, which sort of undermines the notion that there is much of a shortage.

In addition, it's difficult to determine from the WSJ article exactly how much the recruiters were willing to offer in terms of salaries and benefits to those oh-so-hard-to-find STEM workers. Maybe those companies that are complaining about a STEM skill shortage should try emulating Netflix, which is willing pay a little bit extra to get the talent it needs. It doesn't seem to complain about a skills shortage.

Or maybe, as this new Ed. D. dissertation from the University of Pennsylvania reported, recruiters are having trouble because "employers have a requirement for experience for new [STEM] hires." The dissertation research found contrary to reports, there "was not a shortage of new STEM graduates in Ohio."

Furthermore, the recruiters' decision to recruit foreigners because they couldn't quickly find the right STEM skills in the United States has to be taken with a very large grain of salt as well. As Silicon Valley recruiting company Bright.com admitted over the summer, it could find only a handful of computer-related jobs in the Valley where a skills shortage that might justify hiring guest workers can legitimately be claimed to exist.

More details can be found in the Bayer press release and accompanying  report which was disclosed during the Bayer-sponsored “debate” being held today in Washington, D.C. to discuss these “shortages” the corporate recruiters are supposedly having. From the skewed questions (e.g., Are unfilled STEM jobs bad for business?) asked the embarrassing small number of recruiters who bothered to answer the survey in the report, it is easy to see that the exercise was all about whipping up support for the notion that the STEM Crisis is not a myth (as I strongly contend) and that more government money needs to be committed to the efforts (of those on the debate panel) to eliminate the terrible STEM skills shortage plaguing the United States.

The survey results, by the way, have a  +/- 8 percent margin of error at a 95 percent confidence level.

Of course, recruiters and their employers have long been whining about students not having the right science and math skills, as can be seen in this previous Bayer report from 1997. Bayer should have just republished it with a 2013 date to save itself some time and money.

Interestingly, the most recent Bayer study and its findings sound very similar to a doom and gloom piece printed in American Airlines American Way magazine from 2003 in which was cited a projection by the Bureau of Labor Statistics that the United States would be short 10 million workers by 2010. Funny, that shortage didn’t seem to have happened.

Furthermore, the American Way article quoted Norman Maas, at the time the North American senior vice president of human resources for German chemical company BASF,  saying that he:

... figures that by 2010 he’ll have to replace about 75 percent of BASF Corporation’s 13 000 workers in North America. What he can’t figure is where he’s going to get them. Especially when it comes to finding large numbers of highly skilled chemical engineers and managers capable of overseeing a diverse, multilingual workforce.

As senior vice president of human resources for the $8.2 billion chemical company, Maas is not expecting an easy decade.

“The size of the pool gets smaller and smaller, and the demand for those skills gets bigger and bigger," Maas laments. "So you have more companies competing for a smaller and smaller group of talented people.”

Hmm … checking BASF’s annual report for North American in 2010 (pdf), the company listed 16 487 employees with a turnover of 11.2 billion euros (or about US $15.3 billion).

The 2010 BASF annual report doesn’t mention a skills shortage, but does note: “Like many companies, we are experiencing significant demographic shifts. Many of our employees are potentially approaching retirement; 'next generation' employees are entering the workforce with new expectations and ways of working; and ‘minorities’ are becoming majorities in the pools of talent coming out of colleges and universities and across our customer base. Faced with many changes coming together at the same time, we are taking advantage of once-in-a-generation opportunities to transform our workforce and gain competitive advantages.”

So, the dire problem that Maas was concerned with instead actually turned out to be a once-in-a-generation opportunity for BASF; imagine that.

I expect that the vast majority of recruiters surveyed in the most recent Bayer report who claim a STEM skill shortage are buying into overly pessimistic spin on finding STEM skills just as Maas did. And of course, what better way to have a built in excuse if you are unsuccessful at hiring new STEM employees or to look like a hero if you are able to overcome the perils of such a dire shortage?

Photo: iStockphoto

Spiders Trap Toyotas

IT Hiccups of the WeekThe “Apple-like glitches” affecting the Affordable Care Act website and its supporting systems continue to dominate the news, especially with the Obama Administration’s admission over the weekend that the “best and brightest” IT cavalry needs to be called in to rescue it, if that is even possible. The on-going issue will be explored in more depth in the Risk Factor at a later time.  

While it is extremely difficult to turn our eyes away from the slow motion ACA IT train wreck, there were other IT derailments of interest last week as well, including spiders that cause Toyota airbags to unexpectedly deploy (spiders also caused problems with Mazda vehicles a few years ago), another airline reservation system meltdown, this time affecting easyJet in the UK, and Florida’s shaky start to its new unemployment insurance system.

Spiders Like Toyota, But the Feeling Isn’t Mutual

Spiders Force Toyota to Recall 800 000 Cars

Spiders Blocking AC Unit, Force Recall of 870 000 Toyotas

Toyota Recalls 885 000 Vehicles, Spiders Get Part of the Blame

easyJet Suffers European-wide System Failure

easyJet Reservation System Crashes

Technical Issue Hits easyJet Reservation System

easyJet Faces Big Compensation Claim for Reservation System Problems

Florida’s New Unemployment System Bumpy Start

Problems Persist in New Unemployment Claims System

More Phone Lines Opened to Handle Unemployment Claims

Deloitte-Designed Florida Unemployment System Draws Fire

State Downplaying Problems with New Unemployment Claims System

Of Other Interest…

Level 3 Outage Affects East Coast Internet Traffic for 24 Hours

California EDD Refuses to Release Documents on Broken Unemployment Computer System

Long Distance Bus Service in India Hit by Week Long Ticketing Glitch

Software Issue Delays New Park-and-Display Parking Meters in Little Rock, Arkansas

Australian Telecom Optus Refunds A$8.8 Million to 235 000 Customers for 2 Year Billing Error

United Airlines Says It Wasn’t a Glitch and Cancels “Free Tickets” This Time

Photo: iStockphoto

Obamacare Data Hub Security Faces Scrutiny

We already know that the HealthCare.gov website and many of the state-run healthcare insurance exchanges created as part of the Obamacare rollout hit the ground crawling on 1 October. There is much left to be said about the players in that still-unfolding debacle. But there’s an element of the drama—namely data security—that will likely get increasing attention as fixes aimed at letting the uninsured sign up for coverage allows millions of people to finally input their personal information. But Congress is wasting no time: Members of Congress want answers about Obamacare Data Hub security measures

 

In Other Cybercrime News…

  • Hackers broke into a database at PR Newswire that contained login credentials and contact information for the press release distribution service’s customers—tens of thousands of companies and public relations agencies

Image: iStock

California’s EDD Unemployment System Disaster: Predictable Fiasco?

The unemployment insurance payment system fiasco presided over by California’s Employment Development Department (EDD) just keeps getting more bizarre by the week. California has the largest unemployment system in the U.S., distributing about $33 million a day in unemployment checks to some 800 000 claimants when everything is working properly. This week it was revealed that EDD management knew its new unemployment system contained potentially major operational flaws, but decided to roll it out anyway.

As I noted last month, the EDD has spent US $157.8 million upgrading the state’s 30-year-old unemployment payment processing system. The upgraded system was originally supposed to cost $35 million and go live in 2009 (pdf), but a series of ridiculously incomplete (if not incompetent) definitions of the system’s requirements by EDD officials significantly pushed up the project’s cost and delayed its delivery schedule.

The EDD finally transitioned to its upgraded unemployment payment system over this year's Labor Day Holiday weekend (31 August to 2 September). There were a few reported hiccups with the transition at that time, but nothing seemed too far amiss during the first week after the switch. In fact, EDD officials reportedly were congratulating themselves on a job well done.

However, by the second week after the transition, it became increasingly clear that there were the proverbial "technical issues" cropping up within the unemployment system. The EDD announced that while the new “system is working as designed,” there were “some processing delays in [the] transition” from the old to new benefits system.  Nonetheless, the EDD said that it was “working around the clock to catch up on unemployment claims.” A San Francisco Chronicle story quoted an EDD spokesperson who claimed only about 5 percent of the unemployment benefit claims—or about 20 000—were negatively affected by the upgrade's technical issues. She added that, “We apologize for the inconvenience to those affected.”

A few days later, though, the EDD started to hint to the press that the problem might be a bit bigger than it first let on. The EDD said that the new system was misreading legacy beneficiary data, and that it didn’t realize the issue would be so widespread. Rule number one in any transition from a legacy system to its replacement is to ensure that existing data is clean, consistent and complete so that it can be imported into the new system with little difficulty as possible. So that admission of wide-spread data quality issues was not an especially good sign. Word was also filtering out from EDD employees that the scope of the problem was in fact huge, but the EDD wouldn’t dignify those claims by making a comment.

Eventually, the EDD admitted that the number of affected claimants was closer to 50 000 instead of its previously estimated 20 000. The agency reiterated that its employees were “working around the clock and through the weekends to try and get these payments issued for the customers eligible and waiting for benefits.”

A few days later, the EDD revised its numbers upwards again, suddenly acknowledging that 185 000 unemployed Californians had been impacted by the system’s operational problems and that 80 000 claimants still hadn’t received their checks three weeks after the upgraded system went live.

As the debacle entered its fourth week, Marty Morgenstern, California's Secretary of Labor and Workforce Development, finally weighed in, saying that the continuing delays were “unacceptable.” With Gov. Jerry Brown’s support, an order was given “to immediately begin the process of paying backlogged claims for continued UI [unemployment insurance] benefits prior to a final determination of eligibility.”

Morgenstern’s order helped whittle down the backlog, and by the middle of the next week, the EDD officially declared victory (pdf). The agency proudly proclaimed during the first week of October that its “aggressive efforts succeed[ed] in eliminating our backlog of certifications.” That is an interesting choice of words, since the backlog was mostly eliminated by Morgensten allowing a bypass around a core system requirement.  Even so, EDD’s claims to have eliminated the backlog were immediately challenged by many unemployed Californians complaining that they still hadn’t received payments due them, a story at the LA Times stated.

Then another problem surfaced: apparently the claim forms needed to keep unemployment benefits flowing to beneficiaries went missing, the LA Times reported in a separate story. The Times reported that when queried about the missing forms, an EDD spokesperson initially “side-stepped” the question, then gave what amounted to a non-answer, and then wouldn’t respond to further questions from the newspaper about the missing forms.

The ongoing problems had, by early October, spurred a number of state legislators to call for an investigation into the system’s contractor, Deloitte Consulting. As I have previously described in more detail, EDD gave the contract to Deloitte despite the company having a long history of troubled IT projects across California state and local governments.

The most recent bombshell hit this Monday, when the Sacramento Bee followed up on a story first reported by television station KXTV in Sacramento last Friday. The station along with the Bee reported that the new unemployment system was "broken from the start," that EDD officials knew it was broken, and that they moved forward with the system’s roll out anyway. Furthermore, said the reports, when EDD realized that there was indeed a major problem with the system after it went live, officials tried to blame the press for magnifying the severity of the issue. And when that didn’t work, they turned to blaming budget cuts for the lack of testing performed on the system. Just as damning were the reports' revelations about insider e-mails indicating that as many as 300 000 Californians had been affected by issues with the system, not the 185 000 the EDD was publicly claiming.

EDD officials claimed that tests conducted before the new system was rolled out indicated that all the problems would be manageable. And, furthermore, if the tests had suggested otherwise, EDD officials insisted, the agency would have delayed the system's debut. Yet the KXTV story indicates that the errors found during unit tests in early system builds were routinely being passed onto future system builds without ever being corrected. (In some quarters, this is considered poor software development practice, while in others, it is seen as a good business strategy for ensuring future software maintenance work.) Internal EDD IT programmers who sounded the alarm about this practice of kicking the can down the road were said to have been ignored.

Massachusetts, whose own unemployment system has been having problems as well, recently admitted that its original contract with Deloitte to modernize that state’s $46 million unemployment system was “flawed” and allowed Deloitte  “to miss deadlines and still charge the state some $6 million more than originally planned,” the Boston Globe reported in September. State legislators in Massachusetts announced earlier this month that hearings are going to be held to look into the problems with the Deloitte-built unemployment system's roll out.

Last week, Florida began its move to a new unemployment system also developed (late and over-budget) by Deloitte, and, you guessed it: there are reported problems there as well. It is too soon to tell if Deloitte will face another embarrassing government hearing in Florida, but I wouldn’t bet heavily against it.

These latest disclosures seemed to finally force reluctant senior California politicians this week to agree to hold hearings on the EDD development next month or in December. Hopefully when they do, they’ll remember to ask Deloitte why it keeps insisting that the unemployment system is working and why it believes that none of the problems experienced to date are because of a “breakdown or flaw in the software Deloitte developed.”

Photo: Damian Dovarganes/AP Photo

Talking about the STEM Crisis Myth

Last month’s article “The STEM Crisis Is a Myth,” by IEEE Spectrum contributing editor Robert N. Charette, triggered a hearty response from readers. Many commenters shared his view—that there is no shortage of scientists and engineers—and quite a few were against it. It seemed clear that a discussion of the issue should continue.

And so, on 7 October, IEEE and Arizona State University’s Consortium for Science, Policy and Outcomes, convened a conversation between Charette and CSPO co-director Dan Sarewitz at CSPO’s Washington, D.C., office, just north of Dupont Circle. For those of you held back from attending by the government shutdown, the torrential rain, or the fact that you live nowhere near D.C., we’re posting a video of the hour-long event.

Radio fans can listen to Charette’s recent interview on NPR’s “Here and Now”. And Spectrum’s expanded coverage of the STEM crisis can be found here.

Advertisement

Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Contributor
Willie D. Jones
 
Load More