IT Hiccups of the Week: Online Testing Problems Spread

Last week saw the ongoing effects of several IT-related problems initially spotted over the past month. We start with problems several U.S. states have been having with the online testing systems upon which they increasingly rely for carrying out all of their standardized testing.

More States Experience Online Testing Issues

In mid-April, I wrote about how 15 000 Minnesota students who were trying to take their Minnesota Comprehensive Assessment online math test either couldn’t sign in or had their tests ended prematurely. The snag was because of a server issue at the American Institutes for Research (AIR), which the state had hired to run the testing. Those kids weren’t alone; more testing problems cropped up about a week later, frustrating the plans of several other Minnesota school districts, Minnesota Public Radio reported. However, a story at the Minneapolis StarTribune says that AIR is denying that the latter testing problems had anything to do with its system. Nevertheless, the ongoing online testing issues forced the state’s education officials to announce last week that they are pushing back the deadlines by which the exams have to be administered. The math and reading tests were to have been completed by May 10 and science tests by May 17. No new completion dates have yet to be set.

The Minnesota students having trouble taking their online tests apparently have lots of company. According to an Associated Press story, students in Oklahoma, Kentucky and Indiana also have had difficulties with their online tests. Other news reports highlighted testing issues in Ohio and Alabama. Again, server-related issues seemed to be a root cause. The problems in these states were similar to those experienced in Minnesota: students couldn’t log on, suffered slow response times, or were kicked off in the middle of their tests. Testing in Oklahoma and Indiana is run by CTB/McGraw-Hill; in Kentucky, ACT oversees the student assessments.

The problems in Indiana not only have students on edge, but teachers too, as their merit pay is tied to student test scores. McGraw-Hill, which has a four-year, $95 million contract to operate the state’s online testing system, could be fined “$50 000 for each day last week the test was down,” WTRV Channel 6 in Indianapolis reported. McGraw-Hill apologized for the problems students had in both Oklahoma and Indiana, saying “We sincerely regret the problems we have caused,” and offering assurances that everything is fine now. The apology rang pretty hollow to many teachers and parents of students affected by the outages, however.

In Kentucky, the problems have caused state officials to suspend online testing for students who haven’t completed their tests and move back to the tried-and-true, paper-and-pencil approach. According to various news reports, state officials acted after ACT gave conflicting reports on the status of its testing system.

ACT was said to have “apologized for any inconvenience" caused.

CBOE Knew of Software Bug before Meltdown

On 25 April, a “software glitch” at the Chicago Board Options Exchange (CBOE), the largest U.S. options exchange, forced a three-hour delay in trading, the Wall Street Journal reported at the time. The problem came as CBOE executives were in Las Vegas hosting an industry conference. In the immediate aftermath of the outage, the U.S. Securities and Exchange Commission was said by the WSJ to be concerned about the length of the outage, and traders interviewed called it a “big deal” and “unnerving.” After the smoke cleared, the actual damage was characterized as minimal.

A Bloomberg News article reported that the CBOE blamed the outage on an undisclosed “software bug” that was a result of “preliminary work in preparation for reconfiguring its computer systems.” The options exchange was “preparing to change its systems to get ready for extended trading hours on the CBOE Futures Exchange and eventually CBOE options,” Bloomberg stated.

Bill Brodsky, the exchange’s CEO, told Bloomberg that, “Early Thursday morning our team identified and addressed a potential software issue and subsequently believed we were on track for a normal open.” Somewhat controversially, Brodsky then added, “Unfortunately, the nature of a software bug is sometimes only identifiable once the system is operationally ready; such was the case last Thursday at CBOE. As we approached the open, it became apparent that the software issue was not fully resolved, and the decision was made to delay the opening.” Many took this as an admission of poor testing and or contingency management discipline at the CBOE.

According to a Chicago Tribune story, Brodsky explained that the exchange didn’t go to a back-up system because it was worried about “thoroughly preserving the integrity of the orders it already had received that morning.” In addition, the Tribune story quoted a trader who said he was told by an unidentified CBOE staffer that the software bug caused a “corrupt data file load which is necessary to tell all the systems which option contracts are available for trading” that needed to be fixed before trading could start. The CBOE would not confirm that particular explanation of what happened to the Tribune, however.

The outage has caused the CBOE to delay its plans to extend after hours trading until it is “satisfied that there would be no repeat of the technical issues” involved, the WSJ reported on Friday.

New Zealand’s Inland Revenue Department Computer Crashes

In another outage that was predictable if not specifically predicted, the New Zealand Inland Revenue Department’s main computer crashed last Thursday, 2 May. It was down for the morning, but returned to normal that afternoon, the New Zealand Herald reported. The crash’s timing was interesting because just the day before, the Otago Daily Times reported, Inland Revenue Department Minister Peter Dunne “announced the Government's biggest ever overhaul of a Government IT system—a $1.5b upgrade of the department's ‘First mainframe’ computer system.” The department's legacy computer system that needs replacing dates back to 1991.

One of the reasons given for the upgrade, Dunne said, is that  half of the 1100 data entry staff is currently devoted full time to “correcting data entries in the system.” Another reason is that a 2011 internal study (pdf) showed the Revenue Department staff believed the organization was facing "a 40 percent chance of a systems failure that would severely impact its ability to collect and distribute money.” While Thursday's crash impact wasn't major, the event is probably being seen as a near miss "warning" signal of something that could be much worse.

The Revenue Department expects the computer systems upgrade to take 10 years, although longer is also a distinct possibility it said. More systems crashes like last Thursday's are no doubt an increasing possibility as well.

H&R Block Offers $25 Cash Cards In Attempt to Appease Customers

As we noted in March, H&R Block, one of the world's largest tax services providers, had a problem electronically transmitting Form 8863 to the U.S. Internal Revenue Service. As a result, hundreds of thousands of taxpayers had their tax refunds delayed. Due to the mishap, H&R Block was hit with a number of lawsuits claiming, among other things, that affected customers were inadequately compensated for the mistake.

Last week, H&R Block announced that it is sending out a personal apology letter and a US $25 MasterCard cash card to all the customers it had inconvenienced.

There was no word about any of the lawsuits being dropped as a result of the compensation.

Also of interest…

Vancouver’s Port Mann Bridge Computer Glitch Doubles Bridge-toll Bills

Honda Recalls Fit for Stability Control System Problem

Windsor-Detroit Border Crossing Slows Amid Computer System Issue

Bermudan Government Workers’ Pay Delayed by Computer Problems

Australian National Broadband Network Suffers Tech Glitch on Sydney Debut

Photo: Thomas Trutschel/Photothek via Getty Images