Detroit's IT Systems “Beyond Fundamentally Broken”

IT Hiccups of the Week

Last week’s IT Hiccups parade was a bit slower than normal, but there were a couple of IT snafus that caught my eye. For instance, there was the embarrassed admission by Los Angeles Unified School District (LAUSD) chief strategic officer Matt Hill that the new-but-still-problem-plagued MiSiS student tracking system I wrote about a few weeks ago should have had “a lot more testing” before it was ever rolled out. There also was the poorly thought out pasta promotion by Olive Garden restaurants that ended up crashing its website. However, what sparked my curiosity most was the disclosure by Beth Niblock, Detroit’s Chief Information Officer, that the city’s IT systems were broken.

How broken are they? According to Niblock:

“Fundamentally broken, or beyond fundamentally broken. In some cases, fundamentally broken would be good.”

Niblock’s comment was part of her testimony during Detroit’s bankruptcy hearings. Last July, Detroit filed bankruptcy and since then has been in bankruptcy court trying to work out debt settlements with its creditors, some of whom are unhappy over the terms the city offered. Niblock was a witness at a court hearing looking into whether the city’s bankruptcy plan was feasible and fair to its many creditors, and whether the plan would put the city on more sound financial and operational footing.

Critical to Detroit returning to financial and operational soundness is the state of the city’s IT systems. However, since the 1990s, the city’s IT systems have generally been a shambles, and that is putting it charitably. Currently, according to Niblock (who took on the CIO job in February after turning it down twice and maybe wishing she did a third time), the city’s IT systems are “atrocious”, “unreliable” and “deficient,”Reuters reported.

Reuters went on to report Niblock's testimony that the city’s Unisys mainframe systems are “so old that they are no longer updated by their developers and have security vulnerabilities.” She added that the desktop computers, which mostly use Windows XP or something older, “take 10 minutes” to boot. It probably doesn’t matter anyway, since the computers run so many different versions of software that city workers can’t share documents or communicate, Niblock says. That also may not be so bad, given that city computers have apparently been infected several times by malware.

Detroit’s financial IT systems are so bad that the city really hasn’t known what it is owed or in turn, what it owes, for years. A Bloomberg News story last year, for example, told the story of a $1 million check from a local school district that wasn’t deposited by Detroit for over a month. During that time, the check sat in a city hall desk drawer. That isn’t surprising, the Bloomberg story noted, as the city has a hard time keeping track of funds electronically wired to it. The financial systems are so poor that city income-tax receipts need to be processed by hand; in fact, some 70 percent of all of the city’s financial accounting entries are still done manually. The costs of doing things manually are staggering: it costs Detroit $62 to process each city paycheck, as opposed to the $18 or so it should cost. Bloomberg stated that a 2012 Internal Revenue Service audit of the city’s tax collection system termed it as being “catastrophic.”

While the financial IT system woes are severe, the fire and police departments' IT systems may be in even worse shape. According to the Detroit ~~News~~ Free Press, there is no citywide computer aided dispatch system to communicate emergency alerts to fire stations. Instead, fire stations receive the alerts by fax machine. To make sure the alarm is actually heard, fire fighters have rigged Radio Shack buzzers and doorbells, among other homemade Rube Goldberg devices that are triggered by the paper coming out of the fax machine. Detroit's Deputy Fire Commissioner told the Detroit ~~News~~ Free Press that, “It sounds unbelievable, but it’s truly what the guys have been doing and dealing with for a long, long time.”

You really need to check out the video accompanying the Detroit ~~News~~ Free Pressstory which shows fire fighters using a soda can filled with coins and screws perched on the edge of the fax machine so that it will be knocked off by the paper coming out of the machine when an emergency alert is received at the fire station. Makes one wonder what happens if the fax runs out of paper.

The Detroit police department's IT infrastructure, what there is of it, isn’t in much better shape. Roughly 300 of its 1150 computers are less than three years old. Apparently even those “modern” computers have not received software updates, and in many cases, the software the police department relies on is no longer supported by vendors. The police lack an automated case management system, which means officers spend untold hours manually filling out, filing, and later trying to find paperwork. Many Detroit police cars also lack basic Mobile Data Computers (MDC), which means officers have to rely on dispatchers to perform even basic functions they should be able to do themselves. An internal review (pdf) of the state of Detroit’s police department was published in January, and it makes for very sad, if not scary, reading.

If you are interested in how Detroit’s IT systems became “beyond fundamentally broken,” there is a great case study that appeared in a 2002 issue of Baseline magazine. It details Detroit’s failed attempt, beginning in 1997, to upgrade and integrate its various payroll, human resources, and financial IT systems into a single be-all Detroit Resource Management System (DRMS) that went by the name “Dreams.” The tale told is a familiar one to Risk Factor readers: attempting to replace 22 computer systems used across 43 city departments with one city-wide system resulted in a massive cost overrun and little to show for it five years on. Crain’s Detroit Business also took a look back at the DRMS implementation nightmare in a July article.

Detroit hopes, the Detroit News reports, that the bankruptcy judge will approve its proposed $101 million IT “get well” plan, which includes $84.8 million for IT upgrades and $16.3 million for additional IT staff. (In February, according to a story in the Detroit ~~News~~ Free Press, the city wanted to invest $150 million, but that amount apparently needed to be scaled back because of budgetary constraints.) Spending $101 million, Niblock admitted, will not buy world-class IT systems, but ones that are, “on the grading scale… a ‘B’ or a B-minus” at best. And Niblock concedes that getting to a “B” grade will require a lot of things going perfectly right, which is not likely to happen.

On one final note, I’d be remiss not to mention that last week was also the 25th anniversary of the infamous Parisian IT Hiccup. For those who don’t remember, in September 1989, some 41,000 Parisians who were guilty of simple traffic offenses were mailed legal notices that accused them of committing everything from manslaughter to hiring prostitutes or both. As a story in the Deseret News from the time noted:

“A man who had made an illegal U-turn on the Champs-Élysées was ordered to pay a $230 fine for using family ties to procure prostitutes and ‘manslaughter by a ship captain and leaving the scene of a crime.’”

Local French officials blamed the problem on “human error by computer operators.”

Plus ça change, plus c'est la même.

In Other News ….

Coding Error Exposes Minnesota Students' Personal Information

Computer Glitch Sounds Air Raid Sirens in Polish Town

Computer Problems Change Florida County Vote Totals

Billing Error Affects Patients at Tennessee Regional Hospital

Dallas Police Department Computer Problems Causing Public Safety Concerns

New York Thruway Near Albany Overbills 35,000 EZ‐Pass Customers

Olive Garden Shoots Self in Foot With Website Promotion

Apple Store Crashes Under iPhone6 Demand

Scandinavian Airlines says Website Now Fixed After Two Days of Trouble