Risk Factor iconRisk Factor

LA School District Superintendent’s Resigns in Wake of Continuing MiSiS Woes

We turn our IT Hiccups of the Week attention once again to the Los Angeles Unified School District’s shambolic roll out of its integrated student educational tracking system called My Integrated Student Information Systems (MiSiS). I first wrote about MiSiS a few months ago, and it has proved nothing but trouble to the point that it became a major contributing factor in “encouraging” John Deasy to resign his position last week as superintendent of the second largest school system in the United States. He’d  been on the job three and a half years.

Deasy claimed in interviews after his resignation that the MiSiS debacle “played no role” in his resignation, and instead blamed it on district teachers and their unions opposing his crusading efforts to modernize the LAUSD school system. That is putting a positive spin on the situation to put it mildly.

Why? You may recall from my previous post that LAUSD has been under a 2003 federal district court approved consent decree to implement an automated student tracking system so that disabled and special need students’ educational progress can be assessed and tracked from kindergarten to the end of high school. Headway toward complying with the obligations agreed under the consent decree is assessed by a court-appointed independent monitor who publishes periodic progress reports. Deasy repeatedly failed to deliver on the school district’s promises made to the independent monitor over the course of his tenure.

What really helped seal Deasy’s fate was the latest progress report [pdf] from the independent monitor released last week. The report essentially said that despite numerous “trust me” promises by LAUSD officials (including Deasy), MiSiS was still out of compliance. The officials had promised that MiSiS would be completely operationally tested and ready at the beginning of this school year. But, said the report, the system’s incomplete functionality, the ongoing poor reliability due to inadequate testing, and the misunderstood and pernicious data integrity issues were causing unacceptable educational hardships to way too many LAUSD students—especially to those with special educational needs.

An LA Times story, for one, stated that the monitor found that MiSiS, instead of helping special needs students, made it difficult to place them in their required programs. A survey conducted by the independent monitor of 201 LAUSD schools trying to use MiSiS found that “more than 80% had trouble identifying students with special needs and more than two-thirds had difficulty placing students in the right programs,” the Times article stated.

Deasy’s fate had been hanging by a thread for a while. For instance, at several LAUSD schools—especially at Thomas Jefferson High School in south Los Angeles—hundreds of students were still without correct class schedules nearly two months after the school year had started. 

Another story in the LA Times reported that continuing operational issues with MiSiS meant that some Jefferson students were being “sent to overbooked classrooms or were given the same course multiple times a day. Others were assigned to ‘service’ periods where they did nothing at all. Still others were sent home.”

The problems at Jefferson made Deasy’s insistence that issues with MiSiS were merely a matter of “fine tuning” look disingenuous at best.

The MiSiS fueled difficulties at Jefferson, which extended to several other LAUSD  schools, caused a California Superior Court judge about two weeks ago to intervene and order the state education department to work with LAUSD officials to rectify the situation immediately. In issuing the order, the judge damningly wrote that, “there is no evidence of any organized effort to help those students” at Jefferson by LAUSD senior officials.

As a result of the judge’s order, the LAUSD school board last week quickly approved a $1.1 million plan to try to eliminate the disarray at Jefferson High. Additionally, the school board is now undertaking an audit of other district high schools to see how many other students are being impacted by the MiSiS mess and what additional financial resources may be needed to eliminate it.

Fraying Deasy’s already thin thread further was his admission that MiSiS is in need of some 600 enhancements and bug fixes (up from a reported 150 or so when the system was rolled out in August), which would likely cost millions of dollars on top of the $130 million already spent to address them. Further, he also acknowledged that one of the core functions solemnly promised to the independent monitor would be available for this school year—the proper recording of student grades—could take yet another year to fix all the bugs with it, the LA Times reported.

According to the LA Daily News, LAUSD teachers complain that they not only have a hard time accessing the grade book function, but when they finally do, they find that student grades or even their courses have disappeared from MiSiS. Hundreds if not thousands of student transcripts could be complete shambles, which for seniors applying for colleges is causing major concern. Their parents are also unamused, to say the least.

Probably the last fiber of Deasy’s thread was pulled away last week when it turned out that even if MiSiS had been working properly, a majority of LAUSD schools likely wouldn’t have been able to access all of its functionality anyway. According to a story at Contra Costa Times, LAUSD technology director Ron Chandler informed the district’s school board last week that most of the LAUSD schools’ administrative desktop computers were incapable of completely accessing MiSiS because of known compatibility problems.

A clearly frustrated school board wanted to know why this situation was only being disclosed now; Chandler told the board that the initial plan was for the schools to use the Apple iPads previously purchased by the school board to access MiSiS. But questions over Deasy's role in that $1 billion contract put a hold to that approach. The school board was more than a bit incredulous about that explanation since they had not approve the purchase of iPads with the intent that they were to be used by teachers and school administrators as the primary means to access MiSiS.

Reluctantly, the school board approved $3.6 million in additional funding to purchase 3,340 new desktop computers for 784 LAUSD schools to allow them unfettered access to MiSiS.

While Deasy’s resignation will alleviate some of the immediate political pressure on LAUSD officials caused by MiSiS fiasco, the technical issues will undoubtedly last throughout this academic year and possibly well into the next. However, for many unlucky LAUSD students, the impacts may last for many years beyond that.

In Other News…

Baltimore County Maryland Teachers Tackling Student Tracking System Glitches

Tallahassee’s New Emergency Dispatch System Offline Again

Washington State’s Computer Network Suffers Major Outage

Software Glitch Hits Telecommunications Services of Trinidad and Tobago

New Mexico Utility Company Incorrectly Bills Customers

Software Issue Means Oklahoma Utility Company Overbills Customers

Computer Error Allows Pink Panther Gang Member Early Out of Austrian Jail

Dropbox Bug Wipes Out Some Users’ Files

Generic Medicines Might Have Been Approved on Software Error

Australia’s iiNet Apologizes to Hundreds of Thousands of Customers for Three-day Email Outage

Spreadsheet Error Costs Tibco Investors $100 Million

Duke Energy Falsely Reports 500,000 Customers as Delinquent Bill Payers Since 2010

IT Hiccups of the WeekThere were several IT Hiccups to choose from last week. Among them were: problems with the Los Angeles Unified School District’s fouled up new student information and management system that are so egregious that a judge ordered the district to address them immediately; and the UK Revenue and Customs department’s embarassing admission that its trouble-plagued modernized tax system has again made multiple errors in computing thousands of tax bills. However, the winner of this week’s title as the worst of the worst was an oofta by Duke Energy, the largest electric power company in the U.S. Duke officials apologized in a press release to over 500,000 of the utility’s 800,000-plus current and former customers (including 5,000 non-residential customers) across Indiana, Kentucky, and Ohio for erroneously reporting them as being delinquent in paying their utility bills since 2010.

Duke Energy admitted that the root cause of the problem was a coding error that occurred when customers opted to pay their monthly utility bills via the utility’s Budget Billing or Percentage of Income Payment Plan Plus (in Ohio only).  A company spokesperson told Bloomberg BusinessWeek that while customers were sent the correct invoices and their on-time payments were properly credited, the billing system indicated that the customers’ bills were paid late.

 As a result, that late payment information for residential customers was sent by formal agreement to the National Consumer Telecom & Utilities Exchange (NCTUE). The NCTUE is a consortium of over 70 member companies from the telecommunications, utilities and pay TV industries that serves as a credit data exchange service for its members. Holding over 325 million consumer records, NCTUE provides information to its members regarding the credit risk of their current and potential customers. For non-residential customers, the “late payment” snafu had worse consequences: the delinquency reports were sent to the business credit rating agencies Dun & Bradstreet and Equifax Commercial Services.

Duke Energy’s press release said that the company “deeply regretted” the error that has effectively trashed the credit scores of hundreds of thousands of its residential and business customers for years. The utility says the erroneous information has now been “blocked” for use by the NCTUE, Dun & Bradstreet and Equifax, and it has dropped its membership in all three.

The press release mentioned that the company is still investigating whether additional customers who had “unique” billing circumstances were affected by the coding error.

But what the written statement failed to mention is that the utility found the error only after a former customer discovered that she was having trouble setting up service at another NCTUE utility member because of a supposedly poor payment history at Duke Energy. After contacting Duke Energy and asking why she was being shown as a delinquent bill payer when she was not, the utility realized that the woman’s erroneous credit information was only the tip of a very large IT oofta iceberg.

While Duke Energy claims that “we take responsibility” for the error, it is being rather quiet about explaining what exactly “taking responsibility” means for the hundreds of thousands of customers who may have been unjustly financially affected by the erroneous information sent to the three credit agencies over the past four years. It wouldn’t surprise me to see a class action lawsuit filed against Duke Energy in the near future to help the company gain greater clarity on what its responsibility is.

In Other News…

Judge Orders California to Help LAUSD Fix School Computer Fiasco

UK’s Tax Agency Admits it Can’t Compute Taxes Properly

Tahoe Ski Resort Withdraws Erroneous $1 Season Pass

UK NHS Hospital Patients Offered Harry Potter Names

Florida Utility Insists New Billing System is Right: Empty House Used 614,000 Gallons of Water in 18 Days

Audit Explains How Kansas Botched Its $40 Million DMV Modernization Effort

Indiana BMV Finally Sending Out Overbilling Refund Checks

Nielson Says Software Error Skews Television Viewer Stats for Months

Japan Trader's $617 Billion “Fat Finger” Near-Miss Rattles Tokyo Market

IT Hiccups of the Week

This week’s IT Hiccup of the Week concerns yet another so-called “fat finger” trade embroiling the Tokyo Stock Exchange (TSE). This time it involved an unidentified trader who last week mistakenly placed orders for shares in 42 major Japanese corporations.

According to a story at Bloomberg News, the trader placed over-the-counter (OTC) orders adding up to a total value of 67.78 trillion yen ($617 billion) in companies such as Canon, Honda, Toyota and Sony, among others. The share order for Toyota alone was for 1.96 billion shares—or 57 percent of the car company—amounting to about $116 billion.

Bloomberg reported that its analysis “shows that someone traded 306,700 Toyota shares at 6,399 yen apiece at 9.25 a.m. ... The total value of the transaction was 1.96 billion yen. The false report was for an order of 1.96 billion shares. [The Japan Securities Dealers Association] said the broker accidentally put the value of the transaction in the field intended for the number of shares.”

The $617 billion dollar order, which Bloomberg said was “greater than the size of Sweden’s economy and 16 times the Japanese over-the-counter market’s traded value for the entire month of August,” was quickly canceled before the orders could be completed. Given the out-sized orders and that OTC orders can be canceled anytime during market hours, it is unlikely that the blunder would have gone unfixed for very long, but the fact that it happened resurrected bad memories for the Tokyo Stock Exchange.

Back in 2005, Mizuho Financial Group made a fat finger trade on the TSE that could not be canceled out. A Financial Times of London story states that, “Mizuho Securities mistakenly tried to sell 610,000 shares in recruitment company J-Com at ¥1 apiece instead of one share at ¥610,000. The brokerage house said it had tried, but failed, to cancel the J-Com order four times.” The mistaken $345 million trade cost the president of the TSE along with two other exchange directors their jobs.

Then in 2009, a Japanese trader for UBS ordered $31 billion worth of bonds instead of buying the $310,000 he had intended, the London Telegraph reported.  Luckily, the order was sent after hours, so it was quickly discovered and corrected.

A little disconcerting, however, was a related Bloomberg News story from last week that quoted Larry Tabb, founder of research firm Tabb Group LLC. According to Tabb, despite all the recent efforts by US regulators and the exchanges themselves to keep rogue trades from occurring (e.g., the Knight Capital implosion), fat finger trades still “could absolutely happen here.”

“While we do have circuit breakers and pre-trade checks for items executed on exchange,” Tabb told Bloomberg, “I do not believe that there are any such checks on block trades negotiated bi-laterally and are just displayed to the market.”

Don’t insights like that from a Wall Street insider just give you a warm and fuzzy feeling about the reliability of financial markets?

In Other News…

Computer Glitch Affects 60,000 Would-be Organ Donors in Canada

Korean Air New Reservations System Irritates Customers

Ford Recalls 850,000 Vehicles to Fix Electronics

Mitsubishi i-MiEV Recalled to Fix Software Brake Issue

Doctors’ “Open Payments” Website Still Needs Many More Government Fixes

Apple iOS 8 Hit by Bluetooth Problems

Electronic Health Record System Blamed for Missing Ebola at Dallas Hospital

JP Morgan Chase: Contacts for 76 Million Households and 7 Million Small Businesses Compromised

Banking giant JP Morgan Chase filed an official notice yesterday to the U.S. Securities and Exchange Commission (SEC) updating the material information concerning the cyberattack the bank uncovered during the summer. According to the bank’s Form 8-K, for customers using its Chase.com and JPMorganOnline websites as well as the Chase and J.P. Morgan mobile applications:

Read More

FBI’s Sentinel System Still Not In Total Shape to Surveil

IT Hiccups of the Week

Other than the rather entertaining kerfuffle involving Apple’s new iPhone OS and its initial (non)corrective update (along with the suspicious “bendy phone” accusations), the IT Hiccups front was rather quiet this past week. Luckily, an “old friend” came by to rescue us from writing a post on some rather mundane IT snarl, snag or snafu.

Just in the nick of time, the U.S. Department of Justice's Inspector General released his latest in an ongoing series of reports [pdf] about Sentinel, the FBI’s electronic information and case management system. In this report, the IG focused on how Sentinel users felt about working with the system. Sadly yet unsurprisingly, the IG found that Sentinel is still suffering from some serious operational deficiencies two years after it went live.

Read More

Home Depot: Everything is Secure Now, Except Maybe in Canada

This past Thursday, after weeks of speculation, Home Depot, which calls itself the world’s largest home improvement retailer, finally announced [pdf] the total damage from a breach of its payment system: At its 1,157 stores in the U.S. and Canada, 56 million unique credit and debit cards were compromised. This is said to be among the three largest IT security breaches of a retail store, and ranks with some of the largest security breaches of all time.

According to Home Depot’s press release, the company confirmed that the criminal cyber intrusion began in April and ran into September, and “used unique, custom-built malware to evade detection. The malware had not been seen previously in other attacks, according to Home Depot’s security partners.”

The company says that it has now removed all the malware that infected its payment terminals, and that it has “has rolled out enhanced encryption of payment data to all U.S. stores.” The enhanced encryption approach, Home Depot states, “takes raw payment card information and scrambles it to make it unreadable and virtually useless to hackers.” It is a bit curious that the company says “virtually useless” and not “completely useless,” though.

Canadian stores, on the other hand, will have to wait a bit longer. While Home Depot’s Canadian stores have point-of-sale EMV chip and PIN card terminals, “the rollout of enhanced encryption to Canadian stores will be completed by early 2015,” the company says. Canadian Home Depot stores were at first thought to be less vulnerable because of the chip-and-pin terminals being in place, but that apparently hasn't been the case. For some reason, the company is refusing to disclose the number of Canadian payment cards compromised, the Globe and Mail says. The Globe and Mail estimates the total number of cards compromised to be around 4 million.

Home Depot goes on to say in its press release that it has no evidence “that debit PIN numbers were compromised or that the breach has impacted stores in Mexico or customers who shopped online at HomeDepot.com or HomeDepot.ca.”

As usual in these situations, Home Depot “is offering free identity protection services [for one year], including credit monitoring, to any customer who used a payment card at a Home Depot store in 2014, from April on. The company also apologized to its customers “for the inconvenience and anxiety this has caused.”

Home Depot’s data breach was first made public on 2 September by Brian Krebs, the former longtime Washington Post reporter with amazing IT security contacts, who now publishes a must-read security website called Krebs on Security. Several banking sources told Krebs that “a massive new batch of stolen credit and debit cards that went on sale [that] morning in the cybercrime underground,” with Home Depot looking like the source. Krebs went on to write that:

There are signs that the perpetrators of this apparent breach may be the same group of Russian and Ukrainian hackers responsible for the data breaches at Target, Sally Beauty and P.F. Chang’s, among others. The banks contacted by this reporter all purchased their customers’ cards from the same underground store — rescator[dot]cc — which on Sept. 2 moved two massive new batches of stolen cards onto the market.

In fact, it wasn’t until 8 September that Home Depot confirmed that it had in fact suffered a breach. Krebs, who has since written about the breach several times, recently wrote that the breach may not be as severe as indicated (nor as severe as it could have been). Sources have indicated that the malware used — which looks like a variant of what smacked Target late last year — was “installed mainly on payment systems in the self-checkout lanes at retail stores.” The reasoning is that if the malware had penetrated Home Depot’s payment system to the extent that Target’s systems were breached, many more than 56 million payment cards would have been compromised.

Sellers of compromised Home Depot card data are targeting specific states and ZIP codes in the hopes that buyers of the stolen cards will raise fewer red flags in the credit card and banking fraud algorithms. For instance, some 52,000 for Maine Home Depot stores, 282,000 for stores in Wisconsin, and 12,000 for those stores in Minnesota have been offered for sale. Card prices seem to be ranging mostly from $9 to $52 apiece, although for $8.16 million, one could purchase all of the stolen payment card numbers from Wisconsin, the Milwaukee-Wisconsin Journal Sentinel reported. The Journal Sentinel noted that its investigation found that:

Prices start at $2.26 for a Visa debit card with an expiration date of September 2014. The most valuable cards are MasterCard platinum debit cards and business credit cards. The most expensive card compromised in Wisconsin, a MasterCard valid through December 2015, was advertised at $127.50.

Interestingly, while Home Depot’s 56 million payment card breach is larger than Target’s 40 million payment card breach, the severity of the blowback so far is much more muted on the part of customers and investors. Part of the reason seems to be that the discovery of the breach happened at the end of summer, a slow shopping time for Home Depot, while Target’s was announced during the prime holiday buying period, which spooked its customers.

Further, investors have figured that Target’s breach cost the company some $150 million, excluding the $90 million in insurance reimbursements—a sum the company could ill afford given its ongoing retail difficulties. A similar sum may dent Home Depot’s bottom line, but the company is better placed financially to absorb the damage. The company stated in its press release that it has spent at least $62 million in dealing with the breach so far, with some $27 million of it covered by insurance. Home Depot says it doesn’t know how much more it will need to spend, but I suspect it could be an additional couple of hundred million dollars before all is said and done.

A third reason for the muted response may be that customers are now becoming inured in the wake of so many point-of-sales data breaches. For example, last May, the Ponemon Institute was cited in a CBS News report as stating that some 47 percent of adult Americans have had their personal information compromised in the past year. Given the Home Depot breach, as well as many others since, the number is probably even higher now. How many people had their personal information compromised multiple times is unknown, but I suspect it isn’t an insignificant number.

Home Depot’s financial and reputational pain might increase significantly, however, if the joint Connecticut, Illinois, and Californian state attorneys general investigation into the breach decides there is sufficient cause to sue Home Depot. As expected, at least one class action lawsuit each has been filed in both the United States and Canada, and more can be expected. Banks may also decide to sue Home Depot to cover the cost of any credit or debit cards they have to replace and for other financial damages, like some did against Target and earlier against TJX.

As reported by both The New York Times and Bloomberg’s BusinessWeek, Home Depot was repeatedly warned by its own IT security personnel about its poor and outdated IT security since 2008. Corporate management reportedly decided not to increase immediately the company’s security capabilities using readily available systems even in the aftermath of the Target breach and a couple of Home Depot stores being hacked last year, incidents that were not publicly disclosed until now. While the company did eventually decide to upgrade its payment security systems, the implementation effort didn’t get started until April, the same month as the breach. In addition, the papers report, Home Depot seemed to have weak security monitoring of its payment system, even though company management knew it was highly vulnerable to attack.

That Home Depot’s payment system was left vulnerable is interesting, because the company spent hundreds of millions of dollars improving its IT infrastructure over the past decade. Perhaps with revenues of $79 billion in 2013 the company felt it could easily afford the costs of an attack, and therefore, there was no urgent rush to increase its security posture. Brian Krebs notes this apparent lack of urgency as well. He says that even though the company was alerted to something being massively amiss by banks,  “thieves were stealing card data from Home Depot’s cash registers up until Sept. 7, 2014, a full five days after news of the breach first broke.

That alone speaks of an arrogance that belies Home Depot's public statements about how it takes the privacy and security of its customers’ personal information “very seriously.” Local Home Depot store personnel I have spoken with seem very ill-informed concerning the breach and what customers should do about it, which also seems to me a sign of a less than Home Depot’s advertised customer-caring attitude.

Home Depot’s seemingly cavalier IT security attitude isn’t unique, of course. Target didn’t bother to investigate alerts from its advanced warning system showing that it was being hacked until it was JTL — just too late. Just last week, eBay was being slammed again for its “lackadaisical attitude” toward IT security after multiple instances of malicious cross-site scripting that have been unabated since February were found on its UK website. Only after the BBC started asking eBay questions about the scripting issue did it decide that perhaps it should take them seriously. You may remember, it was only last March when eBay, which also proclaims to take customer security “very seriously,” asked all of its users to change their passwords after a cyberattack compromised its database of 233 million usernames, contact information, dates of birth, and encrypted passwords.

To tell you the truth, every time I read or hear a company or government agency claim in a press release that, “We take your security seriously,” in the wake of some security breach, I shake my head in disbelief.  Why not just state honestly, “We promised to take your security seriously and we obviously failed to take it seriously enough. We’re sorry and we will be better prepared from now on.” Alas, that level of candor is probably much too much to ask.

Indiana’s Bureau of Motor Vehicles Overcharged 180,000 Customers for 10 years

IT Hiccups of the Week

Put aside, for a moment, the record theft of credit card accounts from Home Depot. I'll tell you all about that in a later post. Instead let me pick another interesting IT Hiccup from last week's hodgepodge of IT problems, snarls, and screw-ups: The Indiana’s Bureau of Motor Vehicles (BMV) plans to refund some US $29 million plus interest to 180,000 customers for charging them an incorrectly calculated excise tax when they registered their vehicles. The BMV claimed the problem began during the initial changeover in 2004 to its then new $32 million System Tracking and Record Support (STARS) computer system.

Read More

GM: The Number of Models That Could Shut Off While You’re Driving Has Tripled

Guess what I got in the mail yesterday! Nope. But that was a good guess. The letter in my mailbox was a safety recall notice from General Motors, the manufacturer of the car I drive. Why should you care, you ask? I'm one of half a million people who have received the notice about the problem, but we represent less than one percent of the number of drivers affected.

Read More

Looking for the Key to Security in the Internet of Things

As the number of Internet connected-devices in any home skyrockets from a few, to a few dozen, to perhaps even a few hundred—including interconnecting thermostats, appliances, health and fitness monitors and personal accessories like smart watches—security concerns for this emerging Internet of Things (IoT) will skyrocket too. Cisco projects that there will be 50 billion connected devices by 2020; each such node should ideally be protected against malware, spyware, worms, and trojans, as well as overzealous government and commercial interests who themselves might produce their own privacy-compromising intrusions.

It’s a tall order, says Allen Storey, product director at the UK security firm Intercede. But the biggest challenges today are not so much technical problems as they are matters of awareness and education. Consumers need to know, says Storey, that IoT security is a real concern as the first wave of gadgets roll out into the marketplace. And unlike devices with faster processors and bigger memories, security is a product feature that the marketplace may not by itself reward.

Writing in the journal Network Security in July, Storey said that “Without the threat of end-user backlash, there is no strong business case for manufacturers to add a ubiquitous security element into the development process.” Moreover, he said, commercial pressures could in fact only reduce IoT security as many small players rush to be first to market. It's also likely that all the players could pursue siloed security standards that would leave substantial security holes as those devices interconnect with still other Internet-enabled devices (e.g. routers, smartphones, smart watches).

In the absence of any clear industry-wide IoT security standards, Intercede CTO Chris Edwards says consumers should shop for devices that rely on tried and tested security schemes, especially public key cryptography.

“When you’re looking at authenticating devices, the only real standards at the moment that offer any real interoperability tend to be Public Key Infrastructure (PKI),” he says. “The idea here is that you have a secure hardware element in that device that is able to generate and store and use private cryptographic keys that cannot be exported. So you can’t clone that device.”

So PKI chips, like those found in most smart cards, can help secure IoT communications. One other security standard that could be important in the IoT’s early years, Edwards says, is that of the FIDO (Fast IDentity Online) Alliance.

FIDO, a commercial consortium whose members include Microsoft, Google, PayPal, Lenovo, BlackBerry, and MasterCard, offers a lower-overhead variation of PKI that authenticates users and devices in part via biometrics (e.g. fingerprint-sensing chips) and PINs. This in turn makes FIDO more readily scalable to home networks with many devices on them, some of which may not have the battery or processor power to do classic private-public key cryptography for every communication.

“I don’t want the whole world to trust my watch,” Edwards says. “I just want to make sure the front door trusts my watch.”

Apple is conspicuously absent from FIDO's membership roll, which means that the Apple Watch's security will involve a yet to be disclosed set of proprietary security standards. Those protocols will thus probably form an important second web of security standards for the most secure IoT devices.

As an example of an IoT network that uses both PKI and FIDO, Edwards imagines a smartphone that communicates with a smart refrigerator in its owner’s home. The phone and refrigerator have already been introduced to each other and thus don’t need the highest PKI security levels. In that situation, FIDO would suffice for communications between the two devices such as the smartphone telling the fridge to go into low-power mode when the family goes on vacation, or the fridge reporting to the phone that it's time to pick up some milk from the grocery store.)

On the other hand, if the fridge communicates directly to the store to order more milk, the grocery store isn’t going to want to deal with FIDO certifications for its hundreds of customers. It’s more likely to insist on PKI security and authentication when a nearby fridge orders a gallon of milk or a case of beer.

In all, Storey says, the landscape of IoT security standards demands a company that can manage all such secure transactions behind the scenes for the cornucopia of third-party IoT device makers—perhaps like antivirus software today is managed and regularly updated by a small set of private, specialized companies.

“Given the absence of one standards agency producing cover-all protocols, an opportunity has emerged for security vendors and service providers to offer their own umbrella solutions that enable the individual to take control,” Storey wrote. “This is an exciting new dawn, but the industry must first come together to ensure it is a secure one for everyone concerned.”

Detroit's IT Systems “Beyond Fundamentally Broken”

IT Hiccups of the Week

Last week’s IT Hiccups parade was a bit slower than normal, but there were a couple of IT snafus that caught my eye. For instance, there was the embarrassed admission by Los Angeles Unified School District (LAUSD) chief strategic officer Matt Hill that the new-but-still-problem-plagued MiSiS student tracking system I wrote about a few weeks ago should have had “a lot more testing” before it was ever rolled out. There also was the poorly thought out pasta promotion by Olive Garden restaurants that ended up crashing its website. However, what sparked my curiosity most was the disclosure by Beth Niblock, Detroit’s Chief Information Officer, that the city’s IT systems were broken.

How broken are they? According to Niblock:

“Fundamentally broken, or beyond fundamentally broken. In some cases, fundamentally broken would be good.”

Niblock’s comment was part of her testimony during Detroit’s bankruptcy hearings. Last July, Detroit filed bankruptcy and since then has been in bankruptcy court trying to work out debt settlements with its creditors, some of whom are unhappy over the terms the city offered. Niblock was a witness at a court hearing looking into whether the city’s bankruptcy plan was feasible and fair to its many creditors, and whether the plan would put the city on more sound financial and operational footing.

Critical to Detroit returning to financial and operational soundness is the state of the city’s IT systems. However, since the 1990s, the city’s IT systems have generally been a shambles, and that is putting it charitably. Currently, according to Niblock (who took on the CIO job in February after turning it down twice and maybe wishing she did a third time), the city’s IT systems are “atrocious”, “unreliable” and “deficient,” Reuters reported.

Reuters went on to report Niblock's testimony that the city’s Unisys mainframe systems are “so old that they are no longer updated by their developers and have security vulnerabilities.” She added that the desktop computers, which mostly use Windows XP or something older, “take 10 minutes” to boot. It probably doesn’t matter anyway, since the computers run so many different versions of software that city workers can’t share documents or communicate, Niblock says. That also may not be so bad, given that city computers have apparently been infected several times by malware.

Detroit’s financial IT systems are so bad that the city really hasn’t known what it is owed or in turn, what it owes, for years. A Bloomberg News story last year, for example, told the story of a $1 million check from a local school district that wasn’t deposited by Detroit for over a month. During that time, the check sat in a city hall desk drawer. That isn’t surprising, the Bloomberg story noted, as the city has a hard time keeping track of funds electronically wired to it. The financial systems are so poor that city income-tax receipts need to be processed by hand; in fact, some 70 percent of all of the city’s financial accounting entries are still done manually. The costs of doing things manually are staggering: it costs Detroit $62 to process each city paycheck, as opposed to the $18 or so it should cost.  Bloomberg stated that a 2012 Internal Revenue Service audit of the city’s tax collection system termed it as being “catastrophic.”

While the financial IT system woes are severe, the fire and police departments' IT systems may be in even worse shape. According to the Detroit News Free Press, there is no citywide computer aided dispatch system to communicate emergency alerts to fire stations. Instead, fire stations receive the alerts by fax machine. To make sure the alarm is actually heard, fire fighters have rigged Radio Shack buzzers and doorbells, among other homemade Rube Goldberg devices that are triggered by the paper coming out of the fax machine. Detroit's Deputy Fire Commissioner told the Detroit News Free Press that, “It sounds unbelievable, but it’s truly what the guys have been doing and dealing with for a long, long time.”

You really need to check out the video accompanying the Detroit News Free Press story which shows fire fighters using a soda can filled with coins and screws perched on the edge of the fax machine so that it will be knocked off by the paper coming out of the machine when an emergency alert is received at the fire station. Makes one wonder what happens if the fax runs out of paper.

The Detroit police department's IT infrastructure, what there is of it, isn’t in much better shape. Roughly 300 of its 1150 computers are less than three years old. Apparently even those “modern” computers have not received software updates, and in many cases, the software the police department relies on is no longer supported by vendors. The police lack an automated case management system, which means officers spend untold hours manually filling out, filing, and later trying to find paperwork. Many Detroit police cars also lack basic Mobile Data Computers (MDC), which means officers have to rely on dispatchers to perform even basic functions they should be able to do themselves. An internal review (pdf) of the state of Detroit’s police department was published in January, and it makes for very sad, if not scary, reading.

If you are interested in how Detroit’s IT systems became “beyond fundamentally broken,” there is a great case study that appeared in a 2002 issue of Baseline magazine. It details Detroit’s failed attempt, beginning in 1997, to upgrade and integrate its various payroll, human resources, and financial IT systems into a single be-all Detroit Resource Management System (DRMS) that went by the name “Dreams.” The tale told is a familiar one to Risk Factor readers: attempting to replace 22 computer systems used across 43 city departments with one city-wide system resulted in a massive cost overrun and little to show for it five years on. Crain’s Detroit Business also took a look back at the DRMS implementation nightmare in a July article.

Detroit hopes, the Detroit News reports, that the bankruptcy judge will approve its proposed $101 million IT “get well” plan, which includes $84.8 million for IT upgrades and $16.3 million for additional IT staff. (In February, according to a story in the Detroit News Free Press, the city wanted to invest $150 million, but that amount apparently needed to be scaled back because of budgetary constraints.) Spending $101 million, Niblock admitted, will not buy world-class IT systems, but ones that are, “on the grading scale… a ‘B’ or a B-minus” at best. And Niblock concedes that getting to a “B” grade will require a lot of things going perfectly right, which is not likely to happen.

On one final note, I’d be remiss not to mention that last week was also the 25th anniversary of the infamous Parisian IT Hiccup. For those who don’t remember, in September 1989, some 41,000 Parisians who were guilty of simple traffic offenses were mailed legal notices that accused them of committing everything from manslaughter to hiring prostitutes or both.  As a story in the Deseret News from the time noted:

“A man who had made an illegal U-turn on the Champs-Élysées was ordered to pay a $230 fine for using family ties to procure prostitutes and ‘manslaughter by a ship captain and leaving the scene of a crime.’”

Local French officials blamed the problem on “human error by computer operators.”

Plus ça change, plus c'est la même.

In Other News ….

Coding Error Exposes Minnesota Students' Personal Information

Computer Glitch Sounds Air Raid Sirens in Polish Town

Computer Problems Change Florida County Vote Totals

Billing Error Affects Patients at Tennessee Regional Hospital

Dallas Police Department Computer Problems Causing Public Safety Concerns

New York Thruway Near Albany Overbills 35,000 EZ‐Pass Customers

Olive Garden Shoots Self in Foot With Website Promotion

Apple Store Crashes Under iPhone6 Demand

Scandinavian Airlines says Website Now Fixed After Two Days of Trouble

Housing New Zealand Tenants Shocked by $10,000 a Week Rent Increases

GM's China JV Recalling 38,328 Cadillacs to Fix Brake Software

LAUSD MiSiS System Still Full of Glitches

Advertisement

Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Contributor
Willie D. Jones
 
Load More