Risk Factor iconRisk Factor

Indiana’s Bureau of Motor Vehicles Overcharged 180,000 Customers for 10 years

IT Hiccups of the Week

Put aside, for a moment, the record theft of credit card accounts from Home Depot. I'll tell you all about that in a later post. Instead let me pick another interesting IT Hiccup from last week's hodgepodge of IT problems, snarls, and screw-ups: The Indiana’s Bureau of Motor Vehicles (BMV) plans to refund some US $29 million plus interest to 180,000 customers for charging them an incorrectly calculated excise tax when they registered their vehicles. The BMV claimed the problem began during the initial changeover in 2004 to its then new $32 million System Tracking and Record Support (STARS) computer system.

Read More

GM: The Number of Models That Could Shut Off While You’re Driving Has Tripled

Guess what I got in the mail yesterday! Nope. But that was a good guess. The letter in my mailbox was a safety recall notice from General Motors, the manufacturer of the car I drive. Why should you care, you ask? I'm one of half a million people who have received the notice about the problem, but we represent less than one percent of the number of drivers affected.

Read More

Looking for the Key to Security in the Internet of Things

As the number of Internet connected-devices in any home skyrockets from a few, to a few dozen, to perhaps even a few hundred—including interconnecting thermostats, appliances, health and fitness monitors and personal accessories like smart watches—security concerns for this emerging Internet of Things (IoT) will skyrocket too. Cisco projects that there will be 50 billion connected devices by 2020; each such node should ideally be protected against malware, spyware, worms, and trojans, as well as overzealous government and commercial interests who themselves might produce their own privacy-compromising intrusions.

It’s a tall order, says Allen Storey, product director at the UK security firm Intercede. But the biggest challenges today are not so much technical problems as they are matters of awareness and education. Consumers need to know, says Storey, that IoT security is a real concern as the first wave of gadgets roll out into the marketplace. And unlike devices with faster processors and bigger memories, security is a product feature that the marketplace may not by itself reward.

Writing in the journal Network Security in July, Storey said that “Without the threat of end-user backlash, there is no strong business case for manufacturers to add a ubiquitous security element into the development process.” Moreover, he said, commercial pressures could in fact only reduce IoT security as many small players rush to be first to market. It's also likely that all the players could pursue siloed security standards that would leave substantial security holes as those devices interconnect with still other Internet-enabled devices (e.g. routers, smartphones, smart watches).

In the absence of any clear industry-wide IoT security standards, Intercede CTO Chris Edwards says consumers should shop for devices that rely on tried and tested security schemes, especially public key cryptography.

“When you’re looking at authenticating devices, the only real standards at the moment that offer any real interoperability tend to be Public Key Infrastructure (PKI),” he says. “The idea here is that you have a secure hardware element in that device that is able to generate and store and use private cryptographic keys that cannot be exported. So you can’t clone that device.”

So PKI chips, like those found in most smart cards, can help secure IoT communications. One other security standard that could be important in the IoT’s early years, Edwards says, is that of the FIDO (Fast IDentity Online) Alliance.

FIDO, a commercial consortium whose members include Microsoft, Google, PayPal, Lenovo, BlackBerry, and MasterCard, offers a lower-overhead variation of PKI that authenticates users and devices in part via biometrics (e.g. fingerprint-sensing chips) and PINs. This in turn makes FIDO more readily scalable to home networks with many devices on them, some of which may not have the battery or processor power to do classic private-public key cryptography for every communication.

“I don’t want the whole world to trust my watch,” Edwards says. “I just want to make sure the front door trusts my watch.”

Apple is conspicuously absent from FIDO's membership roll, which means that the Apple Watch's security will involve a yet to be disclosed set of proprietary security standards. Those protocols will thus probably form an important second web of security standards for the most secure IoT devices.

As an example of an IoT network that uses both PKI and FIDO, Edwards imagines a smartphone that communicates with a smart refrigerator in its owner’s home. The phone and refrigerator have already been introduced to each other and thus don’t need the highest PKI security levels. In that situation, FIDO would suffice for communications between the two devices such as the smartphone telling the fridge to go into low-power mode when the family goes on vacation, or the fridge reporting to the phone that it's time to pick up some milk from the grocery store.)

On the other hand, if the fridge communicates directly to the store to order more milk, the grocery store isn’t going to want to deal with FIDO certifications for its hundreds of customers. It’s more likely to insist on PKI security and authentication when a nearby fridge orders a gallon of milk or a case of beer.

In all, Storey says, the landscape of IoT security standards demands a company that can manage all such secure transactions behind the scenes for the cornucopia of third-party IoT device makers—perhaps like antivirus software today is managed and regularly updated by a small set of private, specialized companies.

“Given the absence of one standards agency producing cover-all protocols, an opportunity has emerged for security vendors and service providers to offer their own umbrella solutions that enable the individual to take control,” Storey wrote. “This is an exciting new dawn, but the industry must first come together to ensure it is a secure one for everyone concerned.”

Detroit's IT Systems “Beyond Fundamentally Broken”

IT Hiccups of the Week

Last week’s IT Hiccups parade was a bit slower than normal, but there were a couple of IT snafus that caught my eye. For instance, there was the embarrassed admission by Los Angeles Unified School District (LAUSD) chief strategic officer Matt Hill that the new-but-still-problem-plagued MiSiS student tracking system I wrote about a few weeks ago should have had “a lot more testing” before it was ever rolled out. There also was the poorly thought out pasta promotion by Olive Garden restaurants that ended up crashing its website. However, what sparked my curiosity most was the disclosure by Beth Niblock, Detroit’s Chief Information Officer, that the city’s IT systems were broken.

How broken are they? According to Niblock:

“Fundamentally broken, or beyond fundamentally broken. In some cases, fundamentally broken would be good.”

Niblock’s comment was part of her testimony during Detroit’s bankruptcy hearings. Last July, Detroit filed bankruptcy and since then has been in bankruptcy court trying to work out debt settlements with its creditors, some of whom are unhappy over the terms the city offered. Niblock was a witness at a court hearing looking into whether the city’s bankruptcy plan was feasible and fair to its many creditors, and whether the plan would put the city on more sound financial and operational footing.

Critical to Detroit returning to financial and operational soundness is the state of the city’s IT systems. However, since the 1990s, the city’s IT systems have generally been a shambles, and that is putting it charitably. Currently, according to Niblock (who took on the CIO job in February after turning it down twice and maybe wishing she did a third time), the city’s IT systems are “atrocious”, “unreliable” and “deficient,” Reuters reported.

Reuters went on to report Niblock's testimony that the city’s Unisys mainframe systems are “so old that they are no longer updated by their developers and have security vulnerabilities.” She added that the desktop computers, which mostly use Windows XP or something older, “take 10 minutes” to boot. It probably doesn’t matter anyway, since the computers run so many different versions of software that city workers can’t share documents or communicate, Niblock says. That also may not be so bad, given that city computers have apparently been infected several times by malware.

Detroit’s financial IT systems are so bad that the city really hasn’t known what it is owed or in turn, what it owes, for years. A Bloomberg News story last year, for example, told the story of a $1 million check from a local school district that wasn’t deposited by Detroit for over a month. During that time, the check sat in a city hall desk drawer. That isn’t surprising, the Bloomberg story noted, as the city has a hard time keeping track of funds electronically wired to it. The financial systems are so poor that city income-tax receipts need to be processed by hand; in fact, some 70 percent of all of the city’s financial accounting entries are still done manually. The costs of doing things manually are staggering: it costs Detroit $62 to process each city paycheck, as opposed to the $18 or so it should cost.  Bloomberg stated that a 2012 Internal Revenue Service audit of the city’s tax collection system termed it as being “catastrophic.”

While the financial IT system woes are severe, the fire and police departments' IT systems may be in even worse shape. According to the Detroit News Free Press, there is no citywide computer aided dispatch system to communicate emergency alerts to fire stations. Instead, fire stations receive the alerts by fax machine. To make sure the alarm is actually heard, fire fighters have rigged Radio Shack buzzers and doorbells, among other homemade Rube Goldberg devices that are triggered by the paper coming out of the fax machine. Detroit's Deputy Fire Commissioner told the Detroit News Free Press that, “It sounds unbelievable, but it’s truly what the guys have been doing and dealing with for a long, long time.”

You really need to check out the video accompanying the Detroit News Free Press story which shows fire fighters using a soda can filled with coins and screws perched on the edge of the fax machine so that it will be knocked off by the paper coming out of the machine when an emergency alert is received at the fire station. Makes one wonder what happens if the fax runs out of paper.

The Detroit police department's IT infrastructure, what there is of it, isn’t in much better shape. Roughly 300 of its 1150 computers are less than three years old. Apparently even those “modern” computers have not received software updates, and in many cases, the software the police department relies on is no longer supported by vendors. The police lack an automated case management system, which means officers spend untold hours manually filling out, filing, and later trying to find paperwork. Many Detroit police cars also lack basic Mobile Data Computers (MDC), which means officers have to rely on dispatchers to perform even basic functions they should be able to do themselves. An internal review (pdf) of the state of Detroit’s police department was published in January, and it makes for very sad, if not scary, reading.

If you are interested in how Detroit’s IT systems became “beyond fundamentally broken,” there is a great case study that appeared in a 2002 issue of Baseline magazine. It details Detroit’s failed attempt, beginning in 1997, to upgrade and integrate its various payroll, human resources, and financial IT systems into a single be-all Detroit Resource Management System (DRMS) that went by the name “Dreams.” The tale told is a familiar one to Risk Factor readers: attempting to replace 22 computer systems used across 43 city departments with one city-wide system resulted in a massive cost overrun and little to show for it five years on. Crain’s Detroit Business also took a look back at the DRMS implementation nightmare in a July article.

Detroit hopes, the Detroit News reports, that the bankruptcy judge will approve its proposed $101 million IT “get well” plan, which includes $84.8 million for IT upgrades and $16.3 million for additional IT staff. (In February, according to a story in the Detroit News Free Press, the city wanted to invest $150 million, but that amount apparently needed to be scaled back because of budgetary constraints.) Spending $101 million, Niblock admitted, will not buy world-class IT systems, but ones that are, “on the grading scale… a ‘B’ or a B-minus” at best. And Niblock concedes that getting to a “B” grade will require a lot of things going perfectly right, which is not likely to happen.

On one final note, I’d be remiss not to mention that last week was also the 25th anniversary of the infamous Parisian IT Hiccup. For those who don’t remember, in September 1989, some 41,000 Parisians who were guilty of simple traffic offenses were mailed legal notices that accused them of committing everything from manslaughter to hiring prostitutes or both.  As a story in the Deseret News from the time noted:

“A man who had made an illegal U-turn on the Champs-Élysées was ordered to pay a $230 fine for using family ties to procure prostitutes and ‘manslaughter by a ship captain and leaving the scene of a crime.’”

Local French officials blamed the problem on “human error by computer operators.”

Plus ça change, plus c'est la même.

In Other News ….

Coding Error Exposes Minnesota Students' Personal Information

Computer Glitch Sounds Air Raid Sirens in Polish Town

Computer Problems Change Florida County Vote Totals

Billing Error Affects Patients at Tennessee Regional Hospital

Dallas Police Department Computer Problems Causing Public Safety Concerns

New York Thruway Near Albany Overbills 35,000 EZ‐Pass Customers

Olive Garden Shoots Self in Foot With Website Promotion

Apple Store Crashes Under iPhone6 Demand

Scandinavian Airlines says Website Now Fixed After Two Days of Trouble

Housing New Zealand Tenants Shocked by $10,000 a Week Rent Increases

GM's China JV Recalling 38,328 Cadillacs to Fix Brake Software

LAUSD MiSiS System Still Full of Glitches

FCC Fines Verizon $7.4 Million Over Six-Year Privacy Rights “IT Glitch”

IT Hiccups of the Week

The number of IT snafus, problems and burps moved back to a more normal rate last week. There were a surprising number of coincidental outages that hit Apple, eBay, Tumblr and Facebook, but other than these, the most interesting IT Hiccup of the Week was the news that the U.S. Federal Communications Commission (FCC) fined Verizon Communications a record $7.4 million for failing to notify two million customers of their opt-out rights concerning the use of their personal information for certain company marketing campaigns.

According to the Washington Post, Verizon is supposed to inform new customers via a notice in their first bill that they could opt-out of having their personal information used by the company to craft targeted marketing campaigns of products and services to them. However, since 2006, Verizon failed to include the opt-out notices.

A Verizon spokesperson blamed the oversight as being “largely due to an inadvertent IT glitch,” the Post reported. The Verizon spokesman, however, didn’t make it clear as to why the company didn’t notice the problem until September 2012, nor why it didn’t inform the FCC of the problem until 18 January 2013, some 121 days later than the agency requires. (Companies are required to inform the FCC of issues like this within five business days of their discovery.)  

The FCC’s press release annoucing the fine showed that the agency was clearly irritated by Verizon’s tardiness. Travis LeBlanc, the acting chief of the FCC Enforcement Bureau, said that, “In today’s increasingly connected world, it is critical that every phone company honor its duty to inform customers of their privacy choices and then to respect those choices. It is plainly unacceptable for any phone company to use its customers’ personal information for thousands of marketing campaigns without even giving them the choice to opt out.”   

Of course, a better solution would be for the FCC to force companies to allow customers only to opt-in to the use of their personal information, but that discussion is for another day.

On top of the $7.4 million fine, which the FCC took pains to point out is the “largest such payment in FCC history for settling an investigation related solely to the privacy of telephone customers’ personal information,” Verizon will have to include opt-out notices in every bill, as well as put a system in place to monitor and test its billing system to ensure that they actually go out.

Verizon tried to downplay the privacy rights violation, of course, even implying that its customers benefited from the glitch by being able to receive “marketing materials from Verizon for other Verizon services that might be of interest to them.”

Readers of the Risk Factor may remember another Verizon inadvertent IT glitch disclosed in 2010 in which  Verizon admitted that it over-billed customers by $52.8 million for “mystery fees” over three years.  During that time, Verizon customers who called the company to complain over the fees were told  basically to shut up and pay them. The FCC smacked Verizon with a then FCC record-setting $25 million fine for that little episode of customer non-service and IT ineptitude.

Last year, Verizon agreed to pay New York City $50 million for botching its involvement in the development of a new 911 emergency system. Alas, that wasn’t a record-setting settlement; SAIC owns that honor after paying the city $466 million to settle fraud charges related to its CityTime system development.

In Other News…

eBay Access Blocked by IT Problems

Facebook Experiences Third Outage in a Month

Tumblr Disrupted by Outage

Apple iTunes Outage Lasts 5 Hours

Twitter Sets Up Software Bug Bounty Program

Children Weight Entry Error Placed Australian Jet at Risk

Spanish ATC Computer Problem Scrambles Flights

Yorkshire Bank IT Problems Affects Payments

Computer Problem Hits Boston MBTA Corporate Pass Tickets

Unreliable Washington, DC Health Exchange Still Frustrates Users

South African Standard Bank Systems Go Offline

New Zealand Hospital Suffers Major Computer Crash

Computer Crash Forces Irish Hospital to Re-Check Hundreds of Blood Tests

Fiji Airways Says No to $0 Tickets Caused by Computer Glitch

Portugal’s New Court System Still Buggy

Hurricane Projected Landfall Only 2,500 Miles Off

Vulnerable "Smart" Devices Make an Internet of Insecure Things

According to recent research [PDF], 70 percent of Americans plan to own, in the next five years, at least one smart appliance like an internet-connected refrigerator or thermostat. That's a skyrocketing adoption rate considering the number of smart appliance owners in the United States today is just four percent. 

Read More

310,000 Healthcare.gov Enrollees Must Provide Proof Now or Lose Insurance

IT Hiccups of the Week

Last week, there were so many reported IT snags, snarls and snafus that I felt like the couple who finally won the 20-year jackpot on the Lion’s Share slot machine at the Las Vegas MGM Grand casino. Among IT Hiccups of note was the routine maintenance oofta at Time Warner Cable Wednesday morning that knocked out Internet and on demand service across the US for over 11 million of its customers and continued to cause other service issues for several days afterward; the “coding error” missed for six years by German Deutsche Bank that caused the misreporting to the UK government of 29.4 million equity swaps, with buys being reported as sales and vice versa; and the rather hilarious software bugs in the new Madden NFL 15 American football game, which has players flying around the field in interesting ways.

However, for this week, we just can’t ignore yet another Healthcare.gov snafu of major proportions. Last week, USAToday reported that the Centers for Medicare and Medicaid Services sent letters to 310,000 people who enrolled for health insurance through the federal website asking for proof of citizenship or immigration status by 5 September or they were going to lose their health insurance at the end of September.

Read More

LA School District Continues Suffering MiSiS Misery

IT Hiccups of the Week

With schools starting to open for the 2014-2015 academic year across the United States, one can confidently predict that there will be several news stories of snarls, snafus, and hitches with new academic IT supports systems as they go live for the first time. (You may may recall that happening in MarylandNew York, and Illinois a few years ago.)

While most of these “teething problems” are resolved during the first week or so of school, significant IT issues affecting the performance of the new integrated student educational tracking system recently rolled out in the Los Angeles Unified School District—the second largest in the country with 650,000 students—has already stretched beyond the first few weeks of the school term with no definitive end in sight. Furthermore, the many software bugs being encountered were known by LAUSD administrators, but they decided to roll out the system anyway.

Read More

The Routing Wall of Shame

IT Hiccups of the Week

While I have been en vacances the past few weeks, there have been several potential IT Hiccups of the Week stories of interest, including the 200-to-500 year old Indian women getting free sewing machines and Philippine’s fast food giant Jollibee Food having to temporarily close 72 of its restaurants in the Manila region because of problems the company experienced migrating to a new IT system—much to the disappointment of its Chickenjoy fans. However, the one hiccup that stands above the rest was the Internet difficulties reportedly experienced last week by the likes of eBay, Amazon, and LinkedIn, among many others.

Read More

Black Hat 2014: How to Hack the Cloud to Mine Crypto Currency

Using a combination of faked e-mail addresses and free introductory trial offers for cloud computing, a pair of security researchers have devised a shady crypto currency mining scheme that they say could theoretically net hundreds of dollars a day in free money using only guile and some clever scripting.

The duo, who are presenting their findings at this week’s Black Hat 2014 cyber security conference in Las Vegas, shut down their proof-of-concept scheme before it could yield any more than a token amount of Litecoins (an alternative to Bitcoin). The monetary value of both virtual currencies is based on enforced scarcity that comes from the difficulty of running processor-intensive algorithms.

Rob Ragan, senior security associate at the consulting firm Bishop Fox in Phoenix, Ariz., says the idea for the hack came to him and his collaborator Oscar Salazar when they were hired to test the security around an online sweepstakes.

“We figured if we could get 100,000 e-mails entered into the sweepstakes, we could have a really good chance of winning,” he says. “So we generated a script that would allow us to generate unique e-mail addresses and then automatically click the confirmation link.”

Once Ragan and Salazar had finished securing the sweepstakes against automated attacks, they were still left with all those e-mail addresses.

“We realized that … for about two-thirds of cloud service providers, their free trials only required a user to confirm an e-mail address,” he says. So the duo discovered they effectively had the keys to many thousands of separate free trial offers of cloud service providers’ networked storage and computing.

In other words, they had access to many introductory accounts at sites like Google’s Cloud Platform, Joyent, CloudBees, iKnode, CloudFoundry, CloudControl, ElasticBox and Microsoft Windows Azure.

Some of these sites, each offering their own enticement of free storage and free computing as a limited introductory offer, could be spoofed, the researchers discovered. Troves of unique e-mail addresses, using a non-discoverable automated process they developed, could be readily made on the fly and then used to get free storage and processor time.

A spoof e-mail address of course has two components, Ragan says, the local part (the stuff to the left of the “@“ sign) and the domain (to the right). To appear like a random stream of e-mail addresses signing up for any given service, Ragan says they scraped real local addresses from legit e-mail address dumps on sites like Pirate Bay. The domain side they set up using “FreeDNS” servers that attach e-mail addresses to existing domains, a service that can be exploited for domains that have poor security measures in place.

So, say there’s an address dump file on the Internet containing the legit e-mail addresses “CatLover290 at gmail” and “CarGuy909 at Yahoo.” Ragan and Salazar’s algorithm would attach “CatLover290” and “CarGuy909” to one of thousands of spoof URLs they’d set up through the FreeDNS sites. The original e-mail accounts would then be unaffected. But the resulting portmanteau e-mail addresses would appear to be coming from a random stream of humans on the Internet.

Thus, Ragan says, not even a human observer watching the e-mails registering for free cloud computing accounts—none appearing to be produced by a simple algorithm or automated process—would detect anything overtly suspicious. And to further throw off the scent of suspicious activity, they used Internet anonymizing software like TOR and virtual private networks to spoof where the trial account requests were coming from. (Ragan says that generating real-seeming names using name-randomizing algorithms would probably be good enough.)

“A lot of the e-mail confirmation and authentication features rely on the old concept that one person has one e-mail address—and that is simply not the case anymore,” Ragan says. “We’ve developed a platform that would allow anyone to have 30,000 e-mail addresses.”

So they signed up for hundreds of free cloud service trial accounts and, in the process, strung together a free, ersatz virtual supercomputer.

“We demonstrated that we could generate a high amount of crypto hashes for a high return on Litecoin mining, using these servers that didn’t belong to us,” Ragan says. “We didn’t have an electricity bill, and we were basically able to generate money for free out of thin air.”

Ragan says at their scheme’s peak, they had 1000 accounts that were each generating 25 cents per day: $250 of free Litecoin. He says they shut the system down before it generated any real monetary value or made any noticeable performance dent in the cloud service systems.

And Ragan stressed that the devious schemes he and Salazar developed are being disclosed in order to raise awareness of problems in security measures that real criminal elements around the world can and probably already are taking advantage of.

“Not planning for and anticipating automated attacks is one of the biggest downfalls a lot of online services are currently experiencing,” Ragan says.

One measure Ragan says he and Salazar wanted to see that would combat their scheme’s spoofing of cloud service providers was the introduction of random anti-automation controls. Captchas, credit card verification, and phone verification can all be spoofed, he says, if they’re at predictable places in the cloud service signup and setup process.

“Some services don’t want to add a Captcha, because it annoys users,” Ragan says. “But…there are compromises that can be [employed], like once an abnormal behavior is detected from a user account, they then prompt for a Captcha. Rather than prompting every user for a Captcha every time, they can find that balance. There’s always a balance to be made between security and usability.”

Ragan says that’s what he and Salazar want the takeaway from their talk to be: that a lot more consideration is given to how to better implement anti-automation controls and features.


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More