Risk Factor iconRisk Factor

We Need Better IT Project Failure Post-Mortems

In pulling together this special interactive report on a decade’s worth of IT development projects and operational failures, the most vexing aspect of our efforts was finding trustworthy data on the failures themselves.

We initially started with a much larger set than the 200 or so projects depicted in this report, but the project failure pool quickly shrank as we tried to get reliably documented, quantifiable information explaining what had occurred, when and why, who was affected, and most importantly, what the various economic and social impacts were.

This was true not only for commercial IT project failures—which one would expect, given that corporations are extremely reticent to advertise their misfortunes in detail if at all—but also for government IT project failures. Numerous times, we reviewed government audit reports and found that a single agency had inexplicably used different data for a project’s initial and subsequent costs, as well as for its schedule and functional objectives. This project information volatility made getting an accurate, complete, and consistent picture of what truly happened on a project problematic to say the least.

Our favorite poster child for a lack of transparency regarding a project’s failure is the ill-fated $1 billion U.S. Air Force Expeditionary Combat Support System (ECSS) program (although the botched rollout of HealthCare.gov is a strong second).  Even after multiple government audits, including a six-month, bi-partisan Senate Armed Services Committee investigation into the high-profile fiasco, the full extent of what this seven-year misadventure in project management was trying to accomplish could not be uncovered. Nor could the final cost to the taxpayer be ascertained.

With that in mind, we make our plea to project assessors and auditors asking that they apply a couple of lessons learned the hard way over the past decade of IT development project and operational failures: 

In future assessments or audit reports of IT development projects, would you please publish with each one a very simple chart or timeline? It should show, at a glance: an IT project’s start date (i.e., the time money is first spent on the project); a list of the top three to five functional objectives the project is trying to accomplish; and the predicted versus actual cost, date of completion, and delivered functionality at critical milestones where the project being reviewed, delivered, or canceled.

Further, if the project has been extended, re-scoped or reset, please make the details of such a change absolutely clear. Don’t forget to indicate how this deviation affects any of the aforementioned statistics. Finally, if the project has been canceled, account for the opportunity costs in the final cost accounting. For example, the failure of ECSS is currently costing the Air Force billions of dollars annually because of the continuing need to maintain legacy systems that should have been retired by now. You’d think that this type of project status information would be routinely available. But unfortunately, it is rarely published in its totality; when it is, it’s even less likely to be found all in one place.

Similarly, for records related to IT system operational failures, would you please include all of the consequences being felt—not only financially but to the users of the system, both internally and externally?  Too often an operational failure is dismissed as just a “teething problem,” when it feels more like a “root canal” to the people dependent upon the system working properly. 

A good illustration is Ontario’s C$242 million Social Assistance Management System (SAMS), which was released more than a year ago, and it is still not working properly. The provincial government remains upbeat and positive about the system’s operation while callously downplaying the impact of a malfunctioning system on the poor in the province.

More than 100 years ago, U.S. Supreme Court Judge Louis Brandeis argued that, “Publicity is justly commended as a remedy for social and industrial diseases. Sunlight is said to be the best of disinfectants; electric light the most efficient policeman.” Hopefully, the little bit of publicity we have tried to bring to this past decade of IT project failures will help to reduce their number in the future.

Wishful Thinking Plagues IT Project Plans

“An extraordinary failure in leadership,” a “masterclass in sloppy project management,” and a “test case in maladministration” were a few of the more colorful descriptions of UK government IT failures made by MP Edward Leigh, MP for Gainsborough, England when he was the Chairman of the Public Accounts Committee.

Leigh repeatedly pointed that government departments consistently were not only wholly unrealistic in their IT project costs, schedule and technical feasibility, but also didn’t take any responsibility for the consequences of these assumptions.

Read More

Sorry for the Inconvenience

After looking back at the project failures chronicled in the Risk Factor for our recent set of interactive features “Lessons From a Decade of IT Failures,” I became intrigued by the formulaic apologies that organizations put out in their press releases when something untoward happens involving their IT systems.  For instance, below are three typical IT mea culpas:

“We would like to apologize for the inconvenience this has caused.”

“We regret any inconvenience this may have caused patients.”

“We apologize for the inconvenience to those affected.”

The first apology came about as a result of the Nationwide, the UK’s largest building society, charging 704,426 customers’ debit card twice for the same transaction, which it blamed on a batch file that was doubly processed. The second apology was in response to the crash of Northern California’s Sutter’s Health $1 billion electronic health record system crashing for more than a day across seven of its major medical facilities because of a faulty system upgrade. The last apology was prompted by the shambolic mess that marked the rollout of California’s EDD unemployment system that affected at least 185,000 claimants, many of them for weeks.

Apparently, regardless of the impact or duration of an IT failure, it is viewed by the offending organization as being merely an inconvenience to those experiencing them. Ontario’s Social Services Minister Helena Jaczek went so far as to liken the months of havoc resulting from the botched rollout of a new provincial CN$240 million welfare and disability system as reminding her “of when I have my BlackBerry telling me that there are issues and I need an update. . . it is often very inconvenient.”

Hmms, not receiving a desperately needed disability or subsistence check equates to a Blackberry software update. Who knew?

Most apologetic public statements by corporate or government officials at least attempt to provide the pretense that executive management feels bad for the consequences of their organization’s IT oofta. However, there have been other times where perhaps not apologizing would have been a better strategy than the apology given. Below are a few of my favorite “best of the worst apologies” from the Risk Factor blog files which clearly indicate that the organization would have been a lot happier if it didn’t have to deal with those pesky customers or taxpayers.

We start off with two of the largest organizations in Ireland that worked overtime to antagonize their respective customers in the wake of billing system errors. The first is Irish Rail, which discovered that a software upgrade to its vending machines caused tickets to be issued to some 9,000 passengers without actually deducting the fare amounts from their debit cards for several days. On the Friday, a week after the discrepancy was discovered, Irish Rail decided to notify the affected customers by way of a press release on its website, which mentioned that it would be deducting the fare amounts due it beginning on the following Monday.

Irish Rail’s press release also stated, “We apologies [sic] for any inconvenience this fault causes customers,” which for many could include incurring hefty penalty charges for unauthorized overdrafts on their debit cards. When asked why it couldn’t wait a week so its customers could ensure that their accounts had the funds to cover the charges, Irish Rail responded it had every right to collect the money immediately and was going to do so. Unsurprisingly, the affected Irish Rail customers didn’t think much of the company’s apology.

Another “show me the money” demand disguised as an apology came from Eircom, the largest telecom provider in the Republic of Ireland. It, like Irish Rail, had a billing “system error” that did not directly debit some 30,000 customer bank accounts correctly, even though its customers’ bills indicated otherwise. Eircom deemed the incident “regrettable” and further stated that “it’s embarrassing and we're very sorry that it's happened.” However, Eircom was neither too embarrassed nor sorry enough to insist that although it planned to reimburse customers the failed direct debit fee charge of €18.45, customers would still have to pay all monies owed the telecom in their next billing cycle. Ireland’s telecom regulator was as unhappy with Eircom’s payment demand as its customers were, even more so because the utility also failed to inform it of the billing error.

“Teething issues” also featured prominently in several apologies for IT fouls ups. Take, for instance, EnergyAustralia, which claimed that on-going “teething problems” with its newly introduced accounting system were why 145,000 of its customers had not been billed for their electricity or gas usage on time, including 21,000 who had never received a bill from the utility. In the apology the company issued, it tried to downplay the extent of the foul-up by saying, “We are sorry to the small number of customers who haven't had the best experience and we're working round the clock to improve our service.” However, for some 12,500 EnergyAustralia customers, the clock spun around for more a year before their billing issues were finally corrected.

Automotive manufacturer McLaren also apologized that its MP4-12C $229,000 supercar was suffering from “teething problems.” Ron Dennis, the executive chairman of McLaren Automotive and McLaren Group, sent out a letter to customers that stated in part, “As you will have already heard from my staff, we are experiencing some early software bugs resulting in unnecessarily sensitive warning lights, battery drainage in certain conditions and IRIS [infotainment] performance issues. My team and the McLaren retailers are working with the pace and intensity that the McLaren brand demands to fully resolve these bugs rapidly and effectively to ensure that any inconvenience to you is kept to a minimum.” Dennis, however, tried to make up for the inconvenience by promising customers that he was going to give them “a pre-release copy of the new McLaren: The Wins coffee-table book." I wonder how many software bugs it would take to get a personally signed copy of the book.

Additionally, there were a couple of organizations that had to make so many apologies that customers just stopped listening to them, deciding  to head for the exits instead. Take, for example, UK’s RBS Group which had a major system meltdown in the summer of 2012 caused by a routine software-update gone bad that kept millions of its bank customers from accessing their accounts for days, and some even for months. At the time, then Chairman Stephen Hester apologized, saying, “Our customers rely on us day in and day out to get things right. On this occasion we have let them down… Once again I am very sorry for the inconvenience.”

Various RBS Group spokespersons had to apologize several more times that summer as they promised everything would soon be made right, which quickly turned out not to be true.  At the time, RBS promised to invest hundreds of millions of pounds into upgrading its IT systems to keep major disruptions from happening again.

However, RBS has suffered significant glitches since, including in December 2013 on Cyber Monday and once more in June of this year. Although after each incident RBS management stated that it was “sorry for the inconvenienced caused” and that the incident was “unacceptable,” tens of thousands of its discomforted customers have decided to do their banking elsewhere.

While RBS may have seen droves of customers desert it over its IT failures, it is nothing compared to Australia’s telecom company Vodafone which has seen millions of its customers leave because of the company’s persistent IT ooftas. The root of the problem can be traced to 2009, when Vodafone merged its network with rival Hutchinson’s “3” network.  Not surprisingly, the merger’s objective of creating a high-quality, unified network across Australia wasn’t as easy or seamless as envisioned. Customer complaints about poor Vodafone service grew throughout 2010, but really came to a head when a network software upgrade in late 2010 didn’t work as expected. Instead of speeding up network traffic, the upgrade slowed it down. That problem took weeks to fix, angering legions of Vodafone customers. 

Then a different, concurrent software issue caused additional problems across the network. Vodafone, which by now was being referred to in the press as “Vodafail,” had to apologize to its angry customers multiple times that the company was “truly sorry” for the continued “dropped calls, delayed SMS and voicemails, slow data speeds, inconsistent coverage, and long waits when you called us.” For more than 2 million Vodafone fed-up customers who left the company between 2010 and 2013, the company didn’t improve fast enough. Finally, after spending AU$3 billion to upgrade its networks and customer support, Vodafone Australia announced earlier this year that it had started adding customers again.

There was also an interesting apologetic non-apology that happened in my home state of Virginia. In the summer of 2010, a server problem at the Virginia Information Technologies Agency (VITA) knocked out the IT systems used by 27 of Virginia’s 89 state agencies for several days, and a number of agencies were affected for over a week. At the time, the state’s IT infrastructure was in the midst of a $2.3 billion upgrade which was a constant source of contention between Virginia and its contractor Northrup Grumman.

When the server problem was finally fixed, Northrop Grumman vice president Linda Mills put out the expected pabulum and said the company “deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens.” However, Grumman’s “regrets” were immediately undercut by a company spokesperson who, when asked by a Richmond Times-Dispatch newspaper reporter whether Mills’ statement was an apology, declined to comment. Whatever little goodwill that was left in state government for Northrop Grumman quickly vanished.

In May of 2011, after an investigation into the outage, NG agreed to pay a $4.7 million fine for the outage, which is an apology of the best kind, in my opinion.

Our final apology was given through firmly clenched teeth. For years, Her Majesty's Revenue and Customs (HMRC) in the UK worked to upgrade (pdf) its troubled computer systems. HMRC promised that when complete, the new PAYE (pay as you earn) system would significantly reduce both over- and underpayments of taxes. When it was fully introduced in 2010, the HMRC announced that some 4.3 million UK taxpayers were going to receive letters stating that they had paid on average £400 too much in taxes between 2008 and April of 2010. Additionally, another 1.5 million would be receiving letters just before Christmas that they had paid on average £1,428 too little over the same period, and HMRC wanted its money now. Furthermore, the HMRC indicated that another 6 million taxpayers prior to 2008 were likely owned money for taxes paid previous to 2008, and another 1.7 million possibly owed more taxes. This group would be receiving letters soon, too.

The underlying reason for the millions of over-and under-payments was that taxpayers were being placed in an incorrect tax bracket for years because of errors in the new HMRC PAYE computer system database.

Needless to say, the UK public was not a happy bunch at the news. Fueling their unhappiness was the attitude of HMRC Permanent Secretary Dave Harnett, who stated in a radio interview there was no need for him or his department to apologize for the errors or demand for quick payment of owed taxes, because at least to him, the PAYE system was working as designed: “I'm not sure I see a need to apologise... We didn’t get it wrong.”

Politicians of both parties were appalled by that statement, calling Harnett out of touch and arrogant, especially in light of all the reported PAYE system foul-ups. Harnett was forced by senior Conservative party leaders to retract his statement the following day, saying that he was “deeply sorry that people are facing an unexpected bill.”

A few days later, however, HMRC CEO Dame Lesley Strathie made it very clear Harnett’s apology was really a non-apology when she insisted  that HMRC staff made “no mistakes,” and any and all errors were due to taxpayer mistakes. Dame Strathie also said the critiques of HMRC's performance was unfair since it couldn't pick and choose which customers to serve—it had to deal with everyone—whether her government department liked it or not. That bit of insight didn’t go over well, either.

HMRC’s PAYE system continues to cause grief to UK taxpayers, and the HMRC is taking yet another crack at updating its computer systems. Unfortunately, UK taxpayers don’t have a choice of which tax department to use, like customers of RBS or Vodafone do with banks or telecom companies. 

If you have some “worst IT foul-up apologies of all time” stories to add, please let me know.

The Making of "Lessons From a Decade of IT Failures"

In the fall of 2005, IEEE Spectrum published a special report on software that explored the question of why software fails and possible ways to prevent such failures. Not long afterwards, we started this blog, which documents IT project and operational failures, ooftas, and other technical hitches from the around the world.

The tenth anniversary of that report seems to be appropriate time to step back, look through 1750 blog posts, and give an overall impression of what has and hasn’t changed vis a vis software crashes and glitches. Obviously, the risk in both the frequency and cost of cybercrime is one major change, but we decided at least for now to concentrate our limited resources on past unintended IT system development and operations failure.

Read More

Stuxnet-Style Virus Failed to Infiltrate North Korea's Nuclear Program

The famous Stuxnet computer virus that sabotaged Iran’s nuclear program apparently had a cousin designed to do the same to North Korea. But this other U.S. cyber attack failed because agents could not physically access the isolated computers of North Korea’s nuclear program.

Read More

Fuzzy Math Obscures Pentagon's Cybersecurity Spending

U.S. military spending has increasingly focused on cybersecurity in recent years. But some fuzzy math and the fact that funding is spread out among many military services makes it tough to figure out exactly how much money is going toward cybersecurity. That in turn makes it difficult to understand whether each dollar spent really improves the U.S. military’s cyber capabilities.

Read More

Is the Lenovo/Superfish Debacle a Call to Arms for Hacktivists?

As Lenovo has come under fire for pre-installing on their computers the intrusive Superfish adware — and as lawsuits are now being filed against the laptop-maker for compromising its users’ security — one solution to the problem may have been given short shrift. Maybe it’s time, in other words, to release the hackers.

To be clear, nothing here should be read as an inducement to any sort of malicious hacking or other nefarious cyber-activities. The call to arms is instead to hacking in the old Homebrew Computer Club, touch-of-code-and-dab-of-solder sense. After all, when pop-up ads became a scourge of the late 1990s Internet, coders behind the smaller Opera and Mozilla browsers rolled out their pop-up blockers to restore a touch of sanity. Major commercial web browsers like Internet Explorer and Safari only rushed in after the nimbler first responders proved the consumer demand.

Read More

Should Data Sharing Be More Like Gambling?

When you install a new app on your phone, you might find yourself facing a laundry list of things the software says it needs to access: your photos folder, for example, along with your camera, address book, phone log, and GPS location.

In many cases, it’s an all or nothing deal. 

Eric Horvitz of Microsoft Research says companies could do better. Instead of asking users to provide wholesale access to their data, they could instead ask users to accept a certain level of risk that any given piece of data might be taken and used to, say, improve a product or better target ads.

“Certainly user data is the hot commodity of our time,” Horvitz said earlier this week at the American Association for the Advancement of Science, or AAAS, meeting in San Jose. But there is no reason, he says, that services “should be sucking up data willy-nilly.”

Instead, he says, companies could borrow a page from the medical realm and look for a minimally invasive option. Horvitz and his colleagues call their approach “stochastic privacy.” Instead of choosing to share or not to share certain information, a user would instead sign on to accept a certain amount of privacy risk: a 1 in 30,000 chance, for example, that their GPS data might be fed into real-time traffic analysis on any given day. Or a 1 in 100,000 chance that any given Internet search query might be logged and used.

Horvitz and colleagues outlined the approach in a paper for an American Association for the Advancement of Artificial Intelligence conference last year.

If companies were to implement stochastic privacy, they’d likely need to engage in some cost-benefit calculations. What are the benefits of knowing certain information? And how willing would a user be to share that information? 

This sort of exercise can turn up surprising results. In an earlier study, Horvig and Andreas Krause (then at Caltech, but now at ETH Zurich) surveyed Internet search users to gauge their sensitivity to sharing different kinds of information. More sensitive than marital status, occupation, or whether you have children? Whether the search was conducted during work hours. 

Of course, even if a company works out what seem to be reasonable risks for sharing different kinds of data, what it might look like on the user end is still an open question. How do you communicate the difference between a 1/30,000 and a 1/100,000 probability? 

Horvitz said that would be a good problem to have. “Would you want to live in a world where the challenge is to explain these things better,” he asked, “or where companies scarf up everything?”

Rooting Out Malware With a Side-Channel Chip Defense System

The world of malware has been turned on its head this week, as a company in Virginia has introduced a new cybersecurity technology that at first glance looks more like a classic cyberattack. 

The idea hatched by PFP Cybersecurity of Vienna, Va., is taken from the playbook of a famous cryptography-breaking scheme called the side channel attack. All malware, no matter the details of its code, authorship, or execution, must consume power. And, as PFP has found, the signature of malware’s power usage looks very different from the baseline power draw of a chip’s standard operations.

So this week, PFP is announcing a two-pronged technology (called P2Scan and eMonitor) that physically sits outside the CPU and sniffs the chip’s electromagnetic leakage for telltale signatures of power consumption patterns indicating abnormal behavior.

The result, they say, is a practically undetectable, all-purpose malware discovery protocol, especially for low-level systems that follow a predictable and standard routine. (Computers with users regularly attached to them, like laptops and smartphones, often have no baseline routine from which abnormal behavior can be inferred. So, PFP officials say, their technology is at the moment better suited to things like routers, networks, power grids, critical infrastructure, and other more automated systems.)

“On average, malware exists on a system for 229 days before anyone ever notices anything is there,” Thurston Brooks, PFP’s vice president of engineering and product marketing told IEEE Spectrum. “What’s really cool about our system is we tell you within milliseconds that something has happened.”

PFP—an acronym for “power fingerprinting”—requires that its users establish a firm baseline of normal operations for the chips the company will be monitoring. So they begin with P2Scan, a credit-card-size physical sensor that monitors a given chip, board, device, embedded system, or network router for its electromagnetic fingerprints when running normally.

Unlike most malware strategies in the marketplace today, PFP takes a strikingly software-agnostic tack to besting malware, hardware Trojans, and other cyberattacks.

“We’re not trying to actually understand what’s going on inside the machine, like the hackers are,” says Brooks. “We’re trying to define what normal behavior looks like. Then, knowing [that], we can detect abnormal behavior.”

The view of malware as seen from outside the chip, in other words, can be a refreshing one. Hackers can’t detect this type of surveillance, because the scanning tools never actually interact with the chip’s operations. And hackers can be as clever as the most sophisticated programmers in the world. Yet, their code will still very likely be detected because, simply by virtue of performing different tasks than the chip normally performs, it will have a different power profile.

“I am a signal processing guy,” says PFP president Jeff Reed, who is also a professor in the ECE department at Virginia Tech. “Our approach is a very different approach than a person who’s normally schooled in security…We’re trying to understand a disturbance in the signal due to the inclusion of malware.”

Reed and Brooks also point out that counterfeit chips are a vast problem in IT, as Spectrum has documented in recent years. By the FBI’s estimates, for instance, chip counterfeiting costs U.S. businesses some $200 to $250 billion annually.

The problem is just as daunting for the U.S. military, as Spectrum has also chronicled. For example, an investigation by the U.S. Senate Committee on Armed Services uncovered counterfeit components in the supply chains for the CH-46 Sea Knight helicopter, C-17 military transport aircraft, P-8A Poseidon sub hunter and F-16 fighter jet.

The problems were expensive but ultimately rooted out. Yet other dangers remain—especially in such high-security realms, where substandard components could endanger troops and missions, or compromised chips could be used to carry out malicious plots.

But any compromised chip—whether hardware-Trojan-laden or part of a single lot of subpar chips coming from the foundry—can be discovered using their system, PFP says.

The trick, says Brooks, is to grab a sample chip from a lot and perform a (typically expensive) decapping, x-ray analysis, and reverse-engineering of the chip’s code. Then, once it’s been confirmed that the chip works as designed and is within spec, it is run through a standard operation, providing an electromagnetic baseline for P2Scan and eMonitor.

Every other chip in the lot can then be rapidly and cheaply tested against the “gold standard” chip by running the same standard operation and comparing the resulting electromagnetic signature to that of the first chip.

“You determine whether you have a good chip or not,” Brooks says. “You only spend the money to do that on one chip…So you amortize the cost of the forensics across all the chips. So if you have a few million chips, you’re talking about a few pennies [per chip] to do the whole thing—and know that you have a million chips that are all good.”

Cyber Espionage Malware Taps Smartphones, Sends Chills

A mysterious malware campaign resembling an attack on Russian officials from earlier this year could be the most sophisticated cyberattack yet discovered.

This fall, around the time hackers were draining crucial digital lifeblood from Sony Pictures, one of the most sophisticated malware attacks in history (completely separate from the Sony hack) was coming to a close. Presumably retreating after being exposed by security researchers, the cyber espionage campaign targeted smartphones of business, government, and embassy officials around the world. Its structure parallels an earlier attack aimed primarily on Russian executives, diplomats, and government officials, but  the extent of the damage inflicted by the recent campaign—as well as its prospects of returning under a new guise—is still unknown.

Waylon Grange, senior malware researcher at Blue Coat Labs in Sunnyvale, Calif., says he’s taken apart both the malware that infected Sony Pictures’ internal networks and the malicious code behind the Russian hack. And in terms of the relative complexity and sophistication of the designs—though of course not by the level of damage—there’s no contest.

“In terms of sophistication, the Sony malware is really low on the pecking order,” he says. “The Sony malware was more destructive. This one is very selective. When it runs, this one does very well tracking its steps. If anything is wrong or the system is not configured just right, this malware detects it, quietly backs off, doesn’t make any errors, cleans itself up and is gone.”

As a result, Grange says, it's been a difficult cyber infection to study and trace. And its code base and Internet routing are so full of false leads and red herrings that it has, to date, proved impossible to source back to any group, nation, or band of hackers. Whoever it is, Grange says, has assembled a next-generation attack that should make security researchers sit up and pay attention.

And, especially in light of how much horrible mischief the far simpler Sony attack has wrought, businesses and governments should also be educating their workforces on cybersecurity and installing more and better locks on their cyber doors and windows.

In a blog post earlier this month, Grange’s colleagues at Blue Coat unveiled the details of the attack, whose infection route begins with a spearphising e-mail to targeted business, government, and diplomatic users in at least 37 countries. The e-mail poses as an update or special offer for users to download the latest version of What’s App. Unfortunate users who click this link download infected Android, Blackberry and iOS versions of the popular messaging app.

An infected smartphone then records calls made by the user and awaits instructions telling it the Internet address to which it should upload the surreptitiously recorded phone calls.

Such an attack would already be remarkable and impressive, Grange says. But it’s only the first of at least two more layers of command and control structure for the malware campaign.

In the second step, apps check a redundant list of hacked public blogs whose posts contain legitimate text at the top (presumably in order to avoid being de-listed by search engines or otherwise flagged) followed by long strings of encrypted code. The malware then decrypts the code, providing itself a list containing a second set of command and control websites.

These sites, the researchers found, are often compromised Web pages run on outdated content management software in Poland, Russia, and Germany. It’s at these second-tier websites that the malware then decodes its rapidly changing list of drop-zones for offloading the phone call recordings.

Earlier this year, Blue Coat also detected and studied a similar multilayered Windows-based attack that was carried out primarily in Russia. It began with an infected Microsoft Word document that then infected a PC, causing it to follow an even more carefully guarded and circuitous route for receiving instructions. Subsequently infected PCs would first search a series of hacked cloud service accounts, which in turn would point to hacked embedded devices around the world (including wireless routers, satellite TV boxes and digital TV recording devices). Those compromised devices would in turn point back to virtual private networks that contained the instructions for the malware.

Disassembling the infected code, Grange says, led security researchers to multiple conflicting conclusions about its authors. One piece of the infected Android app contained the Hindi character for "error." Several of the infected blog profiles have set their location to Iran. Many infected home routers are in South Korea. Text strings in the Blackberry malware are in Arabic. Another contained the comment “God_Save_The_Queen.”

It was the many layers of red herrings and command and control, Grange says, that inspired Blue Coat to call the original (Russian) malware “Inception,” in homage to the 2010 thriller that contains onion-like layers of story to be peeled away. Blue Coat hasn't explicitly named the smartphone cyberespionage attack, though they strongly suspect it's either by the same hackers or strongly inspired by the "Inception" malware.

“These people are going to great lengths to protect who they are,” he says. “We’ve seen [attackers] use the cloud. But we’ve never seen routers, and we’ve never seen anyone use cloud, router, and private services to hide their identity.”

Grange says the smokescreens have worked so far; he has yet to establish any solid leads on who could have conducted these sophisticated attacks. Yet the lessons learned from the attacks, he said, are not nearly as mysterious. Among them:

• Don’t click links in your e-mail browser—especially in any e-mail from an unknown user, or strange e-mails from known users.

• Don’t root your phone. Because the iPhone, for instance, doesn’t allow for updates outside of the iTunes store, Inception wouldn’t work on a non-rooted iPhone.

• Only update mobile apps through your trusted app store (e.g. iTunes or Google Play).

• Always change the default passwords (“admin,” “password,” etc.) for your household devices.

“We probably haven’t seen the end of these guys,” Grange says. “I’m sure they’ll come back. It’s just a matter of how long have we set them back—and what will be their new toys when they come back.”


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More