Risk Factor iconRisk Factor

California’s Payroll Project Debacle: Another $50 Million Up in Smoke

Ah, I love the smell of napalmed IT projects in the morning!

Not, though, when they are government IT projects and the wafting odor is from taxpayer monies going up in smoke.  And unfortunately, for past few weeks, the stench of burning government IT projects has been especially pungent.

We start off in California, where after burning through some $50 million, California State Controller John Chiang announced last Friday he had decided to terminate the state’s US $89.7 million contract “with SAP as the system integrator for the MyCalPAYS system, the largest payroll modernization effort in the nation.” The planned 5-phase effort mercifully never made it past the first pilot phase.

Furthermore, Chiang said that the Secretary of the California Technology Agency (CTA)  has “suspended further work until the CTA and SCO [State Controller’s Office] together conduct an independent assessment of SAP’s system to determine whether any of SAP’s work can be used in the SCO’s go-forward plan to address the State’s business needs.”

You may remember that Chiang sent SAP a letter last October warning that the project was “foundering and is in danger of collapsing,” and gave SAP one last chance in the form of a demand for urgent get-well efforts from the company. Chiang claimed that there were errors in one out of every three tasks performed by SAP's system, and that there hadn’t been a single pay cycle without material payroll errors occurring.

In Friday’s announcement, Chiang threw in the towel. He said that while he had hoped “for a successful cure to SAP’s failure to deliver an accurate, stable, reliable payroll system, SAP has not demonstrated an ability to do so.” This was especially disheartening, Chiang implied, given that the SAP effort covered only 1300 SCO employees who had “fairly simple payroll requirements.”  There was no way the SAP system could be trusted to support the payroll requirements of the state's "240 000 employees, operating out of 160 different departments, under 21 different bargaining units."

SAP said in response to the news of its contract termination that it was “extremely disappointed in the actions. SAP stands behind our software and actions.... SAP also believes we have satisfied all contractual obligations in this project.”

All of this, of course, suggests that when the napalm smoke clears, a date in court will be in the offing. Chiang as much as said so in the announcement: “The SCO will pursue every contractual and legal option available to hold SAP accountable for its failed performance and to protect the interests of the State and its taxpayers. This includes contractually required mediation and, if necessary, litigation.”

An SCO spokesperson called the project’s performance “frightening,” but what must be really frightening to California taxpayers is the continued inability of the state to manage the acquisition of its IT projects. So far, nearly $254 million has been spent so far in two unsuccessful attempts to get a state government payroll system in place, the LA Times reports. If SAP fights instead of settles, it would at least be a public service, exposing the depth of California’s IT project risk mismanagement.

The upshot is that California will continue to use its decades-old Cobol-based payroll system until it figures out what to do next. And to help it figure that out, the SCO has—in the best tradition of government—set up an IT Procurement Task Force. Whenever in doubt, form a committee.

I hope the Task Force members have strong stomachs; the stench of IT project failure coming out of California is of the mephitis variety.

Read More

IT Hiccups of the Week: University of Wisconsin Loses Another $1.1 Million Amid Payroll Glitches

This week’s IT hiccups and snafus are a varied lot. We’ll start off with the University of Wisconsin’s ongoing payroll and benefits system saga.

$1.1 Million Lost Because of Glitches in UW Payroll System Glitches – More May Follow

The Wisconsin State Journal reported last week that “glitches” with the University of Wisconsin’s controversial payroll and benefits system had resulted in US $1.1 million in improper payments which the university may likely end up having to absorb. In addition, the Journal reported, University President Kevin Reilly warned that further examination of the payroll system “by system staff, an independent analyst and the state auditor are ‘likely to find more issues.’”

This news has not gone over well with Wisconsin state legislators, who were already upset when an audit by the Legislative Audit Bureau released late last month indicated that problems with the UW payroll system had resulted in $33 million in improper payments being made over the past two years. Another Journal article reported that while some $20 million of those $33 million in overpayments have been recovered, much of the remaining $13 million may well have to be written off.

When the $33 million in overpayments was first reported, UW's Reilly put out a statement that said in part, “I am deeply troubled by these mistakes…. We will identify exactly why and how these significant errors occurred, we will validate that steps we have already taken are working, we will take any additional steps that need to be taken, and we will make absolutely sure that similar errors do not happen again.”

Read More

This Week in Cybercrime: Former State Government Employee Used Driver’s License Database Access to Snoop on Thousands

Minnesota Government Employee Wrongfully Accessed Driver’s License Data

It’s hard enough to keep your personal information out of the hands of cybercriminals bent on using it to steal from you or fraudulently acquire things in your name. But it seems like there’s no hope when organizations you trust with your personal details—like the Minnesota Department of Public Safety—mishandle them. That was likely the case for roughly 5000 state residents who found out this week that a former state employee has been charged with illegally accessing the records associated with their driver’s licenses. The data thief, who was once the state's Department of Natural Resources Enforcement Division's administrative manager, was authorized to look at a resident's records when they related to his office’s official business. But between 2008 and last October, he used his credentials to query the state Driver and Vehicle Services database more than 19 000 times. He looked up the names of politicians, judges, county and city attorneys, police officers, news reporters, family members and other state employees. Most of his downloads were of women whose pictures appeared in the database.

According to a Kaspersky Lab Threatpost article, four people who have been notified that their records were wrongfully accessed are suing the alleged perpetrator and other state employees. “They said the data breaches caused severe emotional stress and physical harm and were the result of ‘lax policies and lax enforcement’ that allowed an unsupervised, unmonitored Hunt to continually access records for years,” says the Threatpost article.

Government Agencies, Military Among Users of Vulnerable Industrial Control System

What do the FBI, the Drug Enforcement Agency, the U.S. Marshals Service, the IRS, the U.S. Passport Office, the British Army, and Boeing, have in common? They are just a few of the thousands of organizations whose facilities depend on an industrial control system with a security hole that could allow attackers to remotely control critical building functions such as electronic door locks, lighting systems, elevators, electricity and boiler systems, video surveillance cameras, alarms. The vulnerability in the Tridium Niagara AX Framework was reported on 5 February at the Kaspersky Security Analyst Summit.

Billy Rios and Terry McCorkle, security researchers with Cylance, demonstrated a zero-day attack that yields access to the system’s config.bog file, which holds login credentials and other data for operator work stations, and controls the systems that are managed by them. The exploit, say Rios and McCorkle, takes advantage of a vulnerability that gave them root on the system’s platform. “The platform is written in Java, which is really, really good from an exploitation standpoint,” Rios told Wired. “Once we can own the platform, a lot of the other stuff is very, very straightforward [to attack],” said Rios.

Rios and McCorkle reported that a search turned up roughly 21 000 Tridium systems that were accessible over the Internet.

In a written statement, Tridium revealed that the researchers notified it about the vulnerability in December; it has been working on a patch, which it says it expects to release by 13 February. In an attempt to downplay the vulnerability, the statement noted that, “The vast majority of Niagara AX systems are behind firewalls and VPNs—as we recommend—but clearly, as Rios and McCorkle have shown, there are many systems potentially at risk.” That’s a change of tune from Tridium’s stance just last year, when it told the Washington Post that its systems benefited from security through obscurity.

Tried-and-True Thieving Techniques Taken Up Again

Cyberthieves have developed sophisticated malware that can infiltrate a victim’s computer, allowing a thief to tap into online banking sessions initiated by customers in real time. Such malicious code is capable of conducting fraudulent transactions right under the victim’s nose and covering its tracks by updating the account balance and transaction history display in the victim’s browser. But because banks have developed countermeasures including software that detects anomalies in customers’ online access, some crooks are eschewing session hijacking and going back to the old and familiar: stealing login credentials for subsequent access from a separate computer. This shift was confirmed by researchers at security firm Trusteer, who reported this week that they noticed changes in the Tinba and Tilon financial Trojan programs. According to a 7 February blog post by Amit Klein, Trusteer's chief technology officer, the Trojans divert a customer attempting to access his or her bank’s website to a fake version. The rest is history, says Klein:

“Once the customer enters their login credentials into the fake page the malware presents an error message claiming that the online banking service is currently unavailable. In the meantime, the malware sends the stolen login credentials to the fraudster who then uses a completely different machine to log into the bank as the customer and executes fraudulent transactions.”

Now banks have to be on the lookout for both the new and (relatively) old-school techniques.

Adobe Releases Emergency Security Update

On 7 February, Adobe released a patch for its Flash Player meant to stop hackers from using two zero-day vulnerabilities to take over Windows PCs and Macs. Adobe was already planning to release a Flash Player update on 12 February, but because the software maker was “aware of reports that CVE-2013-0633 is being exploited in the wild in targeted attacks designed to trick the user into opening a Microsoft Word document delivered as an email attachment which contains malicious Flash content," it released the fixes as soon as they were ready. The other vulnerability was being used for so-called drive-by attacks that victimize computer users who navigate to a malicious website hosting an exploit.

UK Government Reasserts Its Right to Snoop on All Electronic Communications

Last April Fool’s Day, the BBC reported that the UK government was planning to introduce legislation that would allow the monitoring of all the “calls, emails, texts and website visits of everyone in the UK” by the Government Communications Headquarters (GCHQ) intelligence agency. The information would be monitored in real-time and then stored for two years before being erased. The government needed the monitoring capability, it said, to be able toinvestigate serious crime and terrorism and to protect the public.”

The government also promised that the legislation would “ensure that the use of communications data is compatible with the government's approach to civil liberties.”

It's good to see the tradition of doublethink is alive and well in the UK.

Almost immediately, members of even the government’s own party said that this legislation was a massive overreach and threatened civil liberties. Telecommunication and Internet providers weren’t too happy either, saying that the program was going to be expensive and a nightmare to implement.

A pre-legislative parliamentary scrutiny committee was set up to look into the feasibility of the proposed legislation, now being dubbed the “snoopers charter.” By late autumn, word was that the committee did not like what it saw and was preparing to say so in a report in early December. The UK Home Secretary, Theresa May, was aggressively pushing the legislation and on 3 December, upon hearing of the committee’s unflattering appraisal of it, launched a preemptive strike on the committee’s findings. She told the Sun newspaper that the legislation had to be passed, otherwise “we could see people dying” and “criminals going free” including “pedophiles who will not be identified.” She also warned of a reduction in “our ability to deal with this serious organized crime.”

May concluded, “Anybody who is against this bill is putting politics before people’s lives.”

However, the committee was unimpressed by May’s "you are either with us or against us" attack.  On 10 December, the Guardian published a story detailing the committee's determination that the legislation was unworkable as written, that it “tramples on the privacy of British citizens,” and further that the estimated cost of the effort of some £1.8 billion over 10 years was “fanciful and misleading.” Nick Clegg, the leader of the government’s Liberal Democrat coalition party, told May, “We cannot proceed with this bill and we have to go back to the drawing board."

So politics and common sense won out, at least for a little while.  There were warning signs that this wouldn't last, however. While May stated that she was “open-minded” about changing the legislation, the Guardian reported that she “remained determined to introduce it before the session ends next spring and get it on the statute book before the next election.”

This week May's snooping desires got a boost as the London Telegraph reported that the cross-party parliamentary Intelligence and Security Committee (ISC) has come out in support of the "snoopers charter," though it also warned that the “the government must do more to convince public of the need for them.”  Hmm, sounds like it time to beat the “it’s all for the sake of the children” drum a bit louder, or maybe, to say, a la Orwell, that the charter is needed as an “act of self-defense against a homicidal maniac.”

According to the Telegraph, the Director General of MI5, Jonathan Evans, said that without the legislation, “it was increasingly difficult to be confident that targets were being fully watched” because of rapid changes in communication technology. And in a related story at the Guardian, the Home Office claims that the charter is urgently needed as “there is already a 25 percent ‘capability gap’ between the tracking data that the security services need to access and their ability to do so.”

Evans did admit to the ISC, though, that the Home Office’s 25 percent figure depended upon some “pretty heroic assumptions,” the Guardian reported. In other words, it was most likely a number that made for a good news sound bite, but that the capability gap has little credibility indeed.

A story at the Daily Mail reports that the UK's intelligence service says it isn't interested in unfettered access to the content of every communication, and that its fetters would still be court orders, which it would continue to obtain. It just wants information on “who sends a message, where and how it is sent, and who receives it.”

Of course, with people's identities closely bound with their cellphones, and with all the GPS and other information that cellphones throw off these days, this metadata is often more important than the information content itself, much of which, by the way, can probably be inferred pretty quickly with advanced data analytics. And if the messages are passing though the communication channels being monitored by the U.S. National Security Agency, the contents can probably be provided to GCHQ without a UK court order request even being filed.

The Daily Mail article also points out that GCHQ isn’t worried whether the messages are encrypted, either. Apparently, it has “options” to deal with it.

How this all plays out, time will only tell. But the idea of a democratic government that maintains its belief in its citizens' right to privacy also claiming in the same breath it also has a right to snoop on all forms of electronic communication reminds me of another Orwell quote: “We have now sunk to a depth at which restatement of the obvious is the first duty of intelligent men.”

Image: iStockphoto

IT Hiccups of the Week: Digital Navigation Error Leads to Dismantling of U.S. Navy Ship

There was a real potpourri of IT-related glitches, snarls, and snafus to choose from last week. We start off with the lingering after-effects of the grounding of the USS Guardian on a Philippine reef—which we first noted a few weeks ago.

U.S. Navy Decides to Scrap Minesweeper Stuck on Ecologically Sensitive Philippine Reef

On 17 January, the U.S. Navy minesweeper USS Guardian ran hard aground on a reef within the protected Tubbataha Reefs Natural Park in Philippine waters where it remains stuck. A preliminary assessment indicates that the ship was following a National Geospatial-Intelligence Agency-supplied Coastal Digital Nautical Chart (DNC) that “misplaced the location of a reef by about eight nautical miles.” The reef is located in a UNESCO World Heritage restricted zone, and any damage caused to the reef is heavily fined.

The Navy had hoped that it could wrestle the USS Guardian free without too much damage to the reef or the ship, but those hopes were dashed when the 23-year-old wooden-hulled ship started taking on water.  As a result, the Navy decided that the best option was to dismantle the ship and remove it as three separate sections. A floating crane from Singapore is being brought in to help with the ship’s removal.

An interesting story last week at the website Maritime Accident Casebook indicates that the navigational snafu has been attributed to human error at the National Geospatial-Intelligence Agency (NGA). According to the story, the NGA decided to update its navigational charts in 2008 using  LANDSAT-derived imagery because of the age and uncertainty of information shown on the nautical charts in that area of the Pacific (some dating back to 1940 and 1942, an earlier MA Casebook article says). Some of the old charts even indicated the presence of “phantom islands.”

Quoting an NGA spokesperson, “One of these images included incorrect information about the location of the section of ocean that includes the Tubbataha Reef. At the time, no other source information existed to validate that imagery data. As a result, the reef was incorrectly placed in the DNC.”

Then, in 2011, the NGA became aware of the error, and corrected all the charts except one: that being the one for the area around the Tubbataha Reef. According to the NGA, this was a result of “a failure to follow established procedure.”

In the wake of the incident, the NGA has reexamined charts covering “more than 116 million square nautical miles of ocean” and found only one other error of a “magnitude similar to the misplacement of the Tubbataha Reef.” That one corresponded to an area off the coast of Chile. Mariners have been warned of the discrepancy.

The Navy expects that it will take about a month to remove the USS Guardian. The fine to be levied is unknown, but it is likely to be substantial. The political price may be substantial as well.

Technical Issues Hit Amazon, Bank of America, PayPal  and Twitter

A cluster of IT glitches last week hit some well-known companies. First, on Monday, there were reports that PayPal customers ended up being charged multiple times for their transactions over a period of about three hours. PayPal has strongly denied The Register's claims the problem lasted 15 hours. A story at FierceCIO says that the multiple-charge problem was the result of instant payment notifications that were delayed in being sent back to customers. The delay caused many customers to think their PayPal transactions didn’t go through, causing them to make one or more additional payments. PayPal says that, “All customers will be refunded for duplicate transactions as soon as possible.”

Then on Thursday, Amazon suffered a 49-minute outage that made its homepage inaccessible, although it said that its other pages were fine. Amazon has been closed-mouthed about what caused the outage, other than to say it wasn’t hacked nor was it a problem with its cloud. It has been estimated that the cost to the company will be around $5 million in lost revenue.

Also on Thursday, Twitter said in a message to its users that there were “intermittent issues affecting Web and mobile users, globally, between approximately 7:00am and 9:50am PST.” The message  went on to say that, “We apologize to users who were affected by this, and we’re working to ensure that similar issues do not occur.” The message did not say what those issues were.

In an apparently unrelated matter, Twitter then announced Friday that it had “discovered one live [security] attack and were able to shut it down in process moments later. However, our investigation has thus far indicated that the attackers may have had access to limited user information—usernames, email addresses, session tokens and encrypted/salted versions of passwords—for approximately 250,000 users.”

As a precaution, Twitter has “reset passwords and revoked session tokens for these accounts.” Affected users will be receiving an email asking them to reset their passwords; if you get one, just be careful it isn’t phish.

Also on Friday, Bank of America said that its electronic banking operations and telephone call centers were inaccessible. The Washington Post reported that the problem was caused by unexplained “technical issues” rather than a cyber attack. And according to a story at the BBC which coincidentally was published on Friday, no one should be surprised by similar outages at other banks this year because of the ever increasing complexity of banking software.

Another Week, Another Stock Market Gaffe

This week’s stock market gaffe happened Friday on India's National Stock Exchange. In this case, an error in the software being used by the brokerage Religare Capital Markets Ltd. caused TaTa Motors' stock price to fall by 10 percent. Religare was quoted by Bloomberg News as saying, “Due to some technical issue in the software, unintended transactions got executed.”

Bloomberg said that the error will likely cost the brokerage some 100 million rupees (around US $1.8 million).

Last year you may recall there was another trading glitch that caused the National Stock Exchange (NSE) Nifty index to plunge over 800 points in a few minutes, wiping out some $58 billion in value from the fourth largest market in Asia.

Saving London’s Iconic Black Cabs – At Least for Now

Finally, last October, I noted that Manganese Bronze, the maker of the iconic London black taxi, announced that it was going into administration—the U.K. version of U.S. bankruptcy law's Chapter 11. The reason was an accounting error that went unseen for over two years when the company switched to new accounting software. The result: the company understated by £3.9 million its historical losses. Given the poor economic health of the company and the intense competition in London’s taxi market, Manganese Bronze stock took a nosedive when the accounting error became public. It looked like only a matter of time before the company, which was then worth roughly £5 million, would go belly up.

Fortunately, last week, Chinese car manufacturer company Zhejiang Geely, which already owned 20 percent of Manganese Bronze, decided to buy the rest of the company and its assets for £11.04 million “through a newly established British subsidiary, Geely UK,” the London Telegraph reported. The new owners say they are “confident” the business will be profitable within three years.

I hope so. London wouldn’t really be the same without those black taxis.


Photo: Naval Aircrewman 3rd Class Geoffrey Trudell/U.S.Navy

This Week in Cybercrime: Hackers Break Into News Outlets’ Computers

Hackers Break Into News Outlets’ Computers to Peek at Reporters’ Notes

On 30 January, the New York Times reported on its site that it was the victim of a sophisticated campaign of cyberattacks aimed, it suspects, at uncovering the names of sources who provided information about the business dealings of Chinese Prime Minister Wen Jiabao and his family. (In fact, we’re learning that the Times was only the latest publication to have its systems raided, but more on that later.) According to the NYT article, Chinese hackers—who tried to cover their tracks by infecting and remotely controlling computers at U.S. colleges then using those compromised machines to send the malicious code—started snooping around the Times’ internal networks as early as 13 September. This after word got out that journalists at the daily’s Shanghai bureau were conducting research into how Wen had amassed a fortune worth billions. According to a researcher at Mandiant, the computer security company the paper hired to exorcise the malicious code:

“[The hackers] set up at least three back doors into users’ machines that they used as a digital base camp. From there they snooped around The Times’s systems for at least two weeks before they identified the domain controller that contains user names and hashed, or scrambled, passwords for every Times employee.”

Mandiant discovered that the hackers used the passwords to access the computers of 53 Times employees. But Times Executive Editor Jill Abramson, who was quoted for the story, says, “Computer security experts found no evidence that sensitive e-mails or files from the reporting of our articles about the Wen family were accessed, downloaded or copied.” The Times was also quick to offer reassurance that no customer data was stolen. But what the hackers did in fact take is still an open question.

Even after the article about Wen was published on 25 October, the hackers continued snooping. The Times article references a December intelligence report prepared by Mandiant. The security firm had uncovered evidence that the “Chinese hackers had [from as far back as 2008] stolen e-mails, contacts and files from more than 30 journalists and executives at Western news organizations, and had maintained a ‘short list’ of journalists whose accounts they repeatedly attack.”

That assessment was confirmed on 31 January, when the Wall Street Journal admitted that hackers trying to monitor the newspaper's coverage of China, hacked into its systems. Bloomberg says it was targeted after publishing an article last June about Xi Jinping, China’s then vice president and current general secretary of the country’s Communist Party. But Bloomberg says that although its computer systems came under attack, they were never breached.

Thousands of Networked Gadgets Double as Gaping Security Holes

Computer World is reporting that faulty implementation of the Universal Plug and Play (UPnP) protocol standard has turned millions of network-enabled devices such as routers, printers, media servers, and even smart TVs into gateways through which hackers can get inside firewalls. On 29 January, security researchers from Rapid7 released a research paper in which they noted that more than 20 percent of the 80 million unique IP addresses they pinged exposed the UPnP Simple Object Access Protocol service to the Internet. This allows one networked device to discover another and remotely turn on the other gadget’s data sharing, media streaming, media playback control and other services. The Computer World article explains that:

“In one common scenario a file-sharing application running on a computer can tell a router via UPnP to open a specific port and map it to the computer's local network address in order to open its file-sharing service to Internet users.

Many had UPnP implemented through a library called the Portable UPnP SDK. Unfortunately, as the Rapid7 researchers discovered, UPnP SDK contains eight remotely exploitable vulnerabilities. Two of them can be used to inject code remotely.

The upshot: More than 23 million networked devices exhibited this vulnerability during the test. Rapid7 told Computer World that a patch has been released, but the firm’s chief security offer predicted in a 29 January blog post that “it will take a long time before each of the application and device vendors incorporate this patch into their products.”

The slow-to-update problem, says Rapid 7, also affects users of a UPnP library called MiniUPnP, which can be exploited for denial of service and remote code execution attacks. New versions released in 2008 and 2009 don’t contain those security holes. But according to Rapid7, 14 percent of the Internet-exposed UPnP devices it pinged were still using MiniUPnP 1.0 and were thus still vulnerable. Though Rapid7 has released a free tool called ScanNow for Universal Plug and Play, and a module that detects vulnerable UPnP services running inside a network, many vulnerable devices will remain unpatched.

“Many PC users don't even update PC software that they frequently use and are familiar with,” Thomas Kristensen, chief security officer at vulnerability research and management firm Secunia told Computer World. “The task of finding the Web interface of a vulnerable networked device, obtaining the firmware update and going through the whole update process will likely be too intimidating for many users,” he said.

Want to Use a Plug-in on Firefox? Ask For It

Mozilla announced this week that it would automatically disable all plug-ins in Firefox except the latest version of Adobe's Flash Player. In order for any plug-in to run, the user will have to manually override the block. This feature, which Mozilla calls “click-to-play,” used to bar only plug-ins that the Firefox browser judged to be unsafe or too far out of date. The move comes on the heels of numerous reports of hackers taking advantage of bugs in plug-ins, particularly the Java browser plug-in. The makers of other browsers such as Chrome and Opera include the click-to-play feature. But Mozilla is the first to turn it on by default. The others require the user to enable it.

Yahoo Mail Hijacking Case Solved

Security researchers at Australia-based BitDefender say they have gotten to the bottom of how some Yahoo Mail accounts have been hijacked over the past month. It seems that a link that is supposed to take them to an MSNBC News site, connects them with a domain registered in the Ukraine. Javascript that finds the user's contacts and sends spam under his or her name is placed on those pages so that its almost impossible not to click on it.

Bill Shocker Malware Spreading Like Wildfire in China

It was revealed this week that a new piece of malware dubbed “Bill Shocker” has infected at least 600 000 mobile devices in China. The malicious code, which targets several of the most popular mobile apps in China, including Tencent QQ Messenger and Sohu News, sends spam to the users’ contact lists—often costing mobile device users a lot of money by going beyond the number of messages included in the unsuspecting users’ messaging plans. In a 30 January blog post, Beijing- and Dallas-based NQ Mobile said that the malware can update itself and "automatically expand to other apps, multiplying the potentially disastrous effects.”

Photo: Jleon/Wikipedia

“Programmer Bob”: Latter-Day Tom Sawyer or Massive Security Risk?

At first I thought this was one of those IT urban legends, like the “disappearing warehouse” story, but according to Verizon's IT security risk team, it's all true.

A few weeks ago, Verizon wrote on its IT security blog that it was asked to perform a security assessment for a U.S.-based client after the latter was “startled” to discover a live “open and active VPN [virtual private network] connection from Shenyang, China!”

What made the client thoroughly worried about this surprisingly open communication port to China was first that it was a U.S. critical infrastructure company; second, it had two-factor authentication for its VPN connection, which had obviously been breached and, third, “the developer [given the pseudonym “Bob”] whose credentials were being used was sitting at his desk in the office.”

In other words, “the VPN logs showed [the developer] logged in from China, yet the employee is right there, sitting at his desk, staring into his monitor.”

It seemed unlikely that Programmer Bob was manipulating the space time continuum, so the client called Verizon's IT security team hoping for a more realistic explanation.

What Verizon discovered was that someone in China had been using Programmer Bob’s credentials to access the client’s computer systems for quite some time on almost a daily basis. The Verizon risk team theorized that Bob’s desktop workstation software had been somehow breached possibly via some zero day malware. So, the team decided to acquire a forensic image of Bob’s workstation to see if it could uncover this malware as well as how it got onto Bob's workstation.

Instead, what Verizon discovered were “hundreds of .pdf invoices from a third party contractor/developer in (you guessed it) Shenyang, China.”

According to the Verizon account, “As it turns out, Bob had simply outsourced his own job to a Chinese consulting firm. Bob spent less that one fifth of his six-figure salary for a Chinese firm to do his job for him. Authentication was no problem. He FedExed his physical RSA token to China so that the third-party contractor could log-in under his credentials during the workday. It would appear that he was working an average 9 to 5 work day.”

Nothing like exploiting a favorable date/time differential.

Programmer Bob, Verizon says, would spend the morning surfing Reddit for a couple of hours (watching cat videos), then take a long lunch, then spend the afternoon shopping on Ebay and updating his Facebook and LinkedIn. He did diligently return to his day job at the end of each day, to e-mail management on his work progress.

More interestingly, programmer Bob seems to have been able to pull off his outsourcing trick at multiple companies in his area. Exactly how wasn’t explained—I assume Bob didn’t have to be physically present at these other companies.

Verizon calculated that good old Bob looked to be earning “several hundred thousand dollars a year, and only had to pay the Chinese consulting firm about fifty grand annually.”

Even more interestingly—and here's where the blogosphere's ears really perked up—the client thought Bob was a superb employee. “For the last several years in a row he received excellent remarks. His code was clean, well written, and submitted in a timely fashion. Quarter after quarter, his performance review noted him as the best developer in the building.”

Verizon’s blog post naturally stirred up a lot of controversy. Some commenters claimed Programmer Bob was a hoax perpetrated by Verizon, or was in fact a Verizon employee. A later post by Verizon insisted that it was a true story, and that programmer Bob was not a Verizon employee.

What I found interesting was how, for everyone else, the story passed through myriad lenses of literary interpretation. Some saw Programmer Bob as a righteous example, a programmer Robin Hood who exercised the same prerogatives as managers who wantonly outsource jobs to China. This was the general take of an article in the UK Guardian, where Steven Poole wrote that Bob “has learned a harsh lesson: exploitation is a job for employers, not staff.”

Others took a slightly more modern perspective, comparing Programmer Bob to Mark Twain’s Tom Sawyer, who famously talked his friends into painting a fence he was tasked with. In this case, the role of Tom's eloquence was played by the pay differential between UK and Chinese programmers who do the same work (in this case, literally the same).

Even the Financial Times of London saw some merit in Bob's unorthodox arrangement, with popular columnist Lucy Kellaway asking, “If I outsourced my work, would you care?” Kellaway asks what the big deal is—lots of folks effectively outsource their work and no one seems to care. For example, she noted, “No one expects politicians to write their own speeches. We know many academics get their PhD students to do their research for them. Fashion designers don’t generally design their own clothes. Colonel Sanders doesn’t make his own fried chicken—though that is partly because he is dead.”

While tempted to outsource her own column, Kellaway admitted her ego “isn’t strong enough to deal with someone who is better at being me than I am.” That makes her Jerry Maguire to Programmer Bob's Ron Tidwell, the character who periodically shouted, “Show me the money.”

Still others viewed the story in the way that Verizon’s original post intended: a warning about how easy it is for a company’s IT systems to be breached by insiders, and how companies need to watch out for this. Yesterday’s Christian Science Monitor story on Bob’s exploits focused on this security angle.

I am sort of surprised that Bob hasn’t surfaced on daytime television yet. I wonder if it's because Bob, described as being a “mid-40’s software developer versed in C, C++, perl, java, Ruby, php, python, etc.,” a “ family man, inoffensive and quiet,” and “Someone you wouldn’t look at twice in an elevator” forgot to tell the appropriate authorities about his various sources of income.

Given a good lawyer, maybe the same one who helped another American icon, the singer Willie Nelson with his tax problems, perhaps Bob can have his folk hero status certified by Oprah. I, for one, would love to hear more about how he did it, though maybe 60 Minutes would be a better venue for the technical details.

And if Programmer Bob is reading this, you’re always welcome to tell your story here at the Risk Factor. Just don't outsource the interview.

Photo: Miha Perosa/iStockphoto

IT Hiccups of the Week: AT&T U-verse Bundle Suffers Three Day Hiccup

Last week was a relatively quiet week for IT uffdas, with the possible exception of subscribers to AT&T’s U-verse bundled service. A “small proportion” of them saw their service go away for three days.

AT&T's U-verse Multi-Day Server Complex Outage

Last Monday night, 21 January, subscribers to U-Verse, the bundled digital TV, high-speed Internet and voice service throughout the Southeastern United States, discovered that none of the services worked. They were unable to, among other things, make emergency calls. AT&T subscribers without the U-Verse bundle had service as usual, however. At first, the outage seemed to be of the annoying but short-lived variety, as subscribers told the Miami Herald that they lost their service at 9:30 P.M. local time and had it back at around 8:00 Tuesday morning.

AT&T encouraged the thought that the outage was a minor glitch, telling the Atlanta Journal-Constitution that it believed that the outage affected only some 6000 subscribers out of its 7.4 million customers spread across 22 states who have the service. AT&T explained that it was a server-related problem, and apologized “for any inconvenience to our customers.”

However, by later Tuesday afternoon, it became apparent that the outage was affecting roughly 75 000 (and likely more) subscribers to the U-verse bundling package. Reports started to come in revealing that the outage hit 14 states and spread as far west as California. AT&T still tried to put a positive spin on the outage, saying that the “issue currently affects less than 1 percent of our U-verse subscribers.” It added that it was working hard to fix the problem at its “server-complex”, but offered little further insight as to when the outage would be fixed or exactly what caused it.

By Wednesday, frustration, especially on the part of small businesses that depended on U-verse, was ratcheted up several notches as it became clear that many of them wouldn't have their service restored until Thursday. That was the case despite AT&T's assurances on Wednesday morning that, “U-verse service has been restored for the vast majority of our customers affected by the outage. We expect any remaining customer issues will be resolved this morning.”

The New York Times reported that AT&T finally gave a reason for the problem: a faulty software upgrade.

AT&T announced on Thursday afternoon that, “U-verse service has been restored for all customers affected by the outage. The software problem causing the issues was resolved by AT&T engineers early this morning. We are not pleased it took so long to fix the issue. AT&T will provide a credit to customers who were affected.”

The outage was a major embarrassment for AT&T, which had just boasted about the reliability of U-verse’s fiber optic and fiber-to-the-premises (FTTP) network  in early January. An AT&T spokesperson bragged that U-verse didn’t suffer outages like those of cable and satellite systems.  

Just to make AT&T’s week complete, equipment failure in Cleveland knocked out 911 emergency service as well as some landlines and 2G wireless service across northern Ohio, for about four hours on Friday; and a North Carolina hospital blamed a problem with AT&T Thursday for knocking its electronic health record system offline for seven hours.

New York Cabbies Lose Money Because of Wireless Connectivity Problem

The New York Daily News reported on Wednesday that a glitch in thousands of wireless communications devices installed in New York City taxis to allow riders to pay by credit or debit card. The malfunctioning systems, provided by Creative Mobile Technologies and connected to Sprint’s network, affected at least 2400 taxis’ card readers as well as their electronic fare meters, televisions, and navigation systems.

Taxi cab drivers were understandably angry about the glitch. Many pay US $120 or more per day to rent their cabs, and the malfunctioning meters rendered them unable to make any money (even with cash-paying customers) for nearly two days because of the glitch. Prospective taxi customers were none too happy either.

There was no word on whether Creative Mobile Technologies would be offering any compensation for the outage.

BATS Trading Error Dialed Back

About three weeks ago, BATS Global Markets, the third-largest U.S. stock exchange, announced that it had discovered during internal system audits two situations where “its computers allowed trades that violated [U.S.] rules intended to ensure all investors get the best prices for equities.” BATS stated that some 436,528 trades involving $420,000 were affected over four years.

But after having studied the suspected incorrect trades further, BATS presented dramatically revised numbers on Friday. According to the Wall Street Journal, BATS reported that there were only 12 000 bad trades involving $17 000.

When the errors were first announced, BATS CEO Joe Ratterman blamed it all on the complexity of the trading environment, which he in turn blamed on government regulators. I guess the complexity of market regulation wasn’t so bad after all.

Where's My W-2? National Grid Workers Still Unhappy With Payroll System

Last month, I wrote about Massachusetts Attorney General Martha Coakley warning National Grid—transmitter and distributor of electricity and natural gas to customers in New York, New Hampshire, Massachusetts and Rhode Island—to begin paying its employees correctly (including all of the overtime hours they worked in the aftermath of Hurricane Sandy) or else face a fine. One of the reasons for the lack of pay was that the National Grid went live with a new accounting system just before Sandy hit.  The changes needed in order to account for National Grid workers performing overtime in amounts and in locations outside of the ordinary created payroll havoc.

Coakley did impose a US $270 000 fine against National Grid earlier this month for its failure to pay its employees in a timely manner. Now it may be the Federal government’s turn to fine the company. According to the Boston Herald, National Grid has indicated to its employees that it might not be able to distribute the W-2 annual wage and tax statements needed to file taxes by 31 January as required, again because of problems with its accounting system. Failure to do so can mean a fine of up to $50 per W-2 statement from the U.S. Internal Revenue Service unless the IRS grants a waiver.

The Herald story says National Grid is insisting publicly that it will meet the deadline, but according to the employee union, the company is telling workers something different. We’ll see what happens come this Thursday, by which time the W-2 statements need to be mailed out.

Photo: Eric Gay/AP Photo

This Week in Cybercrime: Student Expelled After Revealing Security Hole in College Computer System


Student Whistleblower Expelled

It was revealed this week that a computer science student in Canada was expelled in November after he discovered a security flaw in his college’s computer system that could have exposed the personal data of more than 250 000 students. Hamed Al-Khabaz and a classmate found the security hole—which would have let anyone querying the system to access every bit of personal information about students contained in the school’s records—while developing an app that would let students access their campus accounts from mobile devices. When Al-Khabaz and his partner reported the problem, Dawson College administrators and officials at Skytech Communications, the company that sold the computer system to the school, initially gave the students a pat on the head for a job well done. But when Al-Khabaz followed up two days later, using a scanning tool to see if the campus and corporate security teams had made good on their promise to fix the vulnerability in Skytech’s Omnivox system, the pat on the head quickly changed to a swift kick in the pants.

Al-Khabaz says that he received a threatening call from Edouard Taza, the president of Skytech, telling him that the scan was illegal and could get him tossed in jail for up to a year. With that threat in the air, Al-Khabaz signed a non-disclosure agreement making him legally bound to keep silent about the security problem, the subsequent scan, the threatening conversation, and the existence of the non-disclosure agreement. Immediately following that episode, Dawson College officials applied their own dose of shoe leather. The school brought him up on charges of “serious professional conduct,” and 14 of 15 computer science professors voted to expel him from the computer science program. Heaped on top of that was the order that he repay grants he received for his studies.

In its defense, the school insists that the press has it all wrong. At a press conference on 22 January—after Al-Khabaz realized that he had very little left to lose by failing to abide by the terms of the non-disclosure agreement and went public with the details of the incident—school officials said the former student had “made an attempt to gain access to a range of systems” and that his activity constituted “a concerted set of attacks on a range of systems.”

An odd twist in the story is that although Dawson College refuses to readmit him, Skytech is one of a number of firms that have offered him a job.

The Downside of Logging Into Everything With One Password

Once again security has been sacrificed on the altar of ease of use. Twitter and Facebook, in an effort to put themselves at the center of Internet users’ online activity, allow their login credentials to be used as a kind of master key for granting access to third-party apps. And right on schedule, the unintended consequences have arrived.

Some apps, designed to automatically read from and write to a Twitter user’s timeline, see who he or she follows, and update the person’s profile, are supposed to do so only if given permission. But according to Cesar Cerrudo, a security researcher at IOActive, he recently discovered a flaw in Twitter’s code that let these third-party apps access Twitter users’ direct messages—which are supposed to be private—even when Twitter users had not agreed to give the apps that level of access.

In the course of testing the functionality of an app—specifically the feature that allows user to sign in with their Twitter credentials—he noticed that the permission level was initially set to allow the user enough access to read existing tweets and post new ones. But after logging out and signing back in a few times, the app began displaying Twitter direct messages. Meanwhile, the application settings page still indicated that the permission level had not been changed.

After unsuccessfully attempting to figure out the nature of the security flaw, Cerrudo notified Twitter’s security team, which promptly fixed the problem. Unfortunately, Cerrudo told Kaspersky Lab’s Threatpost, Twitter did not issue a general alert to its users making them aware of the issue.

U.S. Military Seeks Automated Cyberattack Defense

The U.S. Department of Defense's Advanced Research Project Agency (Darpa), is on the hunt for new ways to scan and analyze the massive amounts of data generated by the computer networks run by government departments. The effort, part of Darpa’s Cyber Targeted-Attack Analyzer program, is designed to "automatically correlate all of a network’s disparate data sources—even those that are as large and complex as those within the DoD — to understand how information is connected as the network grows, shifts and changes," says an agency news release. Keeping an eye on every bit of a network as extensive and complex as that run by the Department of Defense is a tremendous undertaking. The security and performance-monitoring systems attached to the networks collect untold haystacks of data on a daily basis. Darpa is hoping that employing a new, automated approach will make ferreting out the occasional needle easier. “The Cyber Targeted-Attack Analyzer program relies on a new approach to security, seeking to quickly understand the interconnections of the systems within a network without a human having to direct it,” Richard Guidorizzi, manager of the program told Kaspersky Lab’s Threatpost. “Cyber defenders should then be capable of more quickly discovering attacks hidden in normal activities,” he said.

The program comes on the heels of the U.S. military issuing several solicitations for offensive cyberwarfare capabilities.

Google Back as Sponsor of Hack-a-lympics

The Pwn2Own hacking contest is back—this year with new rules and a bigger cache of prize money courtesy of Google. HP TippingPoint, organizer of the annual event, says the hacker games—which will take place between 6 and 8 March at the CanSecWest security conference in Vancouver, British Columbia—will test entrants’ ability to demonstrate new exploits taking advantage of vulnerabilities in the Chrome, Firefox, Internet Explorer or Safari browsers, or the Adobe Reader, Adobe Flash or Oracle Java browser plug-ins. Big money (US $100 000) will go to the person who hacks Chrome on Windows 7 or Internet Explorer 10 on Windows 8 in the fastest time. The quickest to break into IE9 will get $75 000; the prizes go down from there, to $20 000 for an exploit for Java, which has taken a public beating for its security failings.

Google’s sponsorship is worthy of note, says Computer World, because the search company backed out underwriting the event last year over a disagreement with regard to the rules. Unlike last year, Pwn2Own participants must reveal the full exploits and all the details of the vulnerability used in their attacks. Google was upset that the contests wouldn’t result in vendors having the ability to see and fix the flaws. But it didn’t simply take its ball and go home. It put on a $120 000 Chrome-specific hacker contest at CanSecWest. Google has already confirmed that it will present Pwnium again this year. But the search firm has yet to reveal whether it will take place at CanSecWest.

Waiting For REAL ID? Take a Seat, It'll Be a While

There's an interesting story in next month’s National Defense magazine on the long gestation of the REAL ID Act.

As you may remember, eight years ago the U.S. Congress passed the REAL ID Act of 2005. It would have forced states to start issuing tamper-proof driver licenses and identify cards by 11 May 2008. The reason for the act, a brainchild of Congressman Jim Sensenbrenner of Wisconsin, was to make it harder for terrorists and other criminals to be able to pass off fake IDs in the commission of their crimes. And a REAL ID card would be required to enter a federal building or board a commercial airline flight.

After an outcry from state governors over the projected cost—upwards to US $12 billion they claimed—and from privacy advocates over this creation of a de facto national identity card, the Department of Homeland Security (DHS) decided in March 2007 to move the act's compliance date to December 2009. Then, in January 2008, DHS decided again to postpone the deadline for states to the 11 May 2011 and also changed some of the documentation requirements needed to get a REAL ID in hopes of quieting the critics. DHS estimated then that the states’ implementation costs would not be any greater than $3.9 billion, which DHS would help cover with $280 million in state grants.

After continued grumbling by the states about the cost, and some two dozen state legislatures passing laws or resolutions refusing to comply with the REAL ID requirements, in March 2011, DHS postponed the compliance deadline yet again, this time to 15 January 2013. And then, as this deadline approached and with most states still in non-compliance, late last month DHS for the fourth time delayed the compliance deadline. It will apparently be to sometime in 2015; the department won't announce the exact date until later this year.

A DHS press release announcing this latest delay praised the 13 states that it says have met REAL ID standards: Colorado, Connecticut, Delaware, Georgia, Iowa, Indiana, Maryland, Ohio, South Dakota, Tennessee, West Virginia, Wisconsin, and Wyoming. However, as the National Defense magazine article points out, the Real ID act requires that there exist “five different national databases for states to tap into to verify identities [but those] are not up and running.”

Hmm, I guess meeting the REAL ID standard all depends what you mean by “standard.” Or maybe the word "is."

In addition, as noted in an acerbic post at the CATO Institute, there are pretty good odds that the states who haven’t complied with the REAL ID act will probably never have to, making suckers out of those that did. As CATO points out, it is highly unlikely that the federal government is going to tell the citizens of 37 states they can’t fly in planes or enter federal buildings. How will federal judges feel about all those empty jury boxes? As it happens, I've been called to federal court jury duty next week, in a state that doesn’t meet the REAL ID act.

The pleasure of watching the endless tug of war over federal (unfunded) mandates versus states’ rights exposed by the REAL ID act is compounded by the risible and ever-changing cost estimates to the states of implementing it. Sensenbrenner originally estimated (i.e., pulled out of the air if not another place) that the cost to change state department of motor vehicle computer systems would be about $2 million per state over 5 years, or $100 million overall. The Congressional Budget Office, sharing the same fantasy, generally concurred, estimating that it would be closer to $120 million over the 5 years total.

However, a  2006 study by the  National Conference of State Legislatures (NCSL), the National Governors Association (NGA), and the American Association of Motor Vehicle Administrators (AAMVA) said that Sensenbrenner and the CBO were way off, and did not account for the vast majority of costs that would be incurred. This group estimated that the REAL ID act could cost states more than $11 billion over five years (pdf).

That number was thought to be way off until the DHS admitted in March 2007 that its own estimates of the REAL ID act implementation costs would be from $10.7 billion to $14.6 billion—with another $7.8 billion or so in costs borne by individuals in fees—over ten years.

After its January 2008 changes to the REAL ID requirements, DHS revised its own estimate and claimed that compliance would now cost the states a mere $3.9 billion or so over ten years.  In 2011, the Center for Immigration Studies, an advocate for REAL ID, estimated the cost  to the states would be even less: somewhere  between $350 million and $750 million. That seems remarkably low, given that DHS has said that it has already awarded $263 million in grants from FY 2008 to FY 2011 to states to help them meet the REAL ID requirements—and three-fourths of the states aren't done yet.

Exactly how much money the states have spent so far on top of this DHS grant amount is unknown, but even the DHS knows that meeting the Real ID “standards” aren't cheap. One reason for the latest delay, DHS says, is that “in a period of declining state revenues,” the states are having a hard time finding the money to implement the act's requirements.

My guess is that the DHS will continue to set Real ID compliance deadlines only to postpone them at the last moment, and hope that over time, the vast majority of states will ultimately albeit grudgingly implement REAL ID as they eventually replace their DMV legacy systems. Me, I'm rooting for some genuine enforcement of compliance by 2015. I probably wouldn't ever have to report for federal jury duty again.


Risk Factor

IEEE Spectrum's risk analysis blog, featuring daily news, updates and analysis on computing and IT projects, software and systems failures, successes and innovations, security threats, and more.

Willie D. Jones
Load More