Last Wednesday, corporate users (and some consumers) running Windows XP experienced problems caused by a faulty signature update file released by the security company McAfee.
As a result, McAfee said that many of those customers "reported a variety of symptoms, ranging from a system "blue screen" (not to be confused with BSOD, but due to the issues with Explorer and svchost.exe), loss of network connectivity, inability to use USB, and experiencing a perpetual state of reboot. Users have reported these symptoms when both the file is present on the system (in quarantine), or has been deleted entirely."
McAfee's President and Chief Executive Officer David DeWalt said in a company blog post Friday that, "We deeply regret the impact this may have had on you. In some cases, the outages were lengthy. Even among the vast majority of customers who did not experience operating disruptions, the mere possibility created an unwelcome distraction and reason for concern."
According to McAfee, the updated signature file, which created a false positive error situation, only affected " 'less than half of one per cent of business customers,' and a smaller number of consumer customers", a story in the London Telegraph reported. McAfee has refused to say exactly how many customer systems were affected, however, although some are guessing as high as 300,000. The company says that it helps secure 60 million business users and 75 million consumers.
The total number of customers affected may have been small from McAfee's perspective, but a quick search reveals major problems reported by:
- Coles supermarket in Australia, where 10% or about 1,100 of its cash registers were shut down;
- Rhode Island Hospital, Hasbro Children’s Hospital, Miriam Hospital, Newport Hospital and Bradley Hospital, who lost their computers. This caused some of them to refuse to admit new patients except for the most severe emergency cases.
- The University of Rhode Island where some 1,000 out of its 6,000 computers were reportedly knocked out;
- The University of Michigan's medical school which reported that 8,000 of its 25,000 computers crashed;
- The State of Kentucky, where many of its government offices, the police, hospitals and schools suffered problems ranging from minor to major.
A general roundup of those affected by McAfee's update issue can be found here.
McAfee said that the problem arose because the signature update file release "was designed to target the W32/Wecorl.a threat that attacks system executables and memory. The problem arose during the testing process for this solution. We had recently made a change to our QA environment. Unfortunately, this change resulted in a faulty DAT [update file] making its way out of our test environment."
Few things are more ironic in the IT community than trying to improve a QA process and having it backfire.
McAfee promises more QA protocols and some other measures to keep the issue from ever happening again. I think they promised the same thing after another false positive signature problem in 2008, though.
Anyway, today McAfee is promising its customers that,"If you have already incurred costs to repair your PC as a result of this issue, we're committed to reimbursing reasonable expenses."
This story in USAToday quotes Steve Shillingford, CEO of tech forensics firm Solera Networks, as saying that firms can expect to average 30 minutes of manual labor per affected PC to get it back into working order.
That's a lot of labor hours McAfee may be on the hook for.
Alas, consequential damages suffered seem to be out, however.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.