Alas, another case of how not to acquire a government IT system, this time, in the Australian State of Queensland.
To bring you quickly up-to-speed: on the 24th of March, a new payroll system was rolled out for the Queensland Health service which according to its web site is "a network of 15 Health Service Districts and the Mater Hospitals, Queensland Health delivers a range of integrated services including hospital inpatient, outpatient and emergency services, community and mental health services, aged care services and public health and health promotion programs."
Queensland Health has approximately 78,000 employees on its payroll which are supposed to be paid on every second Wednesday.
Since the new payroll system went live, however, thousands of Queensland Health staff have had continuing payroll issues.
As of this week, the new payroll system still isn't working properly with many Queensland Health staff still not getting paid; some getting paid incorrectly; some employees who had quit a year ago (or died) still getting paid; some getting paid but with blank payroll slips to tell them what was taken out of their pay; and some staff even quitting over the problem
Queensland Health IT staff are so fed up with trying to fix the payroll problems, they are even threatening a stop-work action.
The new payroll system, called the Continuity Project, was a replacement for an old and creaky LATTICE payroll system installed in 1997 that was unreliable and no longer supported by its vendor. IBM was contracted on 1 January 2008 to develop a replacement payroll system at a cost of A$6.19 million (fixed price) and a go-live date of 1 July 2008.
IBM was actually contracted to CorpTech, which the Queensland Auditor-General report on the fiasco, says is a "specialised business unit of Treasury Department and subsequently Department of Public Works providing a whole-of-government role over the acquisition of information technology."
CorpTech acts in many ways like a US federally funded research and development center (FFRDC).
When the IBM developed payroll system went live in March, it was the seventh go-live date - the previous six having been missed.
The total cost of the project (including CorpTech's management oversight and Queensland Health involvement) is as of the end of March, A$101.96 million, including A$21 million for IBM's effort (not counting another A$3.3million that is owed to IBM).
What happened to IBM's fixed price? It when out the window when stakeholders couldn't agree on the payroll system requirements.
According the Auditor-General's report, bickering stakeholders (the report terms the bickering more nicely as "confusion over roles and responsibilities") was only a small part of the project's problem.
Even after spending A$900,000 on system and requirements analysis, IBM assumed that the new payroll system would be "a 'like for like' replacement, using the Department of Housing’s SAP system with very little customisation."
Or as the report states:
"The underlying assumption stated by IBM in the Statement of Scope was: ‘Our understanding is that there is a relatively small amount of functionality required as a minimum to make the interim solution functional for Queensland Health that it is relatively small in nature."
Unfortunately, the Department of Housing SAP payroll system supported only 1,300 employees and had only one pay award structure. Queensland Health 78,000 employees have 13 pay awards and multiple industrial agreements which provide for over 200 different allowances, and in excess of 24,000 different combinations of calculation groups and rules, the Auditor-General report says.
Oops. IBM kind of missed that wee difference in system scale and complexity.
Another issue was that as the system was being developed and tested, no one could quite agree with what a "defect" meant. For instance, the report states that:
"During user acceptance testing, a large number of defects were identified and there was frequent tension between the parties over whether the defects were actual defects or changes in business requirements, which ultimately led to further change requests and increased project costs."
In addition, defects were classified by defect severity, with Severity 1 defect being show stoppers; Severity 2 being major; Severity 3 being minor, and; Severity 4 defects being seen as cosmetic.
However, as the Auditor-General report notes:
"During the third iteration of user acceptance testing, the Project Board agreed at their 9 July 2009 meeting to revise the defect severity definitions. This decision resulted in a number of Severity 2 defects being downgraded to Severity 3 defects. The Project Board also agreed to change the exit criteria for the fourth iteration of user acceptance testing to identifying no Severity 1 defects and putting in place a comprehensive management plan (Solution and Defect Management Plan) for Severity 2, 3 and 4 defects. The Project Board was of the opinion that a number of Severity 2 defects had acceptable workarounds and would not result in incorrect pay calculations."
"These changes had a significant effect on allowing the project to pass the exit criteria for user acceptance testing. The outstanding Severity 2, 3 and 4 defects were to be resolved progressively post Go-Live with the first fixes to be implemented in production prior to the first pay run."
Furthermore, the Auditor-General report states, a full system and integration test was not performed before the new payroll system was rolled out: an executive decision was made to essentially make the Queensland Health employees beta testers, with any problems they discovered fixed on the fly. However, Queensland Health IT staff were not fully trained in how to make such payroll fixes quickly and effectively.
An apparent reason for foregoing a full system and integration test was that there was a worry that the risk of continuing with the LATTICE payroll system for much longer was higher than trying to fix any problems that came up after the new payroll system was rolled out.
The Auditor-General's report, along with these independent assessment reports to Queensland Health by KPMG make for interesting as well as sorry reading. There are many other reasons for the fiasco detailed in the reports along with a host of recommendations to avoid them in the future for those still curious.
The Queensland Government has now issued IBM a show-cause notice indicating its wish to terminate the contract as a means to withhold the A$3.3 million it still owes IBM. It may also seek damages for the 35,000 payroll defects uncovered so far.
However, IBM has stated that the payroll project, in its view, was an "outstanding success" and that it would vigorously defend its record.
"IBM has relentlessly and consistently delivered above and beyond the scope of the contract to assist Queensland Health identify and address concerns with its payroll process... Our commitment to supporting the Queensland Government in its mission to provide quality services to employees remains unchanged."
So let the lawsuits begin!
After reading the reports, I think you'll agree that IBM wasn't solely at fault here: CorpTech, Queensland Health and the Queensland Government deservedly share the blame as well.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.