As I noted last week, US Federal Bureau of Investigation (FBI) Director Robert Mueller told the US House Appropriations Committee’s Subcommittee on Commerce, Justice, Science, and Related Agencies that the follow-on to the notoriously failed $170 million Virtual Case File program called Sentinel would once again slip its schedule and cost targets. The latest cost estimate is in excess of $481 million with a total program slip now of some two years and counting.
Director Mueller told the House Committee that the FBI decided to suspend work on Sentinel in early March to correct some "minor" technical issues and make some design changes. These issues included, according to the New York Times, "slow response times, awkward display pages and screen print that was too small."
I noted at the time that suspending Sentinel over these types of issues didn't sound quite right, and that there had to be something more to it. Well, this week the Department of Justice Office of the Inspector General released its latest report on Sentinel which sheds a lot more light on what these "minor" technical issues really involved.
One quick note, to put the following information in context. The Sentinel development project is divided into four phases, with the phases having different segments. On 2 December 2009, the FBI "conditionally' accepted from the prime contractor Lockheed Martin the release of Phase 2, Segment 4 of the Sentinel system, which is supposed to be the last segment of Phase 2.
This acceptance came even though the FBI testers of the system in November of 2009 found what appear to be glaring - not merely minor - deficiencies. The OIG report says,
"For example, one tester reported that it had taken over 4 minutes to attach a picture file to a Sentinel form and there was no way to stop that process once it had begun. Additionally, a Sentinel PMO official stated that users experienced waiting times of over 30 seconds each for Sentinel to render a blank form for completion, and for Sentinel to save a completed form. FBI employee testers stated that if Sentinel were released at that time, it would decrease user productivity, have a detrimental impact on investigations, and lose credibility as a valuable case management tool. Ninety-one percent of the testers 'did not succeed' when they tried to create one or more Electronic Communications, one of the documents most frequently used in the FBI to convey information between employees. Moreover, 82 percent of the testers stated that the Sentinel capabilities they tested would make the completion of the related tasks 'much harder' than current FBI practices."
The OIG report goes on:
"FBI officials told us that they conditionally accepted Segment 4 because they believed it met most of the FBI requirements. Additionally, the FBI’s conditional acceptance of Segment 4 prohibited Lockheed Martin from incurring any additional Segment 4 development costs without specific written permission from the FBI’s Contracting Officer. Consequently, Phase 2 moved into the operations and maintenance phase."
The OIG report states that the conditional acceptance thus allowed the FBI to use $780,000 of O&M money during the next two months to try to correct the "minor" problems found without having to show that cost on Sentinel's development budget.
The OIG also said that the FBI's original plan, after accepting Phase 2 Segment 4 of Sentinel, was to fully roll that release out to FBI offices by 16 December. However, the usability and response times were so poor that FBI officials decided the release was going to be piloted at two FBI field offices instead. Even the extra $780,000 spent to fix the "minor" technical issues could not make the release sufficiently usable, so those two pilots were put (and still are) on hold.
What is even more flabbergasting is that the FBI hired MITRE before the Phase 2 Release 4 planned acceptance date to delve into "Sentinel's software code and related documentation." The reason for the independent review was Sentinel's poor performance and usability that had been an on-going problem since at least April 2009.
The MITRE review team, the OIG report said, "... concluded that Lockheed Martin had significantly deviated from accepted systems engineering practices, did not follow its own published documentation requirements, and had not adequately followed testing procedures. According to the team’s report, these deficiencies resulted in over 10,000 inefficiencies [patterns in the software code that may result in decreased performance] in Sentinel’s software code. The team found that while none of the 10,000 inefficiencies dramatically affected Sentinel’s performance, collectively they could diminish Sentinel’s performance."
By the way, when did the MITRE independent review team deliver its report to the FBI? While not specifically saying so in the OIG report (the report only says that MITRE's findings were delivered in December), it appears that it was after the FBI had already "conditionally" accepted the Phase 2 Segment 4 release from Lockheed Martin.
Not a clever move by FBI's management.
You can dig into the latest OIG report yourself and find more such nonsensical FBI program management decision (and excuse) making, if you want. Be forewarned, however, if you are a US taxpayer and you suffer from high blood pressure, this report will only increase it several more points.
"Minor" technical issues, indeed. I would hate to see what the FBI Director thinks are major ones.
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.