Other than the rather entertaining kerfuffle involving Apple’s new iPhone OS and its initial (non)corrective update (along with the suspicious “bendy phone” accusations), the IT Hiccups front was rather quiet this past week. Luckily, an “old friend” came by to rescue us from writing a post on some rather mundane IT snarl, snag or snafu.
Just in the nick of time, the U.S. Department of Justice's Inspector General released his latest in an ongoing series of reports [pdf] about Sentinel, the FBI’s electronic information and case management system. In this report, the IG focused on how Sentinel users felt about working with the system. Sadly yet unsurprisingly, the IG found that Sentinel is still suffering from some serious operational deficiencies two years after it went live.
Sentinel, you may remember, was finally deployed in 2012 as a replacement for the FBI’s legacy Automated Case Support (ACS) system, which was originally supposed to be superseded by the infamous US $170-million Virtual Case File system in late 2003 or early 2004. The VCF was itself a component of a larger FBI effort called Trilogy that was begun in 2001 to upgrade and modernize the FBI's IT systems. To best understand how and why VCF became infamous, you owe it to yourself to read “Who Killed the Virtual Case File?” Written by IEEE Spectrum's digital editorial director Harry Goldstein, it is a classic government IT project failure story that appeared in the September 2005 issue of Spectrum. For Agatha Christie murder mystery fans, Goldstein's story will immediately remind you ofMurder on the Orient Express.
Anyway, the origins of Sentinel—which the DOJ Inspector General succinctly describes as providing “records management, workflow management, evidence management, search and reporting capabilities, and information sharing with other law enforcement agencies and the intelligence community”—go back to 2005. The system has carried with it the FBI's hopes that it would permanently erase the unpleasant memories associated with the Virtual Case File debacle. At one time, those hopes looked like they might be fulfilled, as even the usually highly critical U.S. Government Accountability Office, which lambasted the FBI for its VCF risk mismanagement, called the original $425 million Sentinel acquisition a procurement model for the FBI.
A Rocky Start and a Surprise Recovery
Alas, those hopes faded as troubles began to plague the Sentinel project. Lockheed Martin, which won the prime contractor role in March 2006, was removed from its leadership position in 2010 after a devastating independent assessment by MITRE. The assessment indicated that the project, which at that point was into the taxpayers for $461 million and already past its original 2009 completion date, would take at least another $351 million and six more years to complete.
In the wake of the MITRE assessment, plus withering criticism of the project [pdf] by the IG and others, the FBI decided its best course of action was to take over control of the project itself. The FBI decided to take an agile programming approach to try to finish the project quickly and without blowing the budget. To everyone’s surprise, the Bureau successfully deployed Sentinel in July 2012 at a total cost of around $500 million.
I say around $500 million because even the IG doesn’t seem to have a really good handle on Sentinel’s true costs: IT development, operations and maintenance, plus FBI personnel and other internally-borne IT costs are not easily accounted for. For example, the FBI is still operating legacy IT systems that were supposed to be replaced by Sentinel, and those costs are not counted against the cost of Sentinel. The IG reports today’s obligated cost of Sentinel is $551.4 million, not counting the tens of millions of dollars in related FBI internal costs that were and are still being incurred.
The Sentinel project’s unexpectedly smooth roll-out won it many laurels and accolades (including its own). It was even named a ComputerWorld Honors Program 2013 Laureate. And since its 2012 roll-out, there has been nary a perturbing peep about Sentinel in the press.
It Glitters, But It's Not Gold
However, according to the inspector general's latest report, while “Sentinel has had an overall positive impact on the FBI’s operations, making the FBI better able to carry out its mission, and better able to share information,” there remain major problems with Sentinel’s two critical functions of searching for and indexing case information.
The IG report states that Sentinel’s search function is supposed to provide users the ability to locate cases and specific case-related information within Sentinel, while the indexing function's role is to designate and modify the relationship between any two identifiers, such as the relationship between a person and that person’s address. The proper indexing of Sentinel records is critical if FBI agents are to be able to “connect the dots,” the IG states.
For instance, the IG provides the following example:
“Indexing allows Sentinel to add structure to the data it contains, which in turn enables improved search results. [For example], a search for white males who drive black cars using a search engine like those used for internet searches would return all documents that mention any of the following: white males, black males, white cars, or black cars. By adding structure to the data through indexing, Sentinel’s search function is able to return only white males who drive a black car. When a user indexes an entity, the system will suggest potential matches already indexed in Sentinel.”
But the IG found in a survey of FBI agents that only 42 percent stated they “often received the results they needed” from Sentinel. Some 59 percent reported that they “sometimes, rarely, or never received the results they needed.” The IG also said that some survey respondents commented that the search function of the old ACS system was better than Sentinel’s! Furthermore, the IG stated that two issues kept frustrating the system’s users: “Sentinel returned too many search results for a person to reasonably review or no results at all for a document the user knew existed.”
In addition, the IG noted that soon before Sentinel was rolled out in July 2012 to all that acclaim, FBI management was boasting to the IG [pdf] that “the search function is both flexible and powerful enough to accommodate the substantial volume and wide variety of information available for retrieval.” However, according to information uncovered by the IG, at the same time FBI management was singing praises to Sentinel's search function the it knew there were major deficiencies with Sentinel’s search function.
The inspector general didn’t outright state it in his report, but reading between the lines you can see an IG clearly peeved that the FBI wasn’t honest in 2012 (or over the past two years, for that matter) about the true operational state of Sentinel's search capability and how it has hindered FBI personnel. The IG also cast some indirect doubt on how well the FBI’s highly touted agile approach worked: it may have helped save money and get Sentinel up and running quickly. But the question one has to ask now is, at what operational cost?
The IG also found that Sentinel’s indexing function wasn’t popular with users either. FBI Special Agents who now have to index their own case files (they used to hand the function off to an administrative staff member) complain that the process is a major administrative burden, and are frustrated that it “leaves less time for investigative activities." A full 41 percent of survey respondents “reported that they spent more time indexing in Sentinel than they did in ACS.”
I can’t be sure, but I'd bet that with Sentinel (and probably going back to VCF), FBI management wanted to reduce personnel costs by moving the task of indexing from administrative staff to Special Agents. They also probably reasoned that the move would improve the accuracy of the indexing. But as often happens in these cases, the “cost savings” usually turn out to be an illusion.
In fact, FBI management initially told the IG “that the FBI is not currently able to provide Special Agents in the field with assistance in reducing the time it takes to index large structured documents such as bank records, or unstructured documents such as a report of investigation form … or email.” In other words, there is no money to hire admins to do the case indexing grunge work. The IG responded that the FBI better find a technological solution and soon.
There are several other operational deficiencies listed in the IG report, which I won’t go over. Interestingly, some of these seem to be less of a design flaw and more of a combination of comfort with using technology and organizational memory of how things used to be. For instance, users with fewer than 10 years of FBI tenure, and especially those with fewer than 5 years, found it easier to use Sentinel’s search function than those with much longer tenures at the Bureau. That said, 25 percent of “tech savvy” users, still found the search function difficult to learn.
The IG made one telling statement in his report that should trouble all taxpayers in the United States: “Based on the feedback received from Sentinel users, we are concerned that Sentinel does not appear to have met users’ expectations and needs.”
FBI management admitted that there are indeed some issues with Sentinel, but also told the IG that fixes are on the way. In October, the FBI said, there is going to be a major Sentinel software release that will help address many of the concerns the IG raised in his report.
The problems with Sentinel’s search may take longer to resolve, however. FBI management promised the IG that it will begin soliciting user feedback in regard to how to improve the search function. The Bureau will then develop and implement solutions to increase search functionality and operational efficiency. The IG agreed to the FBI's proposed approach but indicated that he was now going to be taking a “trust, but verify” perspective to the statement's made by FBI management. The IG stated that in future he wanted from the FBI “a detailed description of the change[s] made and how the search function was improved as a result.”
As the old saying goes, fool me once, shame on you; fool me twice, shame on me. The IG has made it crystal clear to FBI management that he isn’t going to be fooled again.
In Other News...
Robert N. Charette is a Contributing Editor to IEEE Spectrum and an acknowledged international authority on information technology and systems risk management. A self-described “risk ecologist,” he is interested in the intersections of business, political, technological, and societal risks. Charette is an award-winning author of multiple books and numerous articles on the subjects of risk management, project and program management, innovation, and entrepreneurship. A Life Senior Member of the IEEE, Charette was a recipient of the IEEE Computer Society’s Golden Core Award in 2008.