One person, properly motivated and equipped, can generate terabytes of personal data. While all that fine-grained information could prove invaluable, it also threatens to overwhelm the health care system. Medical anthropologist Theresa MacPhail of Stevens Institute of Technology spent several months during the H1N1 flu pandemic observing researchers with the Centers for Disease Control and Prevention (CDC) as they struggled to make sense of all that big data.
IEEE Spectrum: What did you see at the CDC on a typical day?
Theresa MacPhail: During the first wave of the pandemic, the analysts were getting 1,000 to 1,200 e-mails a day—spreadsheets, lab reports, surveillance alerts, direct answers to their own questions. They had multiple conference calls and at least two or three in-person meetings every day. It was intense.
As you recount in your bookThe Viral Network, your CDC sources told you “the data were crap.” What did they mean?
That the data was divorced from any context—information needed to interpret the data, like what lab did it come from, how were they running their assays. “Good” information came from sources they knew and trusted. There are some older CDC analysts I think of as “context repositories”—they can instantly provide the context for a new piece of information.
Is that something a machine can do?
The CDC, WHO [World Health Organization], and other institutions are trying to design better surveillance algorithms. When you’re identifying a new outbreak, it’s important to look at tweets, blogs, news reports—all that unstructured data that’s so hard to handle. These massive computer programs like Google Flu Trends are designed to scan for certain keywords and spit out reports. But a human still has to go through it.
I think we’re looking at big data wrong. The real usefulness of big data is to show gaps in our thinking. In the Medicare system, for instance, many people weren’t opting for drug coverage, even though it would save them money. So they redesigned the enrollment campaign to get more people signed up. Big data didn’t say why those people didn’t sign up, and it didn’t solve the problem, but it showed us we had a problem.
This article originally appeared in print as “Will Our Data Drown Us?.”