The Internet isn’t a web. It’s a vast ocean of information and we’re just skimming the surface. That’s the big idea behind the Deep Web – the invisible data, from government files to video games, that even Google can’t find. Michael Bergman, the computer scientist who coined the term, estimates that the Web as we know it represents less than 1% of what’s available online. “The Deep Web is quite real and quite valuable,” says Jeff Jarvis, author of What Would Google Do?, “but highly challenging.” The problem is that search engines are basically dumb bots. They crawl between existing links but miss the rest. This includes sites that haven’t been indexed or published and those that require passwords or fall into an unreachable format, like PowerPoint or Microsoft Word. Given the empire Google built by organizing such a shallow pool, the race to plumb the depths is on. The National Science Foundation and the University of Utah has DeepPeep.org, and Amazon chief Jeff Bezos invested in Kosmix.com. “How deep is the Web?” We may find out soon.
David Kushner is the author of many books, including Masters of Doom, Jonny Magic & the Card Shark Kids, Levittown, The Bones of Marianna, and Alligator Candy. A contributing editor of Rolling Stone, he has written for publications including The New Yorker, Vanity Fair, Wired, and The New York Times Magazine.