Peer-to-peer networks are one of the last wild frontiers of the Internet. Anything that can be converted to bits, from the latest pop hit to political samizdat, can be shared between users. The decentralized designs common to public peer-to-peer networks make them resistant to legal or technological disruption, but they also create a weakness: it's hard to find anything but the most popular material, usually illegal music or barely legal pornography. Now, Los Angeles-based Streamcast Networks Inc. hopes to fix that weakness and to improve peer-to-peer's image into the bargain with a new search technology known as NEOnet.

If NEOnet succeeds in allowing users to find rarer content reliably on a peer-to-peer network—in the way that people use search engines today to find content on the Web—then it would be a big boost for peer-to-peer networks to become another channel, alongside the Web and FTP, for the legal distribution of things like software, multimedia products, or documentation.

NEOnet is integrated into Streamcast's signature peer-to-peer software, Morpheus. "If there's one file on the network of computers [that are] running Morpheus, NEOnet will find that file," should a user search for it, claims Ben Wilken, NEOnet's chief architect. "Other search technologies are limited in that you're searching a certain number of computers on a network, but not all."

Neonet Works by employing what are known as distributed hash tables. In short, a hash is like an entry in an alphabetized index, and a hash table is the index itself. Instead of searching a book page by page for a key word, a reader can quickly sort through the index to find a key word and a list of matching page numbers. With a distributed hash table, the page numbers correspond to locations of files, and the index is automatically broken up over computers spread across the network.

Each piece of the index also maintains information about where the other parts of the index are and which computers have which portion of the index. "All the network nodes organize themselves into groups, and each of these groups takes on a responsibility for part of the distributed hash table," explains NEOnet developer Francis Crick, who helped create the software with Wilken and Gitika Srivastava while still at Harvard University, in Cambridge, Mass. [see photo, "Sharing Files"].

Not everyone is impressed with the new technology. Greg Bildson, chief technology and operations officer of peer-to-peer software maker LimeWire LLC, headquartered in New York City, damns his rival with faint praise: "NEOnet theoretically does have advantages for rare content, but it entails a tremendous burden on the network for that rare search."

Bildson's main objection is that computers join the network only for relatively short periods of time, 40 minutes on average. In this situation, keeping the distributed hash tables up to date involves a level of network overhead that Bildson believes isn't worth the effort. He estimates that the majority of current queries already search enough computers to return useful results.

Crick counters, however, that "many people who give that argument don't understand the possible efficiency benefits that come out of using distributed hash tables." Unlike earlier attempts, he says, "NEOnet provides better results for all searches using less traffic" than the existing peer-to-peer search technology.

Bildson's opinion matters because—along with several other ventures—LimeWire and Streamcast share users to a degree. They are all using Gnutella, an open-source protocol that provides basic peer-to-peer file-sharing services. Each vendor builds additional features, such as NEOnet, on top of the basic protocol, but tries to make sure its software stays on speaking terms with the other Gnutella programs. The situation is somewhat analogous to that of different e-mail software vendors who have to maintain interoperability.

But sometimes spats flare up. Before NEOnet's incorporation into the Morpheus software, LimeWire "took steps at the beginning of the year to cut [Streamcast] off the network," remembers Bildson, because computers running Morpheus "weren't responding to searches and were completely leeching off the network."

In the end, Streamcast and LimeWire worked together to resolve the technical issues involved, but Streamcast's CEO Michael Weiss is stung by suggestions that his company isn't contributing its fair share to the success of the Gnutella network. He points to Streamcast's successful legal defense of the network against the Recording Industry Association of America (RIAA), Washington, D.C. "If it wasn't for Morpheus and our litigation, most of these other companies would be sued out of existence," says Weiss.

To the RIAA's chagrin, in August the courts ruled that peer-to-peer software vendors can't be held liable when users of their software illegally swap copyrighted material. There's no liability, the courts said, because the decentralized nature of the networks means that vendors have no control over what users do and because the software can also be used to distribute material legally.

Legal distribution is the future of peer-to-peer, as Weiss sees it, and this vision is why NEOnet is so important to Streamcast's strategy. It seeks, through NEOnet's improved ability, to find files other than music or the pinup du jour, to get users to accept Streamcast's peer-to-peer network as a reliable place to find legitimate information, entertainment, or software, much as the Web is used today. But NEOnet also offers the advantage that a peer-to-peer network can help share the load (and the bandwidth costs) currently associated with hosting such content on the Web.

Weiss calls this ability to find rare content "Googlizing peer-to-peer," and to get the ball rolling, Streamcast is working with Creative Commons, a San Francisco-based nonprofit that encourages duplication-friendly copyright licenses. The companies plan to set up a mechanism that allows easy searches through Morpheus for material licensed for legal peer-to-peer trading.