Duncan Watts: From Sociology to Social Network
Everything changed for an Ivy League professor when he reinvestigated the “six degrees of separation”
Steven Cherry: Hi, this is Steven Cherry for IEEE Spectrum’s “Techwise Conversations.” Today we continue our conversation with Duncan Watts. He’s the author of two books: Everything Is Obvious Once You Know the Answer, published last year by Crown Business, and the 2003 book Six Degrees: The Science of a Connected Age. That book was a rather prescient look at how social networks are formed and maintain themselves. Just to locate it chronologically, Myspace had just been founded in August of that year, and Mark Zuckerberg had released something called Facemash but had yet to start coding Facebook. At the time, my guest was a sociology professor at Columbia University here in New York. In 2007 he moved to Yahoo Labs, also here in New York, where he worked until May of this year, when Microsoft hired a number of researchers from Yahoo to staff up its new New York research center.
Duncan, welcome back to the podcast.
Duncan Watts: Thanks, Steven.
Steven Cherry: Duncan, a lot of our ideas about the spread of ideas come from Stanley Milgram’s famous six degrees of separation study in 1967. I gather no one had tried to reproduce his results until you did, more than 30 years later. You found that hubs aren’t really very important. Maybe you can tell us what are hubs and why they’re not important. And maybe first remind us of Milgram’s study.
Duncan Watts: Okay. So this was a really sort of seminal study. Stanley Milgram was, as many people know, a psychologist. And he was famous for having done some experiments on obedience to authority back when he was at Yale. And these were very controversial, the ones where people thought they were giving electric shocks to some subject, who was really an actor. And the finding there was that people, when they thought they were being told to do something by a white-coated experimenter, would raise the shocks to what they thought were even lethal levels. So this was both a very controversial study and also really revealed Milgram’s genius at designing experiments.
So a few years later he’d moved to Harvard, and he heard about this interesting idea that everybody in the world is connected to everybody else by just six degrees of separation. And he decided to test it with an experiment. So he devised this protocol called the small-world protocol, where he took a bunch of people, about 300 people in Boston itself and in Omaha, Nebraska, and he picked another single person, what he called the target, who was a stockbroker, an acquaintance of his who was a stockbroker who lived just outside of Boston.
And the 300 senders were given packets by Milgram, and the packets were supposed to reach the target person. And in these packets they had all sorts of information about his full name, his address, his occupation, his years in military service and so on, plenty of information to identify him uniquely. But the catch was they could only send it to him if they knew him on a first-name basis. And that being very unlikely, they were to send it instead to someone who they did know on a first-name basis who they thought was closer to the target than they were.
And of the 300 of these packets that began, about 64 of them made their way across America and to the target person. And the average length of these chains that completed was six. And so this is where we believe the number “six degrees of separation” came from.
And it was very surprising to people at the time. Milgram interviewed a bunch of people and—and asked them—described the experiment and asked them how long they thought the chains would be. And they would sort of guess that these chains would be maybe hundreds of steps long. So the fact that they were only six steps long was really shocking to many people.
Now, of course, in the age of Facebook, we’re sort of used to these numbers, and we’re used to the idea of the world being small. But, you know, more than almost 40 years ago, this was very surprising.
So in fact, a few people did try to replicate these experiments in the years following Milgram. But it’s surprisingly difficult to get these experiments to work, mostly because it’s hard to get people to comply with the instructions. And so most of these chains fail before they get to the target. And, you know, this had led people to criticize the result and say, well, you know, only, you know, 20 percent of Milgram’s chains actually completed, and so what about all these other ones? Maybe it’s just true that some chains are short, but most of them are actually really long, and that’s really what the experiment is showing.
So about 30 years later, back in 2001, my colleagues at Columbia and I decided to try to replicate the experiment, but this time using the technology of the day. We wanted to use e-mail and the Web to try to redo Milgram’s experiment but on a much, much larger scale. You know, Milgram, of course, only looked at one target and only had people from two different cities trying to reach that target.
So we managed to recruit about 18 different targets from about 15 countries around the world. And we were able to recruit tens of thousands of other people to try to reach these targets. And so in the course of this experiment, which ran for several months, these chains passed through about 60 000 people in over 160 countries around the world. And so, this was in some respects, a truly global experiment, and one of the first experiments to really exploit the—or first social science experiments to exploit the global scale of the Web.
And what was supersurprising to us about these results is that the findings were very, very consistent with Milgram’s—except for one point. And this is the point about the hubs. And so it sort of seemed to Milgram—one of these individuals in particular—I think about 16 chains completed through him. So, you know, a quarter of the total came through this one guy, who was a local tailor and sort of the equivalent of a dry cleaner these days or a Laundromat owner. And so this person was very connected to the neighborhood and he knew lots of people, and so people would bring their packets in to him in the hopes that he would know the stockbroker, which he did.
So Milgram called this individual a sociometric star. And his hypothesis was that somehow these stars were necessary or helpful in getting the chains to complete. And so this idea has been picked up and then developed over the years into really sort of a whole theory of hubs, of these highly connected individuals who, like hubs in an airline network, connect everybody else.
So right before we did our experiment, about a year or two beforehand, Malcolm Gladwell had published his famous book The Tipping Point, in which he speculates that people like this, these highly connected individuals, are responsible for many of the interesting properties of social networks.
So we really went looking for these individuals in our data, and not to our surprise but to other people’s surprise, we didn’t really find them. And there’s a good reason for this—that, you know, people are not actually like airports. You know, the airline network is something that we have constructed to be a very efficient way of moving people around. And one of the choices, the design choices that’s been made is you create these mega hubs, places like Chicago O’Hare that have millions and millions of people traveling in and out of them every single day and have thousands of flights. And these airports, these hubs, are not just bigger than an average airport; they’re hundreds of times bigger than an average airport.
Now, when you look at individuals, you look at people in social networks, it’s true that some people are more popular than others. It’s true that some people have more friends and are more gregarious than others. And we’re tempted to call these people hubs. But if you actually look at the numbers, they’re only maybe a few times more popular or a few times more connected than an average person. So even on Facebook today, you know, the most connected people have a few thousand friends and a typical person has a few hundred friends. So there’s a difference maybe there of a single order of magnitude, not several orders of magnitude.
And so this is really important, actually, in terms of the properties of networks. And so it was actually not surprising that there are no individuals in the world who are so connected that everything can pass through them. And this is really what we were able to demonstrate.
Steven Cherry: So, stepping back for a minute, you’ve had an interesting career to date. You have a Ph.D. in theoretical and applied mechanics at one Ivy League school but went on to teach sociology at another. How did you make the switch to sociology?
Duncan Watts: Well, that is an interesting story, at least to me. I was studying chaos theory and nonlinear dynamics at Cornell University, working with this mathematician Steven Strogatz. And we were interested in a problem to do with the synchronization of biological oscillators, by which we meant crickets, little chirping animals who, in this particular case, chirp in unison. And we were interested in this question of how these crickets with very limited cognitive abilities could figure out how to chirp perfectly together.
So there’s this whole literature about—theoretical literature on synchronization and oscillators. And what we got interested in was this idea that somehow there was a network involved, that there were—these crickets were listening to each other, and there was some sort of topology to this network that we didn’t understand. And we wanted to see if we could characterize this topology in a way that helped us understand this other problem of synchronization.
So along the way, as often happens in scientific discoveries, this problem sort of got connected in my head with this other problem that I had heard about, this almost sort of folkloric statement about the small world and how people were connected to each other through just six degrees of separation. And I started to wonder whether these two problems were connected—that was it really true of the social world that everyone could be connected in this manner. And if it was true of the social world, was it also true of the world of biological oscillators and could this have something to do with the way that things are synchronizing.
So I sort of started down this path of, you know, mathematical models of networks and the small-world problem and that then expanded into questions that were related to this, such as the flow of information through networks or the spread of a disease or the way that people cooperate. All of these problems have to do with networks and potentially with network structures.
So over the years I became more interested in the applications of this way of thinking—which these days we call network science—to the world of sociology. And because, as it turned out, sociologists had been thinking about many of these questions for many years and there was a field in sociology called social network analysis—which has a lot of similarities to this modern version of network science—the sociologists were also interested.
And so I, you know, in a sort of classic networking experiment on myself, I managed to meet a couple of sociologists at Columbia, in particular this one very distinguished scientist, Harrison White, who 50 years earlier had gone from being a physicist to a sociologist. And I think he felt that, you know, this was an opportunity for him to help somebody else make that transition.
Steven Cherry: So briefly, what was the answer to the cricket question?
Duncan Watts: We never answered the question. That’s the sad part of the story, is that I got so distracted by this sort of side issue, what I thought was a side issue, you know, I really sort of went into this thinking I was going to look this up; it’ll be somewhere in a journal. And I realized that no one had really thought about this problem of the relationship between the topology of a network and the synchronization of oscillators.
Steven Cherry: It is interesting. So you’ve managed to work for two of the biggest West Coast companies around without ever leaving New York. First, why don’t you tell us about leaving academia for Yahoo Labs.
Duncan Watts: Well, interestingly, I never have really thought about it as leaving academia. I, you know, I left a university to go and work for a research lab, but in some ways the environment that we have here, both at Microsoft and also at Yahoo Research, allowed for more creativity and more deep thinking and research than is typically possible at a university.
Steven Cherry: The other big West Coast company was Microsoft. Microsoft almost bought Yahoo a few years ago, and I was personally convinced that a big part of that was the opportunity of getting Yahoo Labs, which was doing some pretty cool work, as you mentioned. A few months ago, Microsoft swooped in and hired most of Yahoo Labs’ New York researchers. Why did you and the others make the switch?
Duncan Watts: Well, I suppose I should really just speak for myself. There were a number of things that I personally found very attractive about Microsoft. Of course I had heard about Microsoft Research. It’s a very well known and an institution that’s much larger than Yahoo Research was. It’s something that is an institution that has a very long history of supporting both basic and applied research. And that’s a combination that is attractive to me and I think probably to many of my colleagues as well, because we like to think about basic science questions, but it’s also interesting to try to think about how that can be applied to real-world problems. So those opportunities all exist here at Microsoft.
I think the opportunity to build something new in New York was also very appealing, certainly to me, that this new outpost of Microsoft Research is, I think, like all the MSR labs, has its own sort of character and operates in a somewhat independent manner. And we really have this focus of—of bringing together the social sciences, psychology, economics, sociology, with computer science and trying to leverage the power of the Web in doing so. And that’s something that’s—that’s very timely right now. I also think that there’s a lot that’s going on in New York right now that we can sort of plug into. We have some great universities here. We have Columbia and NYU in town. Cornell is setting up a new—a technical engineering campus with Technion Institute. Rutgers, Princeton, Penn, Yale, they’re all within, you know, an hour train ride.
Steven Cherry: So you’ve only been there a few months, but what are you working on right now?
Duncan Watts: Well, we have a couple of different projects. One has to do with the structure of viral diffusion. You know, people talk about things going viral all the time these days, but we don’t really know very much about how that actually happens. And we—in particular, we have this sort of vague sense that a viral event, whether it’s a viral media or a meme or even a biological virus, somehow involves, you know, lots of people passing it on to lots of other people. But we haven’t until very recently been able to map out the structure of these events in a very sort of tangible way and then ask questions about what these structures look like.
And so that’s what we’re doing right now with data from Twitter, looking at all the news stories, all the videos, everything that has basically happened on Twitter in the last 12 months and millions and millions of events, and mapping out how they’re shared across the follower graph and defining new metrics that help us understand virality. So that’s one project that we’re really in the guts of right now. And we’re sort of figuring out new things every day.
The other one that we’re working on is totally different. And it has to do with the nature of cooperation, that people—this question of why do people cooperate. And we’ve investigated this in the past by running a series of experiments in our virtual labs that we create online. Maybe people will learn to trust each other. Maybe they’ll realize that, in fact, they should be more generous and people will become more generous over time. Or maybe nothing will happen at all. We don’t—you know, we didn’t really know. And so that’s the other question that we are—we are looking at, is examining this data in great detail and trying to figure out if there are any changes in strategy or in contribution levels over time.
So these are just a couple of examples that illustrate the sorts of things that we think about, things like diffusion and cooperation. It also illustrates that we work sometimes on observational data, stuff that we collect from the wild, and at other times run these very sort of controlled, scripted experiments in highly artificial environments. And so we’re really trying to use the Web in different ways to get at questions that in many cases are very old questions of social science.
Steven Cherry: Very good. Well, those are two great problems and I wish you luck with them. And thanks for taking the time with us today.
Duncan Watts: Thank you, Steven.
This is the second of a two-part interview with Microsoft researcher Duncan Watts, author of the book Everything Is Obvious Once You Know The Answer.
For IEEE Spectrum’s “Techwise Conversations,” I’m Steven Cherry.
Announcer: “Techwise Conversations” is sponsored by National Instruments.
NOTE: Transcripts are created for the convenience of our readers and listeners and may not perfectly match their associated interviews and narratives. The authoritative record of IEEE Spectrum’s audio programming is the audio version.