DARPA’s Spectrum Collaboration Challenge demonstrates that autonomous radios can manage spectrum better than humans can
Illustration: Greg Mably
In the early 2000s, Bluetooth almost met an untimely end. The first Bluetooth devices struggled to avoid interfering with Wi-Fi routers, a higher-powered, more-established cohort on the radio spectrum, with which Bluetooth devices shared frequencies. Bluetooth engineers eventually modified their standard—and saved their wireless tech from early extinction—by developing frequency-hopping techniques for Bluetooth devices, which shifted operation to unoccupied bands upon detecting Wi-Fi signals.
Frequency hopping is just one way to avoid interference, a problem that has plagued radio since its beginning. Long ago, regulators learned to manage spectrum so that in the emerging wireless ecosystem, different radio users were allocated different frequencies for their exclusive use. While this practice avoids the challenges of detecting transmissions and shifting frequencies on the fly, it makes very inefficient use of spectrum, as portions lay fallow.
Today, demand is soaring for the finite resource of radio spectrum. Over the last several years, wireless data transmission has grown by roughly 50 percent per year, driven largely by people streaming videos and scrolling through social media on their smartphones. To meet this demand, we must allocate spectrum as efficiently as possible. Increasingly, that means that wireless technologies cannot have exclusive frequencies, but rather must share available spectrum. Frequency hopping, which Bluetooth uses, will be part of the solution, but to cope with the surging demand we are going to have to go far beyond it.
To tackle spectrum scarcity, I created the Spectrum Collaboration Challenge (SC2) at the U.S. Defense Advanced Research Projects Agency (DARPA), where I am a program manager. SC2 is a three-year open competition in which teams from around the world are rethinking the spectrum-management problem with a clean slate. Teams are designing new radios that use artificial intelligence (AI) to learn how to share spectrum with their competitors, with the ultimate goal of increasing overall data throughput. These teams are vying for nearly US $4 million in prizes to be awarded at the SC2 championship this coming October in Los Angeles. Thanks to two years of competition, we have witnessed, for the first time, autonomous radios collectively sharing wireless spectrum to transmit far more data than would be possible by assigning exclusive frequencies to each radio.
Before SC2, various DARPA projects had demonstrated that a handful of radios could autonomously manage spectrum by frequency hopping, as Bluetooth does, in order to avoid one another. So why can’t we just extend the use of the frequency-hopping technique to a wider array of radios, and solve the problem of limited spectrum that way?
Colossal Challenge: DARPA built the world’s largest radio-frequency emulation test-bed, called Colosseum, to pit teams against one another. Paul Tilghman (bottom, left) and Ben Hilburn present the results to a rapt audience during the second preliminary round.Photos: DARPA
Unfortunately, frequency hopping works only up to a point. It depends on the availability of unused spectrum, and if there are too many radios trying to send signals, there won’t be much, if any, unused spectrum available. To make SC2 work, we realized, we would need to test competing teams on scenarios with dozens of radios trying to share a spectrum band simultaneously. That way, we could ensure that each radio couldn’t have its own dedicated channel, because there wouldn’t be enough spectrum to go around.
With that in mind, we developed scenarios that would be played out in a series of round-robin matches, in which three, four, or five independent radio networks all broadcast together in a roughly one-square-kilometer area. The radio networks would be permitted access to the same frequencies, and each network would use an AI system to figure out how to share those frequencies with the other networks. We would determine how successful a given match was based on how many tasks, such as phone calls and video streams, were completed. A group of radio networks completing more tasks than another group would be crowned the winner for that match. However, our main goal was to see teams develop AI-managed radio networks that would be capable of completing more tasks collectively than would be possible if each radio was using an exclusive frequency band.
We realized quickly that placing these radios in the real world would have been impractical. We would never be able to guarantee that the wireless conditions would be the same for each team that competed. Also, moving individual radios around to set up each scenario and each match would have been far too complicated and time consuming.
So we built Colosseum, the world’s largest radio-frequency emulation test-bed. Currently housed in Laurel, Md., at the Johns Hopkins University Applied Physics Laboratory, Colosseum occupies 21 server racks, consumes 65 kilowatts, and requires roughly the same amount of cooling as 10 large homes. It can emulate more than 65,000 unique interactions, such as text messages or video streams, between 128 radios at once. There are 64 field-programmable gate arrays that handle the emulation by together performing more than 150 trillion floating-point operations (teraflops).
For each match, we plug in radios so that they can “broadcast” radio-frequency signals straight into Colosseum. This test-bed has enough computing power to calculate how those signals will behave, according to a detailed mathematical model of a given environment. For example, within Colosseum are emulated walls, off which signals “bounce.” There are emulated rainstorms and ponds, within which signals are partly “absorbed.”
Working It Out: In one scenario, Slice of Life, set in an outdoor mall, each team’s radio network emulates a different store’s Wi-Fi hot spot. Early in the day, the red team, perhaps acting as the Wi-Fi for a coffee shop, requires the most bandwidth (top). Later, the green team needs more, and starts overlapping with red while jostling for spectrum (middle). Eventually, the three teams settle on a new arrangement as green experiences the heaviest traffic (bottom).Image: DARPA
The emulation provides all the information necessary for the teams’ AIs to make appropriate decisions based on their observations during each emulated scenario. Faced with a cellphone jammer that is flooding a frequency with meaningless noise, for example, an AI might choose to change its frequency to one not affected by the jammer.
It’s one thing to build an environment for AIs to collaboratively manage spectrum, but it’s another thing entirely to create those AIs. To understand how the teams competing in SC2 are building these AI systems, you need a bit of background on how AI has developed in the past several decades.
Broadly speaking, researchers have advanced AI in a couple of “waves” that have redefined how these systems learn. The first wave of AI was expert systems. These AIs are created by interviewing experts in a particular area and deriving a set of rules from them that an autonomous system can use to make decisions while trying to accomplish something. These AIs excel at problems, such as chess, where the rules can be written down in a straightforward fashion. In fact, one of the best-known examples of first-wave AI is IBM’s Deep Blue, which first beat chess master Garry Kasparov in 1997.
There’s a newer, second wave of AI that relies on huge amounts of data, rather than human expertise, to learn the rules of a given task. Second-wave AI is particularly good at problems where humans have trouble writing down all the nuances of a problem and where there often seem to be more exceptions than rules. Recognizing speech is an example of such a problem. These systems ingest complex raw data, such as audio signals, and then make decisions about the data, such as what words were spoken. This wave of AI is the type we find in the speech recognition used by digital assistants like Siri and Alexa.
Today, neither first- nor second-wave AI is used for managing wireless spectrum. That meant that we could consider both waves of AI and the ways in which researchers teach those AIs how to solve problems, to find the best solution to our problem. Ultimately, it is easiest to treat spectrum management as a reinforcement-learning problem, in which we reward the AI when it succeeds and penalize it when it fails. For example, the AI may receive one point for successfully transmitting data, or lose one point for a transmission that was dropped. By accumulating points during a training period, the AI remembers successes and tries to repeat them, while also moving away from unsuccessful tactics.
In our competition, a dropped transmission often happens because of interference from another radio’s transmission. So we also have to think of wireless management as a collaborative challenge, because there are multiple radios broadcasting at the same time. The key to AI-managed radios performing better than traditional, static allocation is developing AIs that can maximize their own points while leaving room for the other AIs to do the same. Teams are rewarded when they make as many successful transmissions as possible without constantly bumping into one another in the pursuit of available spectrum, which would prevent them all from maximizing use of that spectrum.
Orderly Chaos: The AI-managed networks for five teams broadcast first in defined channels (left). Then those channels are removed and the AIs work out how to share spectrum (right). The result looks chaotic, but the networks succeed in completing almost as many connections as before.Image: DARPA
As if that’s not difficult enough, there’s one additional wrinkle that makes spectrum collaboration harder than many similar problems. Imagine playing a game of pickup basketball with people you’ve never met before: Your team’s ability to play together is not going to be anywhere near as good as that of teammates who have trained together for years. To date, the most successful challenges involving multiple agents have been ones where AIs have been trained together. A recent example was a project in 2018, in which the nonprofit AI research company OpenAI demonstrated that a group of five AIs could beat a team of human players in the video game Dota 2.
It’s 9 December 2018, and my DARPA colleagues and I are finally getting our chance to learn if a group of AIs can succeed at such a complex multiagent problem. We’re huddled around a set of computers in a hotel conference room, just a block away from where Colosseum is installed. The hotel has been our command center for a week now, and we’ve analyzed more than 300 matches to determine the top-scoring teams. In three days, we expect to award up to eight $750,000 awards, one for each of the top teams. But for the moment, we don’t actually know how many prizes we’re going to be handing out.
In the first qualifying event a year earlier, teams were judged solely on their relative rankings. This time, however, to win an award, the top teams must also demonstrate that their radios can manage spectrum better than by using traditional dedicated channels.
To compare autonomous radios against exclusive-frequency management, we designed one last set of matches. First, we took a baseline, in which each team was assigned exclusive frequencies, to measure how much data they could transmit. Then we removed the restrictions to see whether a team’s network could transmit more data without hampering the four other radio networks sharing the spectrum.
In the hotel room, we’re waiting anxiously for the last set of matches to complete. If no one is able to clear the bar that we’ve set for them, two years of hard work could be dashed. It occurs to us that in our zeal, we had no backup plan should everyone fail. And it didn’t necessarily soothe our nerves that by this point in SC2’s existence, we’ve started to see the limitations of some approaches.
Fortunately, we’ve also begun to spot some of the keys to success. When the competition began, nearly all of the teams started with first-wave AI approaches. This made sense as a starting point—remember that there are no AI systems being used to manage spectrum. In this first-wave approach, the teams are trying to write the general rules for collaboratively using spectrum.
Of course, each team has written slightly different rules, but every system they developed had some general principles in common. First, the systems should listen for what frequencies each network has asked to use. Second, from the remaining frequency bands, only one radio should be assigned to each band—and teams should be good neighbors by not claiming more than their fair share. And third, if there are no empty frequency bands, radios should select the ones with the least interference.
Unfortunately, these rules fail to catch all the idiosyncrasies of wireless management, which result in unintended consequences that hamper the radios’ ability to work together. During SC2, we’ve seen plenty of examples where these seemingly straightforward rules fail.
For example, remember the second rule, to be a good neighbor and not hog frequencies? In principle, this cooperative approach should provide opportunities for other radios to use more spectrum if they need it. In practice, we saw how this strategy goes awry: In one instance, three teams left a large amount of the spectrum completely unused.
Looking at the results, we realized that one team insisted on using no more than one-third of the spectrum. While this strategy was very altruistic, it also limited the connections they could make to complete their own tasks—and therefore limited their score as well. It got worse when another system noticed that the first system was not scoring enough points, so it limited its own spectrum use to allow the first system to use more, which it would never do. Basically, the systems were being too deferential, and the result was wasted spectrum.
Meet the Teams: Competitors gather after the second preliminary event for a group photo. The competition has attracted researchers from around the world to try for the nearly US $4 million available in prizes.Photo: DARPA
To fix that problem with a first-wave AI, the teams have to write another rule. And when that new rule results in another unexpected outcome, they deal with that by writing another rule. And so on. These constant surprises and consequent new rules are the main shortcoming of first-wave AIs. What can seem like a straightforward problem might end up being more difficult than it appeared to be.
Rather than rely on a few hard-and-fast rules, it seems that a better approach is for each radio to adapt its strategy based on the other radios with which it is sharing spectrum. In effect, the radio should develop an ever-growing series of rules by mining them from a large volume of data—the kind of data that Colosseum is good at generating. That’s why now, during this trial on 9 December 2018, we’re seeing teams shift to a second-wave AI approach. Several teams have built fledgling second-wave AI networks that can quickly characterize how the other networks are playing a match, and use this information to change their own radios’ rules on the fly.
When SC2 started, we suspected that many teams would take the simple approach of employing a “sense and avoid” strategy. This is what a Bluetooth device does when it discovers that the spectrum it wants is being used by a Wi-Fi router: It jumps to a new frequency. But Bluetooth’s frequency hopping works, in part, because Wi-Fi acts in a predictable way (that is, it broadcasts on a specific frequency and won’t change that behavior). However, in our competition, each team’s radios behave very differently and not at all predictably, making a sense-and-avoid strategy, well, senseless.
Instead, we’re seeing that a better approach is to predict what the spectrum will look like in the future. Then, a radio could use those predictions to decide which frequencies might open up—even if only for a moment or two, just enough to push through even a small amount of data. More precise predictions will allow collaborating radios to capitalize on every opportunity to transmit more data, without interfering by grabbing for the same frequency at the same time. Now our hope is that second-wave AIs can learn to predict the spectrum environment with enough precision to not let a single hertz go to waste.
Of course, all of this theoryis useless if the AI-managed systems can’t outperform traditional allocation. That’s why we’re delighted to see, that night in the hotel room with the results rolling out of Colosseum, that six of the top eight teams had succeeded! The teams demonstrated that their radios, when they collaborated to share spectrum, could collectively deliver more data than if they had used exclusive frequencies. Three weeks later, four additional teams would do the same, bringing the total to 10.
We were, of course, ecstatic. But encouraging though the results are, it’s too soon to say when we might see radios using AI actively to manage their use of radio spectrum. The important thing to understand about the DARPA grand challenges is that they’re not about the state of technology at the end of the competition. Rather, the challenges are designed to determine whether a fundamental shift is possible. Look at DARPA’s autonomous driving Grand Challenge in 2004: It took another decade for autonomous technology to start being used in a very limited way in commercial cars.
That said, the results from our initial tournaments are promising. So far, we’ve found that when three radio networks share the spectrum, their predictions are much better than when four or five teams try to share the same amount. But we’re not done yet, and our teams are currently building even better systems. Perhaps, on 23 October 2019 at SC2’s live championship event at Mobile World Congress Americas, in Los Angeles, those systems will demonstrate, more successfully than ever before, that AI-operated radios can work together to create a new era of wireless communications.
This article appears in the June 2019 print issue as “AI Will Rule the Airwaves.”
About the Author
Paul Tilghman is a program manager in the Microsystems Technology Office of the Defense Advanced Research Projects Agency, where he’s been overseeing DARPA’s Spectrum Collaboration Challenge.