How Supercomputing Can Survive Beyond Moore's Law

Today’s technology makes a 1-exaflop supercomputer capable of performing 1 million trillion floating-point operations per second almost inevitable. But pushing supercomputing beyond that point to 10 exaflops or more will require major changes in both computing technologies and computer architectures.

Planning for such challenges has been a major focus for Erik DeBenedictis, a computer engineer at the Advanced Device Technologies department at Sandia National Laboratories in Albuquerque, NM. He has worked with the IEEE Rebooting Computing initiative and International Technology Roadmap for Semiconductors to pave the way for the future of both computing and supercomputing.

DeBenedictis outlined several possible technology paths for supercomputing—the millivolt switch, 3-D integration, and specialized architecture—at the  session titled “Beyond Moore's Law” at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15) held from November 15–20 in Austin, Texas.

This interview has been condensed and edited.

IEEE Spectrum: The “Beyond Moore's Law” session covered three general technology areas that could aid development of exascale supercomputing: the millivolt switch, 3-D integration, and specialized architecture. Which of the three areas piqued the most interest among attendees?

Erik DeBenedictis: I think it was clear that the audience understood what the set issues were and realized that all three paths were desirable options, if they work out. In particular, we don’t know when the millivolt switch will work out. There were only a few people who said: “I’m only really interested in option one, two, or three.” The rest of them were largely supercomputer users who were looking for what’s in the future, and I think we convinced them to pay attention to all the options.

During the discussion session, we went around the room and had a show of hands to see who was interested in each of the technology approaches. And it sort of came out to be an even split. But a lot of people were interested in all of them.

IEEE Spectrum: Did you come away from the conference with any new food for thought about planning for the future of supercomputing?

Erik DeBenedictis: My thoughts after the conference session were focused on the fact that there was a bunch of people outside the door who couldn’t get in. We had a room that held probably 40 people, and around 80 people were interested. I remember the room was full and people were lined up outside the door trying to peer inside. I think the lesson is that this is an interesting area and there should be additional [supercomputing conference] tracks about ensuring the supply of more capable machines. 

IEEE Spectrum: It’s a bit surprising there weren’t many other sessions focused on the future of supercomputing.

Erik DeBenedictis: There was a panel discussion on future computing technologies such as quantum computing and neuromorphic computing across the hall. That session was also standing room only, although they didn’t have people in the hall. They probably had five times as many people in a room five times as big as ours, and they packed the room.

Now, I found that kind of surprising, because that panel discussion was fundamentally different. What we were talking about in our session were things that could result in supercomputing that these guys could use for their current applications, whereas neither neuromorphic computing nor quantum computing is a replacement for most supercomputing applications today.

IEEE Spectrum: What are your next steps in planning for the future of supercomputing?

Erik DeBenedictis: The next thing we have going on is the IEEE Rebooting Computing Summit 4 in Washington, D.C. from December 9–11. We managed to get some of the government agencies to attend in fairly strong numbers. But I’d say the spin we’re going to put on Rebooting Computing Summit 4 is not quite as focused on supercomputing.

The interest of the supercomputer community is a subset of the overall computing industry. In some ways, the pressures are going to be greater on supercomputing because they have driven up the usage and requirements for the logic portion—like floating point logic—really high over the years. There’s little slack and little room for improvement until they hit the physical limits. 

The interest of the government—for example the National Strategic Computing Initiative—has increased focus on data analytics and learning techniques related to neuromorphic computing. Since those areas received less attention in the past, there is perhaps more low-hanging fruit; there is more technology to be gained for less effort. So that’s where the IEEE Rebooting Computing initiative is focused. But we’ll continue to support the supercomputing guys. They have legitimate needs, but their upside potential is somewhat less; there is no low-hanging fruit left.

IEEE Spectrum: Will supercomputing will be able to simply reap the benefits of technology developments in the broader computing industry?

Erik DeBenedictis: The answer is really yes and no. Industry will spend billions developing chips for smartphones. It's not going to actually develop much for supercomputers, but of course a lot of the technology carries across. But it’s only going to come partly for free. 

IEEE Spectrum: Because the technology still needs to be adapted for supercomputing?

Erik DeBenedictis: The industry won’t actually have made the technology for supercomputer architecture, like floating-point format, and so that’ll be a remaining task.

IEEE Spectrum: Does there need to be more awareness of these technology trends among supercomputer users?

Erik DeBenedictis: It’s not just about making them aware. There is a major effort that needs to take place in planning ahead. The lifespan of software tends to be vastly longer than the lifespan of hardware. When we put in new hardware at Sandia [National Laboratories], a top machine has a lifespan of three to four years, and you plan three to four years in advance. So that’s a total life cycle of six to eight years. But the issue is the software also needs to be written with a much longer timeframe in mind.

We’re just a couple generations away from the end of scaling for the current generation supercomputing. If you can project what a supercomputer is going to look like in the second half of its lifespan for a piece of software, why not write that software for the computer it’s going to run on for most of its lifespan, because we can predict it? At least have the software engineers understand what the machine their software runs on is going to look like in 10 to 15 years.

IEEE Spectrum: So there hasn’t been such great communication between the hardware developers and the software developers?

Erik DeBenedictis: I’ll give you an example of this. When all of a sudden multicore [processors] appeared around 2003, that disrupted everything because everybody needed more parallelism than they were planning. The programmers weren’t told until it was too late. With zero notice, dual core came out at lower clock rate and sort of caught the software guys by surprise, because their code was no longer partitioned properly for twice the processor count at half the clock rate.

So the equivalent thing is happening now in that we’re seeing the processors, you know, the [Intel] Knight’s Landing [Xeon Phi processor] and the GPU assists and all that kind of stuff, with a lot of extra threads like GPU threads; they're slow but there's a lot of them. Nobody is really thinking about how to code for the next generation of those processors. Is anybody thinking about how we’ll have to code differently to accommodate the jump from a 1-exaflop supercomputer to 10 exaflops? There is not enough attention being paid to this issue.