If you pack too many copper wires together, eventually you’ll run out of space—if they don’t melt together first. AI data centers are encountering similar limitations with the electronic interconnects that shuttle data among GPUs and memory. Accommodating the data-guzzling demands of AI means the industry is turning to bigger chips with more processors, which means denser connectivity across longer distances within a rack. Startups are unveiling demonstrations of how GPUs can shed their copper interconnects, replacing them with optical links.
Optical links are no stranger to data centers. They traffic data between racks, using pluggable transceivers that convert electrical signals into optical ones. To improve energy efficiency, “integrating optics into chip packages has been a holy grail,” says Clint Schow, an electrical engineering professor at the University of California, Santa Barbara. That’s co-packaged optics, or CPO, and tech behemoths are throwing their weight behind it. Nvidia recently announced production of a networking switch using photonic modulators embedded onto the same substrate as the switch. “That rocked the industry,” says Bardia Pezeshki, CEO of Sunnyvale, Calif.–based startup Avicena.
Nvidia’s announcement is exciting because it brings photonics inside the rack, explains Keren Bergman, an electrical engineering professor at Columbia University and cofounder of Xscape Photonics. But Nvidia plans to bring photonics only to the network switch (for now!). Avicena and other startups are challenging the notion that optics doesn’t have the cost, reliability, and power efficiency to replace copper within a rack. They’re bringing optical interconnects directly onto GPU and memory packages. Even these final meter-long links need more bandwidth than copper can provide, Bergman says. The same basic technology “can go right next to the GPU,” bringing the optics closer to the data source and allowing them to carry bandwidth off the chip package itself.
Startups Innovate Optical Interconnects
Multiple startups have been working on replacing these copper interconnects next to the GPU. Several have landed on chiplets using waveguides called microring resonators to encode data lanes onto optical waves from an external laser and filter the appropriate wavelength at the receiver port. It’s the same basic photonic technology that Nvidia endorsed with its CPO switches—making it “a huge deal,” says Columbia’s Bergman. But instead of just one resonator, startups are using many.
Last month, Ayar Labs announced an optical interconnect between GPUs that incorporates the standard open-source UCIe electrical interface. “That’s an industry first,” says Vladimir Stojanovic, CTO and cofounder. The UCIe forms the on-package electrical link between the GPU and the TeraPhys optical chiplet. The chiplet transmits a copy of the digital signal into a single-mode optical fiber, extending communication over distances up to 2 kilometers. “One GPU talking to another GPU doesn’t even know it’s leaving the package,” says Stojanovic. Ayar Labs’ SuperNova light source feeds 16 wavelengths onto each fiber. A resonator performs wavelength division and multiplexing for each of 8 input and 8 output fiber ports, netting 256 data lanes totaling 8 terabits per second between GPUs. Using the UCIe protocol results in a fully modular design. “Any chipmaker can bolt this on and have an optical converter,” explains Schow.
Mountain View, Calif.–based startup Lightmatter announced a similar take on optical links between GPUs: the Passage L200. It uses chiplets from Alphawave Semi to implement the UCIe interface—but instead of building them alongside, it stacks them on top of the optical circuit using standard chip-on-wafer techniques. “Going 3D means you’re not trying to route electrical signals off the chip’s edge,” says Steve Klinger, Lightmatter vice president. The company demonstrated a fully integrated version in the Passage M1000, an optical interposer made of eight of these building blocks. Each segment sits beneath a GPU or memory tile—mounted directly on top via extremely short UCIe connections. Fibers route optical signals off the interposer.
“The L200 is the modular way to integrate optics, and the M1000 is more aggressive,” explains Schow. The latter solves the bandwidth problem locally with electronics, and the between-package problem optically. But, Stojanovic observes, “They have yet to demonstrate it.”
Bergman’s company, Santa Clara, Calif.–based Xscape Photonics, takes another step by eliminating external light sources, building frequency-comb lasers directly onto the chip. “We can copackage the laser and the link altogether,” says Bergman. The company received US $44 million funding last October to ramp up production of the ChromX platform--a multicolor platform that maximizes the “escape bandwidth” problem inherent to bringing high-bandwidth data off the chip.
Revolutionizing Data Transfer
Pradeep Sindhu, cofounder of Juniper Networks and Fungible, now at Microsoft, offers a skeptical view. In clusters where you want flexible point-to-point links between large numbers of GPUs, the granularity of each switchable data lane matters. One fat pipe between point pairs won’t suffice. Rather, you want many smaller pipes, and too much bandwidth per fiber reduces flexibility. For example, the lofty goal of connecting 512 GPUs, each linked by 200-gigabit-per-second lanes to 64 switches requires more than 30,000 connections. “If you jam 16 wavelengths down a single fiber how do you connect this many GPUs to this many switches?” Sindhu says. Fewer, more powerful switches that electronically parse the fat pipes is one answer—but that swaps redundancy for single-point failure. Moreover, multiwavelength lasers pose concerns about cost, energy efficiency, and reliability.
Another approach skirts both problems. Avicena uses hundreds of blue microLEDs connected through imaging fibers to move data. Pezeshki explains, “If you have a camera look at a TV, you have an optical connection with no lasers involved.” Avicena’s optical chiplet has a small microLED display and a tiny camera, with a mind-boggling frame rate, announced in the modular LightBundle platform based on a frame rate of 10 gigabits per second per lane. Each display of 300 microLEDs carries an accumulated 3 Tb/s, but with high granularity. Eliminating lasers reduces the reliability risk, cost, and complexity, and gains a fivefold energy improvement according to Pezeshki. Sindhu says, “I am optimistic that microLEDs are the leading technology.”
Vladimir Koslov, CEO of optical communications market-research company LightCounting, lauds the startups’ demonstrations. “Some will be successful,” he says. But the path to market is “not a sprint; it’s a marathon.” Moreover, copper still works. He argues that CPO will be limited to switches for years, because that’s where industry has aimed. “I don’t think we’ll see [CPO] on GPUs until early next decade,” he says.
Sindhu says that connecting enough GPUs in a cheap, low power, and reliable way is “the most important packaging problem of this era.” People will look back at whoever solves it. “The winner will take all,” he says.
Rachel Berkowitz is a freelance science writer and editor with a Ph.D. in geophysics from the University of Cambridge. She is a corresponding editor at the American Physical Society's Physics Magazine. Her work has appeared in Scientific American, New Scientist, Science News, Physics Today, and the newsrooms of several U.S. national laboratories.
