An artificial intelligence project recently funded by Silicon Valley pioneer Elon Musk aims to find a new way to compare supercomputers to the human brain. Instead of trying measure how quickly wetware or hardware can do calculations, the project measures how quickly the brain or a computer can send communication messages within its own network. That benchmark could provide a useful way of measuring AI’s progress toward a level comparable with human intelligence.
The AI Impacts project is the brainchild of two PhD students from the University of California, Berkeley, and Carnegie Mellon University. They have developed a preliminary methodology for comparing supercomputers to brains: traversed edges per second (TEPS), which measures how quickly a computer can move information around within its own system. A typical TEPS benchmark requires computers to simulate a graph and search through it. That’s not possible with the brain, so instead, the researchers compared the computer’s performance to a rough estimate of how frequently the brain’s neurons fire off electrical signals.
“A big pragmatic benefit of measuring the brain in terms of communication is that it hadn't been done before,” says Katja Grace, a researcher at the Machine Intelligence Research Institute in Berkeley who is working on a doctorate in Logic, Computation, and Methodology at Carnegie Mellon University. This method, says Grace, “provides a relatively independent estimate of the price of computing hardware roughly comparable to the brain.”
The AI Impacts project received $49,310 from the Boston-Based Future of Life Institute this summer. The grant made the pair of graduate student researchers one of 37 research teams to receive a slice of $7 million in funding donated by Elon Musk, founder of Tesla Motors, SpaceX, and the Open Philanthropy Project. Musk has been funding such AI-focused research in an effort to guide the development of smarter AI while minimizing potential dangers.
IBM’s Sequoia supercomputer currently holds the TEPS benchmark record with 2.3 x 1013 TEPS. Grace and her collaborator, Paul Christiano, a PhD student in theoretical computer science at Cal Berkeley, calculated that the human brain should be at least at least as powerful as Sequoia at the lower end of their TEPS estimates. At the upper end, their max estimate of the human brain’s capabilities suggest that it’s 30 times as powerful as IBM’s number cruncher at 6.4 times 1014 TEPS.
They’ve pegged the cost of the human brain’s performance at somewhere between $4,700 and $170,000 per hour in terms of current computer prices for TEPS. Grace and Christiano say they previously came up with a “fairly wild guess” that TEPS prices could improve by a factor of 10 every four years. That means computer hardware costing $100 per hour to operate could become competitive with the human brain during a time period between seven to 14 years.
But don’t panic, worrying that AI will replace humans en masse just yet. The researchers point out that there are many “ifs” and assumptions baked into their calculations. For example, they don’t have much information about how quickly TEPS performance might progress in computer hardware. It’s possible that progress could slow down in the near future.
Even if the TEPS price goal estimates prove reasonably accurate, there is no guarantee that just having the requisite computer hardware will lead to the emergence of AI on the level of human intelligence. A laptop’s worth of computing power doesn’t automatically spawn Microsoft Word, Grace pointed out. Similarly, humans would need to create the proper software to enable the emergence of more powerful AI.
“We have very little idea how efficiently the brain uses its computational resources, and how that will compare to the efficiency of systems that humans design,” Grace said. “So even if we knew how much hardware was needed to do what the brain is doing in the way the brain is doing it, this might be very different from the amount of hardware human engineers need to achieve the same functions once they have any way to achieve those functions.”
Still, the TEPS benchmark may provide another useful way to compare AI with human-level intelligence in the coming years. Measuring communication within the brain’s neurons is somewhat easier than trying to measure computations, because nobody knows exactly how computations are represented in the brain.
The effort to discover a reliable measure for comparing AI’s progress with the human brain represents just one part of the broader AI Impacts project. Grace and Christiano also want to understand whether AI research could make “abrupt and surprising progress at any point” or if it will mainly improve by small, incremental steps. If the latter proves true, researchers will have a much easier time predicting AI progress.
“In aid of this, we are looking at other technologies that have seen abrupt progress, which is interesting in itself,” Grace said. “The biggest jump in any technological trend that we have found was from nuclear weapons.”
To find more examples, the researchers recently introduced “research bounties” paying between $20 and $500 to anyone who submits examples of either “discontinuous technological progress” or people acting to prevent a risk that was at least 15 years away.
Grace and Christiano have also begun looking to Bitcoin hardware as possible evidence of how strong incentives can speed up the pace of hardware improvement. At some point, they anticipate that the widespread use of AI could also boost research progress. They eventually hope to build “a quantitative model of how fast artificial intelligence research should be expected to grow in an economy with increasing quantities of artificial intelligence available to do research.”
Editor’s Note: The original story said that the Machine Intelligence Research Institute is part of the University of California, Berkeley. It has been corrected to reflect the fact that the Machine Intelligence Research Institute is an independent organization located in Berkeley.