Jun Rekimoto stands in the corner of his office, frowning at a small white refrigerator. He’s not worrying about his diet—in fact, there’s no food inside. Rather, he’s showing off a new method of interactivity with the world of appliances, as a red frowny face lights up on the door of the fridge. “You see, it won’t open until I smile,” he says, pulling on the door to demonstrate that it’s locked shut. Then the pensive researcher’s face is transformed by a beaming grin. The fridge responds with a smiley face of its own, in bright green, and the door swings open.
This emotionally engaged fridge is one of Rekimoto’s many inventions for Sony, but don’t expect to see expressive appliances in stores anytime soon. Sony, as one of the most successful consumer-electronics companies on the planet, has plenty of research-and-development labs devoted to improving its televisions, computers, cameras, and so forth. But Rekimoto works at Sony Computer Science Laboratories (CSL), in Tokyo, an exclusive scientific sanctum where researchers are encouraged to pursue their wackiest ideas and push the limits of technology. Rekimoto needn’t concern himself with whether or not his creations will turn into products in the near term, because Sony wants him to invent gadgets for a future that consumers haven’t even imagined yet.
Rekimoto trained his own imagination early. He became fascinated by computers when he was growing up in Tokyo, and in 1971, at the tender age of 10, he went to a bookstore and purchased a textbook about programming. There was only one barrier to his learning: The personal computer didn’t exist yet, and a 10-year-old wasn’t likely to gain access to a university’s mainframe. So he wrote out his programs in a notebook and imagined the results. “I wrote them down like a mantra,” he says, “like magic words.”
Nowadays he has plenty of computers at his disposal. Rekimoto is sitting in his office at the University of Tokyo—he added a professorship in information science to his job at Sony a few years ago. The table in front of him is littered with miscellaneous electronic parts and tiny screwdrivers, finished projects hang from the walls and sit stacked on shelves, and a team of graduate students work in the lab across the hall.
Although Rekimoto’s projects are wide-ranging, most focus on the interfaces between humans and technology. The topic first fascinated him during his undergraduate studies at the Tokyo Institute of Technology in the early 1980s. There, he came across an article by Alan Kay, a legendary computer scientist from Xerox’s Palo Alto Research Center, that discussed graphical computer interfaces. Rekimoto loved the idea, and decided to devote his career to crafting new ways for people to interact with machines.
After earning both bachelor’s and master’s degrees in information science, Rekimoto took a job in the R&D department of NEC Corp., a computer and communications giant, where he worked on interfaces for Unix. Had he followed the traditional Japanese career path, Rekimoto would have stayed at NEC until retirement. But in 1993, when he came back from a stint as a visiting scientist at the University of Alberta, in Canada, he found himself in Japan’s post-bubble economy. NEC and many of the other big technology companies no longer had big budgets for R&D.
In Canada, he had played around with virtual-reality systems but had been disappointed with their unrealistic graphics. A better approach, he thought, would be to layer information over the real world—a strategy now called augmented reality. But he knew such research would be expensive and wouldn’t pay off in the short term, making it unlikely that NEC would support it.
So when he received an offer to join Sony CSL, he jumped at the chance. He remembers that CSL founder Mario Tokoro warned him that the job wouldn’t be anything like working for one of Japan’s technology giants. “Mario told me, ‘You’ll have freedom, but freedom is very hard, freedom means responsibility—there are no excuses,’ ” Rekimoto recalls. He’d have to dream up his own assignments and push himself to complete them.
One of his earliest projects, NaviCam, used a handheld computer with a video camera, something remarkably similar to today’s smartphones. With this device, a user could scan bar codes and the correlating information would appear on the screen, overlaid on the user’s view of his or her surroundings. For example, if a user scanned a code on the front door of the CSL office, the display showed the paths to various meeting rooms and researchers’ offices as arrows on the floor.
Some of his experiments did go on to inform Sony’s commercial products, like his work with near-field communication, or NFC. Nearly ten years later Sony introduced electronic gear, including camera lenses and speakers, that communicate with a smartphone via Rekimoto’s NFC system.
Rekimoto thinks that a good user interface enhances human capacities, a view he first heard expressed by Douglas Engelbart, the inventor of the computer mouse. “He said that the mouse was just a tiny piece of a much larger project aimed at augmenting the human intellect,” Rekimoto remembers. “So my work has become more about human augmentation.”
One recent project, Flying Head, involves a headset that the user wears to both see through the eyes of and control a small unmanned aerial vehicle, or UAV. “You wear the headset and you feel that you can fly—you can be a helicopter,” he says. Rekimoto imagines such a system being useful for a nonpilot specialist who needs to access a dangerous area, such as the ruins of Japan’s Fukushima Daiichi nuclear power plant. A UAV could fly above the radioactive rubble, where wheeled robots would have difficulty progressing, and the specialist guiding it remotely could examine the damage from a safe distance.
Even the grin-operated fridge fits into Rekimoto’s vision. That project, he explains, also aims for human augmentation: By encouraging its users to smile more often, the fridge can improve their mental well-being. “If it is our destiny to merge with machines, we should think about what parts of humans can be augmented by that merger,” he says. “Technology shouldn’t just improve our intellectual abilities—technology should make you happy.” Rekimoto and his fridge exchange another smile.
This article originally appeared in print as “The Willy Wonka of Electronics.”