Technology showcased in the movie Minority Report, which enabled Tom Cruise to swipe through midair images in the 2002 film, could soon become a staple of hospital operating rooms. A new gesture-controlled computer interface aims to give surgeons easier access to medical images during marathon surgical operations.
The experimental medical system takes advantage of Leap Motion controllers that can sense and track people’s hand gestures. South Korean researchers developed their own “GestureHook” software that can translate the gestures captured by a Leap Motion device into commands for several different types of medical software.
“We thought that using gestures as a new interface for controlling software in hospitals would provide access to computers for surgeons during procedures,” says Ben Joonyeon Park, a software developer on the Medical Information Development Team at the Asan Medical Center in Seoul, South Korea. Manipulating software programs this way, says Park, will let surgeons view 2-D or 3-D images of patients’ body parts without relying on assistants or dirtying their hands by touching a mouse or keyboard.
The system has undergone only one preliminary test, with a plastic surgeon during a double jaw surgery operation. But Park and his colleagues hope that the gesture-controlled system could also help radiologists, who spend much of their time browsing X-ray, MRI, and CT scan images. Their research was detailed in the 14 January 2016 issue of the journal Computational and Mathematical Methods in Medicine.
The team, from the Asan Medical Center and the University of Ulsan College of Medicine, considered a few different motion-tracking technologies at first. Among them was the Microsoft Kinect device that first found success in video games but has also been used for other motion-control applications. But the Korean researchers eventually settled on the Leap Motion device because it provided vector coordinates that were easier to plug into the software they developed.
The biggest innovation came on the software side. Park and his colleagues created their GestureHook software to act as a middleman between the Leap Motion hardware and the myriad medical programs currently in use. Such programs include the PACS (Picture Archiving and Communication System) software that allows surgeons and radiologists to look at 2-D scans or even manipulate 3-D images of certain organs or body parts.
Testing showed that the gesture-controlled system was a few seconds faster, on average, than using a mouse and keyboard to manipulate the viewing angle of head scans from 10 multiple-fracture patients. Researchers also tested the accuracy of the system in recognizing basic hand gestures such as tapping or picking fingers, grabbing hands, or performing mouse clicks and double clicks. The system generally performed well, aside from having some trouble recognizing a left finger pinch gesture and the mouse double-click gesture.
“I don't think the device itself is accurate enough yet to use in hospitals, but this can be corrected by finding out ways to detect outliers,” Park explains. “We will definitely look into other devices such as smart rings, but haven't found the perfect fit yet.”
The South Korean team, which plans to apply for a patent on their system, says its next step involves testing how individual physicians can train the software on their own personalized gestures. They also hope to continue improving the system’s gesture recognition accuracy and the range at which it can detect hand gestures.
The gesture-control technology could even someday work with certain wearable devices. Surgeons who want to keep their hands clean might prefer something such smart bands or rings, Park says. There is also still the possibility of trying out the Microsoft Kinect because it can recognize a much wider range of user activities and gestures than Leap Motion can.
And who knows? A Minority Report-style version of the technology might someday emerge, with images floating in midair.
“I guess the next big leap would be combining this software with holographic technologies,” Park says.
Jeremy Hsu has been working as a science and technology journalist in New York City since 2008. He has written on subjects as diverse as supercomputing and wearable electronics for IEEE Spectrum. When he’s not trying to wrap his head around the latest quantum computing news for Spectrum, he also contributes to a variety of publications such as Scientific American, Discover, Popular Science, and others. He is a graduate of New York University’s Science, Health & Environmental Reporting Program.
