"Minority Report" Tech Meets the Operating Room

A gesture-controlled system could allow surgeons to swipe through medical images without dirtying their hands

3 min read
"Minority Report" Tech Meets the Operating Room
Photo: Ben J. Park/Asan Medical Center

Technology showcased in the movie Minority Report, which enabled Tom Cruise to swipe through midair images in the 2002 film, could soon become a staple of hospital operating rooms. A new gesture-controlled computer interface aims to give surgeons easier access to medical images during marathon surgical operations.

The experimental medical system takes advantage of Leap Motion controllers that can sense and track people’s hand gestures. South Korean researchers developed their own “GestureHook” software that can translate the gestures captured by a Leap Motion device into commands for several different types of medical software.

“We thought that using gestures as a new interface for controlling software in hospitals would provide access to computers for surgeons during procedures,” says Ben Joonyeon Park, a software developer on the Medical Information Development Team at the Asan Medical Center in Seoul, South Korea. Manipulating software programs this way, says Park, will let surgeons view 2-D or 3-D images of patients’ body parts without relying on assistants or dirtying their hands by touching a mouse or keyboard.

The system has undergone only one preliminary test, with a plastic surgeon during a double jaw surgery operation. But Park and his colleagues hope that the gesture-controlled system could also help radiologists, who spend much of their time browsing X-ray, MRI, and CT scan images. Their research was detailed in the 14 January 2016 issue of the journal Computational and Mathematical Methods in Medicine.

The team, from the Asan Medical Center and the University of Ulsan College of Medicine, considered a few different motion-tracking technologies at first. Among them was the Microsoft Kinect device that first found success in video games but has also been used for other motion-control applications. But the Korean researchers eventually settled on the Leap Motion device because it provided vector coordinates that were easier to plug into the software they developed.

The biggest innovation came on the software side. Park and his colleagues created their GestureHook software to act as a middleman between the Leap Motion hardware and the myriad medical programs currently in use. Such programs include the PACS (Picture Archiving and Communication System) software that allows surgeons and radiologists to look at 2-D scans or even manipulate 3-D images of certain organs or body parts.

Testing showed that the gesture-controlled system was a few seconds faster, on average, than using a mouse and keyboard to manipulate the viewing angle of head scans from 10 multiple-fracture patients. Researchers also tested the accuracy of the system in recognizing basic hand gestures such as tapping or picking fingers, grabbing hands, or performing mouse clicks and double clicks. The system generally performed well, aside from having some trouble recognizing a left finger pinch gesture and the mouse double-click gesture.

“I don't think the device itself is accurate enough yet to use in hospitals, but this can be corrected by finding out ways to detect outliers,” Park explains. “We will definitely look into other devices such as smart rings, but haven't found the perfect fit yet.”

The South Korean team, which plans to apply for a patent on their system, says its next step involves testing how individual physicians can train the software on their own personalized gestures. They also hope to continue improving the system’s gesture recognition accuracy and the range at which it can detect hand gestures.

The gesture-control technology could even someday work with certain wearable devices. Surgeons who want to keep their hands clean might prefer something such smart bands or rings, Park says. There is also still the possibility of trying out the Microsoft Kinect because it can recognize a much wider range of user activities and gestures than Leap Motion can.

And who knows? A Minority Report-style version of the technology might someday emerge, with images floating in midair.

“I guess the next big leap would be combining this software with holographic technologies,” Park says.

The Conversation (0)

This CAD Program Can Design New Organisms

Genetic engineers have a powerful new tool to write and edit DNA code

11 min read
A photo showing machinery in a lab

Foundries such as the Edinburgh Genome Foundry assemble fragments of synthetic DNA and send them to labs for testing in cells.

Edinburgh Genome Foundry, University of Edinburgh

In the next decade, medical science may finally advance cures for some of the most complex diseases that plague humanity. Many diseases are caused by mutations in the human genome, which can either be inherited from our parents (such as in cystic fibrosis), or acquired during life, such as most types of cancer. For some of these conditions, medical researchers have identified the exact mutations that lead to disease; but in many more, they're still seeking answers. And without understanding the cause of a problem, it's pretty tough to find a cure.

We believe that a key enabling technology in this quest is a computer-aided design (CAD) program for genome editing, which our organization is launching this week at the Genome Project-write (GP-write) conference.

With this CAD program, medical researchers will be able to quickly design hundreds of different genomes with any combination of mutations and send the genetic code to a company that manufactures strings of DNA. Those fragments of synthesized DNA can then be sent to a foundry for assembly, and finally to a lab where the designed genomes can be tested in cells. Based on how the cells grow, researchers can use the CAD program to iterate with a new batch of redesigned genomes, sharing data for collaborative efforts. Enabling fast redesign of thousands of variants can only be achieved through automation; at that scale, researchers just might identify the combinations of mutations that are causing genetic diseases. This is the first critical R&D step toward finding cures.

Keep Reading ↓ Show less