Ophthalmologists vs. AI: It's a Tie

A child with cataracts
Photo: Heydt/VWPics/AP Photo

Last week, we reported on an algorithm that recognizes skin cancer as well as the world’s best dermatologists. That computer program was trained using 130,000 images from more than 2,000 diseases. It, like most artificial intelligence (AI) breakthroughs, relied on big data.

Now, a team in China has demonstrated that AI also has the potential to aid medical diagnoses in situations where there is limited high-quality data available. An AI program trained with just 410 images of congenital cataracts (a rare disease that causes irreversible blindness), plus 476 images of disease-free eyes performed as well as doctors in diagnosing the condition, recognizing its severity, and offering a treatment suggestion.

Inspired by a 2015 research paper from Google’s DeepMind artificial-intelligence company, which described how a machine-learning algorithm beat professional human players at a series of arcade games based on minimal starting information, Haotian Lin, an ophthalmologist at Sun Yat-sen University in China, and colleagues had the idea of creating an AI agent to mine their clinical database on childhood cataracts.

Working with Xiyang Liu’s team at Xidian University, they created CC-Cruiser, an AI program able to diagnose congenital cataracts, predict the severity of the disease, and suggest treatment decisions. The program was created using deep-learning algorithms trained with the aforementioned images of affected and control eyes from children.

Then, the researchers put CC-Cruiser to five tests. First, in a computer simulation, the AI program was able to distinguish patients and healthy individuals with 98.87 percent accuracy. It also achieved above 93 percent accuracy in estimating each of three indicators of the disease’s severity: lens opacity areas, density, and location. The program also provided treatment suggestions with 97.56 percent accuracy.

Next, the team conducted a clinical trial using 57 images of children’s eyes from three collaborating hospitals in China. None of the chosen hospitals specialize in diagnosing or treating this condition, because the team hopes the platform will eventually be most useful in helping hospitals like these, which lack on-site specialists. Again, CC-Cruiser performed well: 98.25 percent identification accuracy; over 92 percent on all three severity factors; and 92.86 percent accuracy in treatment suggestions.

In yet another test of the AI’s capabilities, the program was presented with 53 low-quality cases mined from the Internet. It handled them with a high level of accuracy. But the researchers still weren’t done. After that, the program successfully identified three needle-in-a-haystack cases, correctly pointing out three cataract cases in a data set with 300 normal cases.

Finally, in an effort to simulate real-world use, they pitted the program directly against individual ophthalmologists. CC-Cruiser and three ophthalmologists—an expert, competent, and a novice—went head-to-head diagnosing 50 clinical cases. The computer and doctors performed comparably.

The program did incorrectly flag a few cases in the hospital trial, and Lin hopes that a larger dataset could improve its performance. The team plans to build a collaborative cloud platform to do so, but Lin emphasize that the technology is “insufficient” to determine the best course of treatment with 100 percent accuracy. “Doctors should therefore make good use of the machine’s suggestion to identify and prevent the potential misclassification and complement their own judgment,” Lin told IEEE Spectrum in an email.

So, it’s unlikely that CC-Cruiser will make ophthalmologists obsolete anytime soon. Especially because there is one key skill that it cannot do as well as doctors: “The human ability to communicate and interact affectively is indispensable for medical treatment,” said Lin. “The simulation of human emotion is very challenging for machine[s]. The face-to-face interaction between doctors and patients will be one of the last bastions of human intelligence.”

The team hopes that, with further clinical trials, doctors in non-specialized hospitals could use the program to identify the condition and send patients to specialized centers. Patients may eventually use it themselves and see out a specialist if there were a concern, added Lin. “The ultimate goal of artificial intelligence is to leverage it, in combination with human abilities, to make the world a better place.”

Advertisement

The Human OS

IEEE Spectrum’s biomedical blog, featuring the wearable sensors, big data analytics, and implanted devices that enable new ventures in personalized medicine.

 
Editor
Eliza Strickland
New York City
Contributor
Emily Waltz
Nashville
 
Contributor
Megan Scudellari
Boston
 

Newsletter Sign Up

Sign up for The Human OS newsletter and get biweekly news about how technology is making healthcare smarter.

Advertisement