The First Frontier for Medical AI Is the Pathology Lab

But before adopting startup PathAI’s tools, doctors must see if they are worth the cost

10 min read
Illustration: Carl De Torres
Illustration: Carl De Torres

illustrationIllustration: Carl De Torres

This is how a pathologist could save your life.

Imagine you’re coughing up blood, and a chest scan reveals a suspicious mass in your lungs. A surgeon removes a small cylindrical sample from the potential tumor, and the pathologist places very thin slices of the tissue on glass slides. After preserving and staining the tissue, the pathologist peers through a microscope and sees that the cells have the telltale signs of lung cancer. You start treatment before the tumor spreads and grows.

And this is how a pathologist could kill you: The expert physician would just have to miss the cancer. Or, more likely, misclassify the cells viewed on the slides as the wrong cancer subtype. Rather than getting a targeted therapy that beats your cancer into remission, you receive conventional chemo that buys you a few more months of life.

An artificially intelligent pathologist probably wouldn’t make that mistake. Trained on vast troves of digitized slides showing an enormous variety of tumors, artificial-intelligence (AI) systems will likely provide more accurate diagnoses than human pathologists, at least on fairly rote diagnostic tasks. They may even pick up on subtle features that the best-trained human eyes could never see. In this crucial, high-stakes branch of medicine, AI tools may soon offer diagnoses—and treatment recommendations—that are as close to infallible as we’re likely to get in the foreseeable future. And they’ll do so in a matter of seconds.

Lately, dazzlingly high success rates for AI-based systems in recognizing the presence of certain specific illnesses have prompted speculation that such tools will replace doctors. But the developments in pathology show us a more likely outcome: that machines will make the ever-increasing complexity of modern medicine manageable for human beings. This human-machine combination will outperform what either could do individually. At first, the improvement will be small. But eventually, it will be great.

“The promise of machine learning is to augment what a pathologist can do alone,” says Ulysses Balis, director of the division of informatics at the University of Michigan’s pathology department and chief strategy officer of a digital pathology company called Inspirata. “These technologies allow the profession to scale with increased demand.”

It’s clear that the era of AI medicine has begun. Over the past year, a raft of diagnostic tools powered by machine-learning algorithms have entered the clinical marketplace, making it easier to spot wrist fractures, diabetic eye disease, and signs of stroke with little or no human input. But these early applications are merely automating tasks otherwise performed by expert diagnosticians, and they’re typically just interpreting imagery, such as X-rays and CT scans. The software may offer a slight edge over a trained specialist in analytic precision and accuracy, and it’s almost always faster. Yet the technology hasn’t radically expanded what’s diagnostically possible today.

AI pathology, in contrast, will be radical—and it’s coming soon. In 2019, several companies will ask the U.S. Food and Drug Administration to authorize the first AI-backed tools for this field. Unlike fields such as radiology and ophthalmology, in which diagnoses are typically limited to the visual realm, diagnoses in pathology can incorporate the tools of biochemistry, immunology, and genetics, adding molecular detail to images of thinly sliced and stained tissue specimens.

Combining all that data enables AI to draw diagnostic inferences that would be impossible for the world’s best clinicians. So says Andrew H. Beck, a pathologist by training who cofounded and runs a three-year-old startup called PathAI, based in Boston. He says his tools will bring real improvements in the accuracy of diagnoses and the efficacy of treatment. “Pathology will be one of the areas where we first see AI truly revolutionize medicine,” Beck declares.

Beck is not alone in this line of thinking. Software giants (including Google and IBM), medical-device manufacturers (including Philips and Leica Biosystems), and dozens of startups are developing pattern-recognition algorithms to help pathologists spot cancerous cells or other diseased cells using digitized imagery of tissue on glass slides. Proponents note that fewer than 2 percent of today’s medical graduates elect to go into pathology; smart software could alleviate the global shortage and lighten the workload on overburdened experts.

“These are intelligent guides that will help pathologists do their jobs more efficiently and effectively,” says Michael J. Becich, a pathology informatics researcher at the University of Pittsburgh Medical Center. “It is really a democratization of expertise in cancer care,” says Thomas Fuchs, a computational pathologist at the Memorial Sloan Kettering Cancer Center, in New York City. Both Becich and Fuchs recently started their own companies with similar goals to those of Beck’s PathAI.

Beck’s edge is his deep knowledge of both pathology and software. He’s a pathologist who sought out computer science training to bring his discipline—one rooted in 19th-century microscope techniques—into the 21st century. “Andy knows molecular biology and genetics, he knows deep learning, and he has the wherewithal to integrate all these things,” says Stuart Schnitt, a breast cancer pathologist at Boston’s Brigham and Women’s Hospital and a scientific advisor to PathAI. To drive home his point, Schnitt uses a sports analogy: “He’s the equivalent of a ‘five-tool’ baseball player.” Which is to say, a well-rounded virtuoso.

It’s an apt analogy for someone who chose to station his company less than 200 meters from Fenway Park, Boston’s hallowed baseball stadium. At the company headquarters, Beck offers a demo of the PathAI platform. He zooms in and out on a digital image of a tiny section of cancerous lung tissue, toggling between a standard view of the microscope slide and colorful overlays that enrich the view by highlighting specific cells or cancer-linked proteins.

Other digital pathology startups are also providing this Google Map–like perspective of the tumor’s cellular topography and underlying molecular patterns. But the real power behind the PathAI system is invisible to the user. The company trains its machine-learning algorithms on digitized slides coupled with clinical data, such as tumor aggressiveness, treatment plans, and patient outcomes, giving it the ability to do statistical analyses that are well beyond the ability of any human brain.

The company’s models not only do recon on the enemy, detecting cancer cells and rating the advancement of tumors; they also suggest lines of attack. They do this, in part, by counting the immune cells that have surrounded the tumor and determining whether those cells have certain properties that make them useful for the latest immunotherapies—treatments that amp up the body’s natural defenses to fight cancer.

All that information is invaluable to drug developers like Bristol-Myers Squibb (BMS), one of the many pharmaceutical giants using PathAI’s platform to determine why a mere fraction of clinical trial participants respond to anticancer drugs. The startup is now bringing in a steady stream of licensing royalties from drug companies, adding that revenue to the US $15 million it has raised in venture capital.

Michael Montalto, BMS’s head of translational pathology, explains that his team now relies on PathAI’s technology to determine whether tumor cells in a biopsy sample are cloaked in a disguise protein, keeping immune cells from recognizing the cancer cells as dangerous. Drugs like BMS’s immunotherapy agents effectively unmask the tumor—but they work only on those cancers that have concealed themselves in this way. And that’s just one example of AI’s usefulness, he says. “We are really driving toward using this technology routinely across all our trials,” Montalto says.

Cancer’s Secrets


imgTo help doctors plan an attack on a lung tumor, PathAI’s software maps the types of tissue present (top), showing in red the epithelial cells that are key to cancer’s progression. It also makes a map of immune cells (bottom), shown as yellow squares, to determine whether new immunotherapy drugs might be effective against the tumor.Images: PathAI

The big market opportunity for AI pathology companies, though, lies not in the research setting. It’s in the standard diagnostic workups used to determine the nature of every cancer patient’s tumor—and to guide treatment options. There’s just one obstacle to seizing that market: The entire infrastructure of pathology has to change. “You can only use these algorithms if your slides are digitized before your pathologist goes to look at them for diagnosis, and there are not a lot of places that do that,” says Jeroen van der Laak, a computational pathologist from the Radboud University Medical Center, in the Netherlands.

Although many pathology labs now make digital copies of glass slides for archival purposes or after-the-fact research projects, there are only a few early adopters, mostly in Europe, that scan them up front for diagnosis. Hospitals have been slow to incorporate automated whole-slide imaging because the technology is expensive: upwards of $250,000 for the scanner, plus the additional cost of storing gigapixel-size image files.

The investment is worth it, insists Anil Parwani, head of digital pathology at the Ohio State University Comprehensive Cancer Center, one of the only sites in the United States where pathologists now digitally scan slides as part of their routine diagnostic workflow. Parwani says his hospital’s fully digital platform will pay for itself within five years, thanks to improvements in doctor productivity and reductions in diagnostic errors. Digitizing slides also allows for online file sharing, rather than shipping physical slides for remote diagnosis or second opinions. Plus, “it’s made the workflow more robust,” Parwani says, as pathologists can instantly compare biopsies taken months apart or review cases on the go.

If digital-slide scanning is paired with powerful quantification algorithms, the added value should become obvious, says David West, founder and CEO of Proscia, a digital pathology startup in Baltimore. “This will likely become the standard much more quickly than people expect,” he says. And when it does, “the role of the pathologist is certainly going to change. The best pathologists are going to become informaticians, and the best pathology labs are going to be informatics driven.”

“There’s definitely a disruptive nature to this,” West adds.

Beck, ofPathAI,started down the road to disruption as a medical student at Brown University in the 2000s, when he began dabbling in quantitative image analysis. Working with pathologist Murray Resnick, Beck helped develop a computer program that evaluated the size, shape, and other features of esophageal cells to determine a patient’s risk of cancer. It wasn’t a deep-learning algorithm, but his interest in quantified medicine propelled him to Stanford, where he followed his pathology residency with a Ph.D. in the laboratory of AI scientist Daphne Koller. That research culminated in the development of the Computational Pathologist, or C-Path, system, a fairly primitive machine-learning tool for grading the severity of breast tumors. In 2011, the group published its findings, demonstrating one of the first applications of AI in pathology.

At the time, says Koller, “no one was taking this very broad, data-driven approach to this problem.” In previous attempts to automate tissue analysis, researchers had generally told their programs what features to look for—as Beck and Resnick had done five years earlier in their study of esophageal cancer. With C-Path, Beck fed his algorithm hundreds of features, practically every one he could think of and measure. He let the computer code take care of the rest.

AI is sometimes criticized for being a “black box.” Because deep-learning programs like C-Path effectively teach themselves how to interpret images, it’s impossible to know exactly how these algorithms arrive at their final decisions. Yet, “just because it’s a black box doesn’t mean you can’t get very useful ideas from it,” says Matt van de Rijn, a pathologist at Stanford and one of Beck’s mentors. Thanks to C-Path, for example, Beck discovered that the most predictive features for breast cancer survival were not in the tumor cells themselves, but rather in the surrounding region, where few human pathologists thought to look. “That was an amazing finding that could very well lead to new interpretations in pathology,” Van de Rijn says.

After Stanford, Beck moved back east to start his own research group at the Beth Israel Deaconess Medical Center, an affiliate of Harvard Medical School, where he stepped back from machine-learning algorithms and focused largely on cancer epidemiology. Then, in 2015, an international competition launched by Dutch researchers pulled him back into the disruptive world of AI.

Radboud University’s Van der Laak spearheaded the contest, which challenged machine-learning specialists to find new techniques for early detection of breast cancer. In particular, Van der Laak asked researchers to find invasive breast cancer lurking inside lymph nodes, a determination that’s essential to plotting the correct course of treatment. “It’s a task that every pathologist hates because it’s a lot of work and it’s not really intelligent work,” he says. If an algorithm could do the task as well as or better than a human, Van der Laak figured, it would show doctors that AI was an asset—something that could free overstretched pathologists to focus on more complex tasks—and not something to be feared as a job killer.

The Cancer Metastases in Lymph Nodes Challenge (Camelyon) did not have the cachet or financial payout of an Ansari X Prize or DARPA Grand Challenge. But people involved say it spurred innovation in computational pathology just as those better-known contests helped jump-start industries for private spaceflight and autonomous cars. “Everyone was driving each other to get better,” Beck says, “because everyone wanted to win.”

photoThe Expert’s View: After training as a pathologist and an AI scientist, Andrew Beck became convinced that machine-learning tools could revolutionize the practice of pathology.Photo: PathAI

Beck’s team came up with a simple method that yielded big results. They devised a two-step verification system to ensure all patches of tissue initially labeled “clean” by the AI were indeed cancer-free. The resulting algorithm proved as good as and sometimes better than an expert pathologist at determining whether slides contained tumor cells, and also at determining where the cancerous masses sat within the larger tissue sample. Beck’s team ultimately beat out 22 other groups to come out atop the Camelyon leaderboard.

Besides bragging rights (and a gold-colored 1-terabyte external hard drive presented as a prize at the 2016 IEEE International Symposium on Biomedical Imaging, in Prague), Beck says that the victory also gave him the confidence to venture out on his own. In January 2017, he resigned from his tenure-track position at Harvard and devoted himself to PathAI.

The company is now working on three types of decision-support tools. First, PathAI is developing algorithms to take on pathologists’ most loathed and repetitive chores, such as identifying metastases in lymph nodes (the application in the Camelyon challenge) and other simple determinations of whether cancer cells are present. These aren’t difficult tasks, Beck says, but they’re time consuming and not considered the best use of human expertise. PathAI is currently partnered with Philips, the Dutch health-technology giant, on one such image-analysis system for automatically detecting cancerous lesions in breast tissue.

The second application involves determining the “grade” of cancer. Jonathan Epstein, a pathologist at Johns Hopkins University, in Baltimore, describes this decision about the aggressiveness of a tumor as “difficult, subjective, and one of the most critical aspects of treatment.” Epstein, an advisor to PathAI and an expert on urological cancers, is working with the company to train its algorithms to diagnose tumors of the prostate and other organs.

Lastly, the company is further developing biomarker detection tools like the one that pharmaceutical companies are using to understand who can benefit from their drugs. If validated in clinical trials, those same algorithms could help doctors personalize drug choices for all patients.

To date, PathAI has tested its software on cancers of the lung, bladder, skin, prostate, breast, colon, and stomach. “The platform is very transferable, and that’s why we’ve been able to work on pretty much all major solid tumors,” says Beck, adding, “The process gets faster and better with every new project and every new indication.”

As with any new technology, there’s a risk of overselling what machine learning can do for the field, but University of Michigan pathologist David McClintock insists that pilot studies have shown that the promise is real. “When appropriately deployed, machine-learning tools can be of assistance,” he says. “I don’t think that’s hype. That’s a fact.” The biggest obstacles to AI-powered improvements in patient care may be getting regulators to approve these new tools and getting doctors to use them.

But as the technology matures, one big question looms: Could AI go beyond assistance and eventually replace human pathologists entirely? Beck dismisses the possibility out of hand. “It’s just this cliché that people can’t get out of their heads,” he says. Machine learning may help with specific diagnostic tasks, he says, but finding the best treatment for a sick patient requires synthesizing many types of clinical information, including cell stains, protein annotations, genetic profiles, and electronic health records. Careful judgment is required to put all the information together and come to a definitive diagnosis and treatment plan. That synthesis is where human pathologists show their worth, Beck says: “AI is not going to figure that out by itself.” 

This article appears in the December 2018 print issue as “The AI Medical Revolution Starts Here.”

This article is for IEEE members only. Join IEEE to access our full archive.

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, podcasts, and special reports. Learn more →

If you're already an IEEE member, please sign in to continue reading.

Membership includes:

  • Get unlimited access to IEEE Spectrum content
  • Follow your favorite topics to create a personalized feed of IEEE Spectrum content
  • Save Spectrum articles to read later
  • Network with other technology professionals
  • Establish a professional profile
  • Create a group to share and collaborate on projects
  • Discover IEEE events and activities
  • Join and participate in discussions