AI Beats Doctors At Breast Cancer Detection. Sometimes

The human capacity for insight remains unbeatable in a remarkable variety of circumstances. Some of them life-saving. Which is why the combination of AI and human interpretation will probably prevail, at least for the foreseeable future. JL


Brianna Abbott reports in the Wall Street Journal:

Google’s health research unit said it has developed an artificial-intelligence system that can match or outperform radiologists at detecting breast cancer, according to new research. But doctors still beat the machines in some cases. The cancers that the AI system caught were generally more invasive than those caught by the radiologists. “I think this is a testament to how difficult the task is and how weirdly good humans are at it, even with some of the best data in the world.”
Google’s health research unit said it has developed an artificial-intelligence system that can match or outperform radiologists at detecting breast cancer, according to new research. But doctors still beat the machines in some cases.
The model, developed by an international team of researchers, caught cancers that were originally missed and reduced false-positive cancer flags for patients who didn’t actually have cancer, according to a paper published on Wednesday in the journal Nature. Data from thousands of mammograms from women in the U.K. and the U.S. was used to train the AI system.

But the algorithm isn’t yet ready for clinical use, the researchers said.
The model is the latest step in Google’s push into health care. The Alphabet Inc. GOOG 0.07% company has developed similar systems to detect lung cancer, eye disease and kidney injury.
Google and Alphabet have come under scrutiny for privacy concerns related to the use of patient data. A deal with Ascension, the second-largest health system in the U.S., allows Google to use AI to mine personal, identifiable health information from millions of patients to improve processes and care.
The health data used in the breast-cancer project doesn’t include identifiable information, Google Health officials said, and the data was stripped of personal indicators before given to Google.
Radiologists and AI specialists said the model is promising, and officials at Google Health said the system could eventually support radiologists in improving breast-cancer detection and outcomes, as well as efficiency in mammogram reading.
“There’s enormous opportunity, not just in breast cancer but more widely, to use this type of technology to make screening more equitable and more accurate,” said Dominic King, the U.K. lead at Google Health. “It feels like this is another step towards this technology actually making a difference in the real world.”Breast cancer is the second-leading cause of cancer death in women after lung cancer, and roughly one in eight women in the U.S. are likely to develop breast cancer throughout their lifetime, according to the American Cancer Society. Early breast-cancer detection and treatment can save lives, experts said, and most health systems have screening protocols.
But many cases of breast cancer are missed. And sometimes mammograms are flagged for women who don’t have breast cancer or whose cancer is generally harmless, leading to extra testing or unnecessary treatment.
“It’s this balance of finding the important cancers and not causing undue distress over false positives that aren’t going to hurt a woman,” said Emily Conant, a radiologist and division chief of breast imaging at Penn Medicine.
When developing the AI system from the U.K. dataset, researchers fed the algorithm mammograms from the U.K. National Health Service’s breast-screening program. The U.S. dataset comprised mammograms taken from Northwestern Memorial Hospital in Chicago. Whether a woman had breast cancer was previously determined, and researchers told the algorithm which cases had confirmed breast cancer.
The AI system was then tested on different mammograms of more than 25,000 women in the U.K. and 3,000 women in the U.S. from those datasets. The AI system reduced missed cases by 9.4% in the U.S. and 2.7% in the U.K. compared with the original radiologist diagnoses. It also reduced incorrect positive readings by 5.7% and 1.2%, respectively.
In the U.K., where two radiologists typically read a mammogram, the study found that the model didn’t perform worse than the second reader and could potentially reduce their workload by 88%.
The researchers then had six U.S. radiologists who didn’t make the original diagnoses look at 500 U.S. mammograms and compared their responses with the AI system’s. The radiologists also received the patients’ history and past mammograms when available, while the AI system didn’t. The AI system outperformed the average radiologist in determining whether the women would develop breast cancer.While the AI system caught cancers that the radiologists missed, the radiologists in both the U.K. and the U.S. caught cancers that the AI system missed. Sometimes, all six U.S. readers caught a cancer that slipped past the AI, and vice versa, said Mozziyar Etemadi, a research assistant professor in anesthesiology and biomedical engineering at Northwestern University and a co-author of the paper.
The cancers that the AI system caught were generally more invasive than those caught by the radiologists; the researchers didn’t have an explanation for the discrepancies.
“I found it sobering,” said Ziad Obermeyer, acting associate professor of health policy and management at the University of California, Berkeley who studies machine learning and health and wasn’t involved in the research. “I think this is a testament to how difficult the task is and how weirdly good humans are at it, even with some of the best data in the world.”
Researchers now want to see how the model would behave in the clinic.
“The real world is more complicated and potentially more diverse than the type of controlled research environment reported in this study,” Etta Pisano, chief research officer at the American College of Radiology, wrote in an editorial in the journal Nature about the paper.
Google Health said it is talking with health systems and research groups about how best to incorporate the AI system into clinical workflow.