Study: AI As Good As Humans At Interpreting Medical Diagnoses

That may good news as the demand for quicker, more accurate diagnoses is growing. The question is whether the state of diagnosis generally, whether human or machine, is adequate. JL

Nicola Davis reports in The Guardian:

Deep learning systems correctly detected a disease state 87% of the time – compared with 86% for healthcare professionals – and correctly gave the all-clear 93% of the time, compared with 91% for human experts. Deep learning systems could act as a diagnostic tool and help tackle the backlog of scans and images. They could (also) prove useful in places which lack experts to interpret images. “H ype over AI in medicine obscures the lamentable quality of almost all evaluation studies. Deep learning can be a powerful technique, but clinicians should be asking what does it actually add to clinical practice?”
Artificial intelligence is on a par with human experts when it comes to making medical diagnoses based on images, a review has found.
The potential for artificial intelligence in healthcare has caused excitement, with advocates saying it will ease the strain on resources, free up time for doctor-patient interactions and even aid the development of tailored treatment. Last month the government announced £250m of funding for a new NHS artificial intelligence laboratory.
However, experts have warned the latest findings are based on a small number of studies, since the field is littered with poor-quality research.
One burgeoning application is the use of AI in interpreting medical images – a field that relies on deep learning, a sophisticated form of machine learning in which a series of labelled images are fed into algorithms that pick out features within them and learn how to classify similar images. This approach has shown promise in diagnosis of diseases from cancers to eye conditions.
However questions remain about how such deep learning systems measure up to human skills. Now researchers say they have conducted the first comprehensive review of published studies on the issue, and found humans and machines are on a par.
Prof Alastair Denniston, at the University Hospitals Birmingham NHS foundation trust and a co-author of the study, said the results were encouraging but the study was a reality check for some of the hype about AI.
Dr Xiaoxuan Liu, the lead author of the study and from the same NHS trust, agreed. “There are a lot of headlines about AI outperforming humans, but our message is that it can at best be equivalent,” she said.
Writing in the Lancet Digital Health, Denniston, Liu and colleagues reported how they focused on research papers published since 2012 – a pivotal year for deep learning.
An initial search turned up more than 20,000 relevant studies. However, only 14 studies – all based on human disease – reported good quality data, tested the deep learning system with images from a separate dataset to the one used to train it, and showed the same images to human experts.
The team pooled the most promising results from within each of the 14 studies to reveal that deep learning systems correctly detected a disease state 87% of the time – compared with 86% for healthcare professionals – and correctly gave the all-clear 93% of the time, compared with 91% for human experts.
However, the healthcare professionals in these scenarios were not given additional patient information they would have in the real world which could steer their diagnosis.Prof David Spiegelhalter, the chair of the Winton centre for risk and evidence communication at the University of Cambridge, said the field was awash with poor research.
“This excellent review demonstrates that the massive hype over AI in medicine obscures the lamentable quality of almost all evaluation studies,” he said. “Deep learning can be a powerful and impressive technique, but clinicians and commissioners should be asking the crucial question: what does it actually add to clinical practice?”
However, Denniston remained optimistic about the potential of AI in healthcare, saying such deep learning systems could act as a diagnostic tool and help tackle the backlog of scans and images. What’s more, said Liu, they could prove useful in places which lack experts to interpret images.Liu said it would be important to use deep learning systems in clinical trials to assess whether patient outcomes improved compared with current practices.
Dr Raj Jena, an oncologist at Addenbrooke’s hospital in Cambridge who was not involved in the study, said deep learning systems would be important in the future, but stressed they needed robust real-world testing. He also said it was important to understand why such systems sometimes make the wrong assessment.
“If you are a deep learning algorithm, when you fail you can often fail in a very unpredictable and spectacular way,” he said.