Your Phone May Be Smart But Your Doctor Still Knows More Than An App

Human doctors are outperforming apps, algorithms and websites in diagnostic tests. For the time being. JL

Karen Kaplan reports in the Los Angeles Times:

Human physicians outperformed a collection of 23 symptom-checker apps and websites by a margin of more than 2 to 1,
If you’re feeling sick and you want to know what’s wrong with you, there’s an app for that. But the diagnosis won’t be as accurate as the one you’d get from a doctor — not by a long shot.
In a head-to-head comparison, real human physicians outperformed a collection of 23 symptom-checker apps and websites by a margin of more than 2 to 1, according to a report published Monday in the journal JAMA Internal Medicine.
Even when the contestants got three chances to figure out what ailed a hypothetical patient, the diagnostic software lagged far behind actual doctors. Indeed, the apps and websites suggested the right diagnosis only slightly more than half of the time, the report says.
The research team — from Harvard Medical School, Brigham & Women’s Hospital in Boston and the Human Diagnosis Project in Washington, D.C. — asked 234 physicians to read through a selection of 45 “clinical vignettes” to see how theywould handle these hypothetical patients. Each vignette included the medical history of the “patient” but no results from a physical exam, blood test or other kind of lab work.
Most of the doctors were trained in internal medicine, though the group included some pediatricians and family practice physicians too. About half of them were in residency or fellowship, so their training was not yet complete.
Even so, of the 1,105 vignettes they considered, they listed the correct diagnosis first 72% of the time, according to the study.
The 23 symptom checkers evaluated a total of 770 vignettes in an earlier study by some of the same researchers. The apps and websites (including several from professional medical organizations, such as the American Academy of Physicians, the American Academy of Pediatrics and the Dutch College of General Practitioners) listed the correct diagnosis first just 34% of the time.
Both the doctors and the computer programs were able to include more than one ailment in their differential diagnosis. So the researchers also compared how often the correct diagnosis was among the top three responses.
For the doctors, that happened 84% of the time. For the symptom checkers, it was 51% of the time.
Though the humans trounced the computers across the board, there were situations in which doctors did a particularly good job of naming the correct diagnosis first. For instance, their margin in cases with common conditions was 70% to 38%. In cases with uncommon conditions, it grew to 76% to 28%.
The seriousness of the malady made a difference too. In cases with low acuity, doctors bested software by 65% to 41%. But in cases with high acuity, that gap widened to 79% to 24%.
“Physicians vastly outperformed computer algorithms in diagnostic accuracy,” the researchers concluded. Full disclosure: Three of the study authors are doctors, and none are apps.