A recent small scale study by Harvard Medical School indicates that software symptom-checkers still have a long way to go when identifying the reasons for medical conditions. This result seemed fairly consistent across different software so it is more a statement of the readiness of these systems rather than the accuracy of any one in particular.
The study concluded that, ‘Doctors are much better than symptom-checker programs at reaching a correct diagnosis, though humans are not perfect and might benefit from using algorithms to supplement their skills’. Because computers can’t perform a physical exam, the trial compared the performance of physicians based solely on the information provided. Even in this case, physicians performed markedly better than symptom-checker software, getting the diagnosis right 74% of the time versus only 34% for software systems. This poor showing for software systems leaves one to question whether these software systems should be in use at all as having an inaccurate diagnosis can be very harmful at times.
It also raises the question of how physicians would have performed if they had done a physical exam, which is the gold standard for any diagnosis. While the development of computer analytics continues to make great strides in many fields, there are clearly limits when the problem being solved is more nuanced than might first appear to be the case. Another limitation of an Artificial Intelligence application is the ability to question the value or accuracy of certain data, something that only experience and thinking outside of the box can accomplish.
Creating medical diagnostic software is much like creating self-driving cars in that there are layers of complexity that the human mind takes for granted but that software might miss. This is especially true with an environment that is constantly changing as in operating a motor vehicle or in medicine. That being said, both doctors and drivers make mistakes and likely both can benefit from computer systems that augment decision making instead of replacing humans.
That being said, it is hard to imagine the replacement of the physical exam and questioning of a trained physician, which can uncover additional symptoms, eliminate others, and make sense out of contradictory data. Something as simple as a rash might mean very little to a computer system but point the trained physician to an underlying cause.
As a technology provider, it might seem odd that we would be pointing to the limitations of software in clinical decision support but experience has led us to conclude that the human mind has abilities that computers can’t come close to, at least in their present incarnations. While the promise of Artificial Intelligence has been around for a long time, the applications of it remain elusive. At least for now, we will go see our doctor if we are having medical problems (after using a symptom checker).