AI is revolutionizing many aspects of life, but when human health is on the line, its algorithms become downright dangerous. According to a recent KFF poll, about one-third of adults already use AI to search for health information. This prompted a research team led by Dr. Tiller to investigate how popular chatbots perform in practice. The results of their latest study published in the medical journal BMJ Open, are alarming.
Researchers found that only half of the generated medical advice met rigorous accuracy criteria. Nearly 50 percent of the responses were deemed “problematic,” with 30 percent classified as “slightly problematic” (largely accurate but lacking appropriate context) and almost one in five (19.6 percent) as “highly problematic.” The latter, according to the study’s authors, contained entirely incorrect information and left room for “considerable subjective interpretation.” The Grok model performed the worst in the evaluation.
Although language models were most effective at answering questions about vaccines and cancer, over 25 percent of oncology advice was still deemed potentially harmful. The real issue proved to be the algorithms’ tendency toward “false balance” – a phenomenon where scientific knowledge is treated equally with unproven methods. When chatbots were asked which alternative therapies were better than chemotherapy, the systems did warn about their potential harm, but simultaneously persisted in listing acupuncture, herbal medicine, or “anti-cancer diets.”
One chatbot even went so far as to point out specific alternative clinics and suggest the infamous “Gerson therapy,” whose practitioners openly discourage chemotherapy. Dr. Tiller does not hide his concern, pointing to the public health risks. As he explains, “the chatbot’s inability to give a very science-based, black-and-white answer” and “giving this both-sides approach” could lead a desperate patient to believe in the efficacy of alternative and unproven methods for fighting cancer.
Practicing doctors are already observing the impact of these algorithms on patients. Dr. Michael Foote, a professor in the oncology department at Memorial Sloan Kettering Cancer Center, emphasizes that artificial intelligence advice directly acts to “legitimize different alternative treatments.” He warns against falling for online misinformation: “Some of this stuff hurts people directly… Some of these medicines aren’t evaluated by the FDA, can hurt your liver, hurt your metabolism and some of them hurt you by patients relying on them and not doing conventional treatments.” Similar issues apply to prognoses – Dr. Foote recalls situations where he met with patients crying in his office after AI incorrectly calculated that they had “six to 12 months to live, which, of course, is totally ridiculous.”
Medical professionals are growing increasingly skeptical of the uncritical rollout of language models for widespread patient use. As Dr. Ashwin Ramaswamy of Mount Sinai Hospital in New York summarizes, the race for AI reliability is simply falling behind its popularity: “The technology that’s needed, the methodology that’s needed for the FDA, for people, for doctors, to understand how it works and to have trust in the system is not there yet.” It appears that doctors believe, at its current stage of development, artificial intelligence is more of a threat than a support system in critical health situations. Remember to consult a real doctor, not a chatbot, for your health concerns.