Nearly three years after OpenAI launched ChatGPT and ushered in a global frenzy over large language models, chatbots are weaving themselves into seemingly every part of society in China, the U.S., and beyond. For patients like my mom, who feel they don’t get the time or care they need from their health care systems, these chatbots have become a trusted alternative. AI is being shaped into virtual physicians, mental-health therapists, and robot companions for the elderly. For the sick, the anxious, the isolated, and many other vulnerable people who may lack medical resources and attention, AI’s vast knowledge base, coupled with its affirming and empathetic tone, can make the bots feel like wise and comforting partners. Unlike spouses, children, friends, or neighbors, chatbots are always available. They always respond.
Entrepreneurs, venture capitalists, and even some doctors are now pitching AI as a salve for overburdened health care systems and a stand-in for absent or exhausted caregivers. Ethicists, clinicians, and researchers are meanwhile warning of the risks in outsourcing care to machines. After all, hallucinations and biases in AI systems are prevalent. Lives could be at stake.
Over the course of months, my mom became increasingly smitten with her new AI doctor. “DeepSeek is more humane,” my mother told me in May. “Doctors are more like machines.”
[...]
China’s health care system is rife with severe inequalities. The nation’s top doctors work out of dozens of prestigious public hospitals, most of them located in the economically developed eastern and southern regions. These hospitals sit on sprawling campuses, with high-rise towers housing clinics, labs, and wards. The largest facilities have thousands of beds. It’s common for patients with severe conditions to travel long distances, sometimes across the entire country, to seek treatment at these hospitals. Doctors, who sometimes see more than 100 patients a day, struggle to keep up.
[...]
My mother’s reliance on DeepSeek grew over the months. Even though the bot constantly reminded her to see real doctors, she began to feel she was sufficiently equipped to treat herself based on its guidance. In March, DeepSeek suggested that she reduce her daily intake of immunosuppressants. She did. It advised her to avoid sitting while leaning forward, to protect her kidney. She sat straighter. Then, it recommended lotus root starch and green tea extract. She bought them both.
[...]
With her consent, I shared excerpts of her conversations with DeepSeek with two U.S.-based nephrologists.
DeepSeek’s answers, according to the doctors, were full of errors. Dr. Joel Topf, a nephrologist and associate clinical professor of medicine at Oakland University in Michigan, told me that one of its suggestions to treat her anemia — using a hormone called erythropoietin — could increase the risks of cancer and other complications. Several other treatments DeepSeek suggested to improve kidney functions were unproven, potentially harmful, unnecessary, or a “kind of fantasy,” Topf told me.
[...]
Dr. Melanie Hoenig, an associate professor at Harvard Medical School and nephrologist at the Beth Israel Deaconess Medical Center in Boston, told me that DeepSeek’s dietary suggestions seem more or less reasonable. But she said DeepSeek had suggested completely wrong blood tests and mixed up my mother’s original diagnosis with another very rare kidney disease.
“It is sort of gibberish, frankly,” Hoenig said. “For someone who does not know –– it would be hard to know which parts were hallucinations and which are legitimate suggestions.”
Researchers have found that chatbots’ competence on medical exams do not necessarily translate into the real world. In exam questions, symptoms are clearly laid out. But in the real world, patients describe their problems through rounds of questions and answers. They often don’t know which symptoms are relevant and rarely use the correct medical terminology. Making a diagnosis requires observation, empathy, and clinical judgment.
[...]
As my mother bonded with DeepSeek, health care providers across China embraced large language models.
Since the release of DeepSeek R1 in January, hundreds of hospitals have incorporated the model into their processes. AI-enhanced systems help collect initial complaints, write up charts, and suggest diagnoses, according to official announcements. Partnering with tech companies, large hospitals use patient data to train their own specialized models. One hospital in Sichuan province introduced “DeepJoint,” a model for orthopaedics that analyzes CT or MRI scans to generate surgical plans. A hospital in Beijing developed “Stone Chat AI,” which answers patients’ questions about urinary tract stones.
[...]
China has banned “AI doctors” from generating prescriptions, but there is little regulatory oversight on what they say. Companies are left to make their own ethical decisions. Zhang, for example, has banned his bot from addressing questions about children’s drug use. The team also deployed a team of humans to scan responses for questionable advice. Zhang said he was overall confident with the bot’s performance. “There’s no correct answer when it comes to medicine,” Zhang said. “It’s all about how much it’s able to help the users.”