Artificial intelligence (AI) may be better than physicians at making clinical decisions in virtual urgent care settings - at least when it comes to common medical conditions, according to a new study by Tel Aviv University.
The study, conducted by Prof. Dan Zeltzer of the Berglas School of Economics at Tel Aviv University and published this week in the Annals of Internal Medicine, evaluated the performance of AI in Cedars-Sinai Connect, a virtual urgent care clinic in Los Angeles that operates in collaboration with Israeli health tech start-up K Health. The findings were also presented at the annual conference of the American College of Physicians (ACP).
Researchers analyzed 461 adult patient visits between June and July 2024, focusing on relatively common complaints such as respiratory, urinary, eye, vaginal, and dental symptoms. Each visit began with an AI-powered intake system, which generated diagnostic and treatment recommendations - including prescriptions, lab tests, and referrals - before the patient’s video consultation with a physician.
AI scores higher than physicians in most cases
According to the study, the AI system’s recommendations were rated as “optimal” in 77% of cases, compared to 67% of the decisions made by the treating physicians. The AI recommendations were judged better than the physician’s in 21% of visits, while physicians outperformed the AI in 11% of cases. In the remaining 68%, both recommendations received the same rating.
The ratings were based on evaluations conducted by a panel of four experienced physicians, each with over 10 years of clinical practice. They assessed the quality of recommendations on a four-point scale: optimal, reasonable, inadequate, or potentially harmful.
The panel also found that AI made fewer potentially harmful recommendations - 2.8% compared to 4.6% for physicians.
Guideline adherence and data sensitivity
“The AI consistently adhered to clinical guidelines, avoided unnecessary antibiotics, and identified red flags in patient histories,” said Prof. Zeltzer. “For example, it correctly flagged eye pain in a contact lens wearer as a potential sign of infection—something that could be easily overlooked in a fast-paced virtual setting.”
The algorithm, trained on millions of anonymized medical records, only provides recommendations when it reaches a high confidence threshold. In around 20% of cases, the AI declined to offer guidance due to insufficient certainty.
Human judgment still has its place
While the AI demonstrated notable strengths, the study also highlighted areas where physicians maintain a distinct advantage. Human doctors were better able to interpret nuance, respond to new information during live consultations, and assess the patient’s condition in real time.
“In cases where a patient reported shortness of breath due to COVID-19, for instance, a physician might determine it was mild congestion and not require emergency care - something the AI, based solely on structured inputs, might misclassify,” Zeltzer explained.
Study limitations and future applications
One limitation of the study was that it did not track whether physicians reviewed the AI’s recommendations before making their final decisions, making it difficult to assess the AI's influence on physician behavior. The study focused solely on comparing the quality of the recommendations themselves.
“This is one of the first studies to evaluate AI in real-world medical practice - not based on textbooks or test cases, but on actual patient visits,” Zeltzer said. “The findings are promising and suggest that AI could serve as an effective clinical decision-support tool, especially in virtual care environments.”
Prof. Zeltzer added that while AI is unlikely to replace physicians, it could help reduce diagnostic errors, draw attention to critical data, and speed up clinical workflows. “The future of healthcare will likely be a partnership between medical professionals and AI systems,” he said.