Column
How Do You Actually Measure Medical AI Capability? HealthAgentBench and the 42% Gap
The best LLM agents today succeed on only about 42% of realistic healthcare tasks in HealthAgentBench — a 54-task benchmark across 7 clinical categories. Here's how healthcare organizations can use this benchmark to make better AI procurement decisions.
Hello. This is Keito Inoshita from Affectosphere Group.
“We want to adopt medical AI, but we can’t objectively evaluate whether it actually works in our environment.”
This is one of the most common concerns from hospital DX teams and medical device manufacturers. Vendor demos look impressive. But the gap between demo performance and real clinical performance is often invisible — until after deployment.
Liu et al. (arXiv:2606.31179) built HealthAgentBench precisely to address this problem. It evaluates LLM agents across 54 realistic healthcare tasks in 7 categories. The finding: even the best-performing system achieves a success rate of approximately 42%.
Today’s 3 Points
- HealthAgentBench covers 7 clinically realistic task categories — including EHR analysis, medical image interpretation, and multi-step clinical decision-making — providing an objective comparison baseline beyond vendor demos.
- The best available AI agents succeed on only about 42% of tasks, confirming a significant capability gap for autonomous clinical deployment.
- Tasks combining large search spaces with complex multi-step reasoning are especially difficult — and this category-level performance variation is the most useful signal for procurement decisions.
① Why a Unified Benchmark Matters
Medical AI evaluation has historically been fragmented.
“Diagnostic accuracy of X%.” “Detection sensitivity of Y% for condition Z.” Individual metrics exist, but they don’t translate into how a system performs across the varied, compound tasks of real clinical work.
HealthAgentBench addresses this by designing its 7 categories around real hospital workflows — EHR analysis, medical imaging, multi-step decision support, and more. This isn’t a single-capability test. It’s a simulation of the compound scenarios that actually occur in clinical settings.
For procurement teams, this matters. “Where does this vendor’s system rank on HealthAgentBench, and in which categories?” is a question that replaces demo impressions with data.
② How to Read the 42% Success Rate
At first glance, a 42% success rate for the best available system sounds alarming.
But the interpretation depends on context.
This is the success rate for autonomous, end-to-end task completion — no human assistance, no double-checking. A system used as a decision-support tool alongside a clinician, rather than as an autonomous agent, operates under different standards.
The 42% also belongs to the top-performing system. Many deployed medical AI systems may score lower. Cross-referencing vendor claims against HealthAgentBench scores becomes a practical way to surface overestimation.
And performance varies significantly by category. The aggregate 42% masks wide variation across the 54 tasks. That distribution — which categories perform well, which lag — is the most actionable signal for deployment planning.
③ Mapping “Which Categories Are Ready Now”
The benchmark highlights that tasks combining large exploration spaces with multi-step reasoning are particularly difficult.
This implies a corollary: tasks with limited exploration space and clear reasoning steps may already be within reach of current systems.
For hospital DX teams evaluating a medical AI system, this “performance map by task category” is highly practical.
Routine anomaly detection in structured lab values, for instance, may yield high success rates with current systems. Multi-step treatment planning that integrates multiple heterogeneous clinical contexts likely still requires human involvement. The distinction isn’t always obvious from demos — HealthAgentBench makes it measurable.
Implementation: Embedding HealthAgentBench in Procurement
For hospital DX teams, healthcare AI startups, and healthtech investors, here is a practical way to operationalize this benchmark.
Start by mapping your target use case to HealthAgentBench’s 7 categories. Is the problem EHR-related? Image interpretation? Multi-step decision support? Identifying the category first focuses the evaluation.
Then require vendors to provide HealthAgentBench scores for their systems. Vendors without benchmark results should be asked to run equivalent evaluations. This alone filters out systems that look good in demos but lack measurable performance baselines.
Set a performance gate: before progressing to pilot, require a minimum success rate threshold in the relevant benchmark categories. This converts subjective vendor evaluation into a structured selection process.
For healthtech investors and medical device manufacturers, checking which HealthAgentBench categories a prospective investment or partnership actually excels in helps surface whether the claimed capability is genuine or overestimated.
42% Is a Starting Point, Not a Verdict
The 42% success rate is not an indictment of medical AI. It’s an honest map of the current landscape.
It shows where capability gaps exist, pointing researchers toward what to solve next, and telling implementers what not to automate yet. This kind of honest accounting is exactly what the field needs — especially in a domain where overestimation affects patient safety.
As HealthAgentBench — or benchmarks like it — becomes standard infrastructure for medical AI evaluation, the conversation shifts from “this looks impressive” to “here’s what it can and cannot do, measured.”
The people who benefit most from that shift are patients.
Reference
- Qianchu Liu et al. (2026). HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments. arXiv preprint.
* This article was written in part with AI assistance and may contain inaccuracies.