CliniCARE-Bench

What CliniCARE-Bench measures

Foundation models now match or exceed clinicians on medical-knowledge exams, yet exam performance is a poor proxy for what clinical deployment demands. A nurse at change of shift, a quality officer auditing sepsis-bundle compliance, or a nephrologist assessing a returning patient does not answer a self-contained multiple-choice question, each conducts a small research project against a heterogeneous, longitudinal, uncurated electronic health record.

CliniCARE-Bench (Clinical Calibrated Audit of Medical Reasoning in EHR) is a deployment-oriented benchmark for selective autonomy in retrospective clinical audit. To our knowledge it is the first clinical agent environment to support end-to-end auditing of an agent's reasoning process, retrieval, evidence grounding, policy use, and the decision trace that links them, scored end to end rather than on the final answer alone.

The environment

The suite is 25 clinician-authored, evidence-intensive scenarios, each instantiated with 30 patient cases (750 in total), over the de-identified MIMIC-IV database (v3.1 core, Note v2.2, ED v2.2, four modules across 41 tables). Rather than receiving a preassembled context, the agent requests specific slices of one patient's chart through governed, clinician-verifiable tools (patient/encounter discovery, notes, labs, vitals, medications, microbiology, procedures, imaging, orders), with a logged read-only SQL fallback and a sandboxed Python runtime for derived quantities. Every retrieval is a discrete, logged call, so a verdict can be traced back to the evidence it rests on.

Many tasks are not purely factual: they ask whether documented care was concordant with a governing policy, e.g. KDIGO, CMS SEP-1, AABB, or AHA/ACC guidance, supplied as a searchable corpus the agent must itself identify and apply.

A calibrated four-way verdict

Every case forces one of four verdicts, Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous, separating cases an agent can resolve from those limited by missing evidence versus genuine residual ambiguity. Reference labels come from a four-round Clinical Board review: 75 cases (10%) are clinician-validated, and the remainder use calibrated, model-assisted consensus labels, reported separately.

Scored on more than accuracy

Beyond verdict accuracy (with a macro-F1 across the four classes), CliniCARE-Bench scores insight grounding (citation precision + evidence recall), abstention behavior (over-commitment vs. over-abstention, and whether the two Indeterminate classes are told apart), a weighted MUST-DO / MUST-NOT process rubric (a single critical violation, e.g. fabricating a lab value, fails the task), policy citation and compliance, and cost. A trap-adjusted accuracy credits a verdict only when the label is right and no prohibited shortcut was taken.

Findings

Three patterns hold. Accuracy clusters in a narrow band, frontier models (Opus 4.8, GPT-5.5) do not pull away from the field, and the cheapest system is competitive with the most expensive. Under-abstention dominates the error profile: recall on Yes / No is high, but recall on the Indeterminate classes is far lower and near zero on Medically Ambiguous, even though agents retrieve ample evidence before committing, an elicitation gap, not a retrieval one. And trap-adjusted accuracy is markedly lower than raw accuracy (by roughly 6–13 points): a substantial share of correct verdicts are reached through a prohibited shortcut, which reorders the leaderboard.