PSEBench

What PSEBench measures

Patient safety event (PSE) triage determines whether a clinical event is reportable under jurisdiction-specific policy, a high-stakes task usually performed manually by patient-safety experts. PSEBench evaluates whether an LLM can support this workflow through policy-grounded reasoning rather than mere classification: grounding decisions in legal evidence, identifying missing decision-critical facts, and abstaining when facts or policy criteria are ambiguous.

It is the first comprehensive benchmark for PSE reportability triage that jointly evaluates policy-grounded verdicts, legal-evidence grounding, targeted information-seeking over missing facts, and explicit recognition of policy-level uncertainty.

Construction: clause cards + closed-loop verification

The methodology centers on the clause card, a structured decision specification that factorizes opaque regulatory guidance into auditable variables: reportability criteria, boundary conditions, required facts, and expected verdicts. A clause card is a tuple (s, B, E, ρ, y, G, Φ), event type, boundary conditions, basic event elements, the mapping from conditions to elements, the intended verdict, the governing legal basis, and instantiation constraints.

Combining clause cards with diverse real-world clinical anchor materials, an LLM generates event narratives while automated verifiers check both the structured facts and the generated text against the clause card. This closed-loop process yields by-construction ground truth, narratives that faithfully inherit the triage verdict, legal basis, and decision-critical facts of their clause card, without expensive post-hoc annotation. It also enables systematic construction of missing-information and uncertain variants.

Instantiated on MN29, the pipeline produced 5,074 cases: 3,455 complete, 1,362 missing-information, and 257 uncertain. Two patient-safety experts validated a sample of 90 cases, strongly concurring with the pipeline's by-construction ground truth.

Headline finding

High verdict accuracy frequently masks critical failures. Strong models exceed 92% verdict accuracy (M1), yet their evidence grounding (M3) and boundary-condition hit rate (M4) lag well behind, they often guess the right verdict for the wrong or hallucinated reasons. The starkest gap is uncertainty awareness (M7): only the strongest models meaningfully escalate gray-zone events, while most, including all medical-specialty models, collapse toward a confident "Reportable" verdict. Simple classification accuracy is an unreliable proxy for real-world triage readiness.