READY
AI agents are moving from demos into enterprise workflows, but existing evaluation methods remain insufficient for determining whether an agent is ready for real deployment. Existing benchmarks often measure task completion or capabilities such as reasoning, retrieval, and tool use. They do not fully answer the enterprise deployment question:
When is an agent reliable enough to be employed in a real organization, under what level of human oversight, and at what operating cost?
READY is a reliability-oriented evaluation and gating framework for enterprise AI agents. It is the first deployment gate an agent must pass before it can be trusted in enterprise use with bounded human oversight. Passing the gate does not mean the agent requires no supervision, it means the agent has reached a minimum reliability threshold, its remaining risks are measurable, and the required human-oversight policy can be specified.
Enterprise readiness is not merely task completion. It is grounded, auditable, policy-compliant, safe, and cost-effective task completion under realistic enterprise constraints.
The organizing principle
Enterprise teams adopt agents along a reliability gradient. They start where the blast radius is smallest, read-only agents that retrieve and synthesize information, then move toward agents that write back into systems of record, and finally deploy agents that hold open conversations with users to complete tasks.
Three quantities expand together as agents take on more autonomy:
- Autonomy surface, what the agent is allowed to do:
retrieve and synthesize → change system state → negotiate and complete a task with a live user. - Failure consequence,
misleading insight → incorrect system mutation → unsafe or policy-violating user interaction. - Oversight economics, the cost of evaluating and supervising the agent.
Each step up the gradient inherits the requirements of the last and adds new ones: write-action safety once an agent can change enterprise state, then open-interaction robustness, adversarial safety, and repeated-trial reliability once it converses with users.
Oversight economics
Oversight economics has two linked components. Offline evaluation cost is paid before deployment, benchmark construction, expert annotation, rubric design, repeated trials, red-teaming, trace review, and safety validation. It answers: is the agent good enough to enter deployment, and under what constraints? Online human-oversight cost is paid after deployment, review, approval before high-risk actions, escalation handling, auditing, correction, and monitoring. It answers: how much human supervision is required to use this agent safely in production?
READY links these two costs. Offline evaluation determines whether the agent passes the gate; online oversight determines the cost of employing the agent after it passes. As quality improves, the required human-oversight cost should decline, a measurable path from supervised deployment toward higher automation.
| Agent quality | Exam result | Required oversight | Deployment implication |
|---|---|---|---|
| Low quality | Fails gate | Not deployable | No production use |
| Minimum viable | Passes with high oversight | Heavy human review | Limited but safe deployment |
| Improving | Passes with moderate oversight | Confidence-gated review | Scalable supervised deployment |
| High quality | Passes with low oversight | Exception-based review | High automation ROI |
This trade-off is made quantitative on the Cost of Reliable AI page, an interactive model of total cost (token + human oversight) as a function of the reliability target.
Benchmark design principles
READY should be designed around deployment realism, whether an agent can operate under realistic enterprise constraints, not merely whether it can produce a plausible answer.
- Task realism. Tasks resemble real enterprise work, with realistic documents, tables, APIs, user instructions, policies, system states, and edge cases. Each task specifies a user goal, enterprise role, available tools, data sources, business policy, expected output or final state, risk level, required human-review behavior, and an evaluation rubric.
- Evidence and action traces. The benchmark evaluates not only final answers but intermediate evidence, tool calls, workflow actions, and escalation decisions.
- State-based evaluation. For write-back tasks, the final enterprise state is inspected directly, the final answer alone is not enough.
- Risk-weighted scoring. A minor formatting error is not scored the same as updating the wrong customer record or leaking private information. Errors range from low-severity (style, formatting) to critical (privacy leak, unauthorized action, irreversible harmful write).
- Human-in-the-loop evaluation. The benchmark measures whether the agent escalates when appropriate, whether the escalation artifact is useful, whether a human can efficiently verify the work, and whether the agent reduces total human effort.
The benchmark's output is not only a score. It determines whether the agent passes the deployment gate, what risk envelope it can operate within, and what human-oversight policy is required.
The core evaluation engines
These primitives recur across autonomy modes and benchmarks — the things an enterprise should build an evaluation engine for once, then reuse across its agent portfolio.
Grounded Claims
Does every important claim trace to retrieved evidence that genuinely supports it?
The core primitive for read-only insight agents, and relevant whenever an agent makes claims or justifies actions. Two failure modes matter most:
- Fabricated support, the agent cites a source that does not actually say what the agent claims.
- Unsupported synthesis, the agent draws a conclusion that no source backs.
Representative metrics: claim-level citation precision and recall, evidence sufficiency, unsupported-claim rate, contradicted-claim rate, source relevance, and correct handling of conflicting evidence.
Calibrated Escalation
Does the agent know what it does not know, and escalate rather than confabulate?
A reliable enterprise agent is evaluated not only on answering correctly, but on abstaining correctly. When it cannot verify a claim, faces genuine ambiguity, or lacks sufficient information, the correct behavior is to flag the case for human review rather than assert a conclusion.
Representative metrics: abstention recall (does it abstain when it should?), over-abstention (does it punt on answerable cases?), escalation-rationale quality, low-confidence detection, and ambiguity handling.
Decision-Grade Insight
Is the output complete, accurate, and decision-relevant against an expert rubric?
Research questions rarely have a single gold answer, so output is evaluated against expert rubrics rather than only exact-match answers.
Representative metrics: completeness, factual accuracy, depth of analysis, decision relevance, risk prioritization, clarity of recommendations, and explicit assumptions and limitations.
Active Evidence-Seeking
Does the agent recognize missing information and go get it, rather than paper over the gap?
A strong agent recognizes missing information and attempts to fill the gap, issuing the right look-ups rather than answering from the first records returned, and never covering missing evidence with plausible language.
Representative metrics: missing-information detection, correct follow-up question generation, appropriate tool or source selection, and avoidance of unsupported inference.
Auditable Traces
Can a human reconstruct what the agent did, why, and what evidence supported it?
Auditability is a deployment requirement, not just an observability feature. Are retrievals, tool calls, decisions, write operations, and escalations logged in a reviewable way?
Important trace elements: retrieved sources, claims and supporting evidence, tool calls, API inputs and outputs, state changes, human-review requests, escalation rationale, and error-recovery behavior.
Within Authority
Does the agent obey domain rules, role-based permissions, and standard operating procedure?
The agent must act within role-based authority and standard operating procedure, and obtain required confirmation or approval before committing high-risk actions.
Representative metrics: unauthorized-action rate, policy-violation rate, approval-bypass rate, correct distinction between draft / submit / approve / execute, and correct use of role permissions. A useful headline metric is task completion under policy compliance, crediting the agent only when it both finishes the task and breaks no rule.
Safe Writes
Did the agent perform the correct action on the correct object with the correct values?
The defining evaluation primitive for state-changing agents. When the agent changes system state, the benchmark must inspect the final system state, final text output is insufficient. Was the action authorized, reversible, and appropriate to the risk level?
Representative metrics: correct record selection, correct field update, correct workflow transition, duplicate-action rate, wrong-object mutation rate, final-state match against expected state, and reversibility / blast-radius scoring. Recovery and idempotency (handling stale data, failed API calls, retries) are part of this primitive and are weakly served by existing public benchmarks.
Attack-Resistant
Does the agent hold its policy when an input is crafted to break it?
Especially important for open, user-facing agents, but relevant wherever agents consume untrusted inputs. Can the agent resist prompt injection, jailbreaks, social engineering, and attempts to extract hidden instructions or sensitive data?
Representative metrics: jailbreak resistance, prompt-injection resistance, sensitive-information leakage rate, cross-user data-leakage rate, malicious-use refusal accuracy, hidden-policy leakage rate, and secure tool-use behavior. A critical point: defenses tuned for single-turn chatbots may not transfer to multi-step agents, so the guardrail must be evaluated in the agentic setting directly.
Repeatable Results
Does the agent succeed consistently across repeated trials of the same task?
Open conversation introduces variance. An agent that resolves a task correctly once but fails the same task on the next attempt is not deployable. A useful metric is pass^k, the probability that an agent succeeds on all k independent trials of the same task, which exposes reliability that a single pass rate hides.
Representative metrics: pass@1, pass^k, variance across repeated trials, sensitivity to user-simulator behavior, robustness to paraphrase, and robustness to conversation length.