Policy-Driven Task Environments for Enterprise Agent Evaluation

Client context

An enterprise software company shipping a customer-facing support agent needed to know — before launch — where the agent would break, ignore policy, or fail to escalate to a human.

Challenge

Internal testing covered happy paths only. There was no systematic way to measure policy adherence, recovery from tool failures, or long-horizon reliability, and every model update meant days of manual re-testing.

Data strategy

We designed executable task environments mirroring the client's production stack — ticketing, CRM, refund APIs — each parameterized with policy constraints and injected failure conditions. Expert operators produced golden trajectories and deliberately-broken runs to populate both sides of the evaluation.

Workflow

1. Environment design — replica tool APIs with controllable failure injection
2. Task design — scenario matrix across policy, tooling, and edge conditions
3. Expert execution — golden trajectories plus adversarial runs
4. Failure analysis — clustering observed failures into an actionable taxonomy
5. Evaluation loop — automated regression harness for every model release

Quality controls

Trajectories were replay-validated before acceptance. Policy adherence judgments used dual review with documented disagreement resolution, and the failure taxonomy was re-calibrated after each evaluation round.

Outcome

The final suite of 120 environments and a 14-mode failure taxonomy became the client's release gate. Regression evaluation that previously took days of manual QA now runs per release candidate, and escalation accuracy — previously unmeasured — became a tracked launch metric.