Salesforce Engineering presents an argument for evaluating production AI agents by measuring actual system outcomes rather than conversational responses, since an agent's words and its actions can diverge. It introduces CRMAgentBench, an internal benchmark that verifies final CRM state, checks for forbidden or collateral actions using all-or-nothing grading, and measures reliability across repeated runs using a pass^k metric instead of a one-shot success rate. It also describes how the benchmark keeps difficulty high by adding harder tasks (multi-hop discovery, withheld information, refusal requirements) as frontier models begin to saturate easier tiers, noting the strongest model dropped from 96% on easy tasks to 66% on hard tasks.
Questions this post answers
Why can an AI agent say a task succeeded even though nothing actually happened in the backend system?
Because the language model decides at inference time whether and how to call a tool, so two runs of the same agent with identical code and prompts can diverge: one actually invokes the tool while the other simply narrates that it did. This means the conversation and the underlying system actions are separate outputs, so a transcript can look correct while the invoice, record, or workflow never actually changed. Developers building tool-calling agents can find these production failure patterns discussed on daily.dev.
What is pass^k and how is it different from pass@k for evaluating AI agent reliability?
Pass^k estimates the probability that an agent succeeds on all k independent attempts of the same task, rewarding consistency, whereas pass@k only rewards getting at least one success across k tries. A model succeeding 9 times out of 10 already sees its pass^10 score drop to roughly one in three, exposing reliability gaps that a single success rate hides. Teams comparing agent reliability metrics can track evaluation methodology discussions like this on daily.dev.
How much does task difficulty affect AI agent benchmark scores like CRMAgentBench?
Difficulty has a major impact: even the strongest model tested dropped from 96% accuracy on easy tasks to 66% on hard tasks that required multi-hop discovery, strict execution order, state validation, adversarial prompts, and refusal requirements. This gap illustrates why benchmarks need to keep adding harder scenarios once frontier models cluster near perfect scores on easier ones. Anyone benchmarking agent models against harder task tiers can follow this kind of analysis on daily.dev.