--- title: "Grab Bench: Evaluating AI on Grab-shaped production work" url: https://daily.dev/posts/grab-bench-evaluating-ai-on-grab-shaped-production-work-fvaxlxyds source_url: https://engineering.grab.com/grab-bench-evaluating-ai type: article source: "Grab Tech Blog" published: 2026-08-12T01:00:55.382Z updated: 2026-08-12T01:01:27.663Z tags: ["llm", "ai-agents"] reading_time: 10 upvotes: 5 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Grab Bench: Evaluating AI on Grab-shaped production work **[Grab Tech Blog](https://daily.dev/sources/grab)** · 10 min read · 5 upvotes · 1 comments ## Summary Grab's engineering team built Grab Bench, an internal AI evaluation framework designed to catch subtle failures that public leaderboards miss. Unlike generic benchmarks, it tests models on Grab-shaped production tasks — SQL generation, tool calling, passenger-profile reasoning, and agentic coding — using synthetic but realistic data. Key design choices include deterministic scoring over LLM-judge confidence, explicit shortcut detection (e.g., cite-everything baselines, visible-test-only coding agents), and a split between teaching artifacts and hidden certification cases. The framework records row-level failure tags so engineers can debug specific failure modes rather than just comparing aggregate scores. A key lesson: more reasoning isn't universally better, and prompt/model settings must be tuned per task rather than globally. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you prevent AI benchmarks from being gamed by shortcuts like citing all evidence or only passing visible tests? Shortcut baselines must be built into the benchmark itself and made to fail. Running empty-output, cite-all-evidence, schema-only, and no-op agent baselines as explicit checks verifies the scorer penalizes those behaviors. For coding agents, hidden workspace tests, anti-gaming checks, and implausible difficulty curve detection block visible-test-only overfitting. If a shortcut baseline can pass, the benchmark is not measuring the intended contract. _Teams hardening their own AI evals track patterns like these on daily.dev._ ### When should I use deterministic scoring vs an LLM judge for evaluating AI model outputs? Use deterministic scoring whenever the task has a verifiable contract: ontology values, evidence IDs, canonical tool names and parameters, or test pass/fail results. Reserve LLM judges for open-ended tasks like SQL generation where correctness depends on business intent and query shape. Deterministic scoring produces an audit trail that shows exactly which constraint was violated, rather than whether an answer 'seems good'. _Developers choosing between eval strategies for their AI pipelines find relevant comparisons on daily.dev._ ### How do you structure an internal AI benchmark so it stays reproducible without letting teams overfit to the test cases? Split artifacts into two categories: teaching artifacts (task contracts, scorer logic, examples, baselines, canaries) that are openly inspectable, and certification artifacts (hidden splits, seeds, raw outputs) kept behind access controls. Before trusting a comparison run, gate checks verify that oracle solutions pass, weak baselines fail, and canaries catch harness regressions. Shared examples teach the method; hidden cases check generalization. _Engineers building internal eval infrastructure for AI systems share approaches like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > Building the benchmark around your own production-shaped tasks is the part public leaderboards cannot give you, and the shortcut detection, catching cite-everything or visible-test-only agents, is a detail most internal evals miss. The finding that more reasoning is not universally better matches what I saw building my own eval stack: tuning per task beat a global setting every time. I wrote up what I'd buy versus build after doing it from scratch here:... ## Similar posts on daily.dev - [Evaluating AI Agents in Practice: Benchmarks, Frameworks, and Lessons Learned](https://daily.dev/posts/evaluating-ai-agents-in-practice-benchmarks-frameworks-and-lessons-learned-tohpewvfd) · InfoQ · 2 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/grab-bench-evaluating-ai-on-grab-shaped-production-work-fvaxlxyds)