Grab's engineering team describes building Jarvis Pro, an internal AI assistant for account managers that routes user questions to a specific task type before generating an answer, rather than generating a fluent response directly. The design addresses a failure mode where an early prototype produced polished but operationally wrong recommendations by missing that a merchant's real problem was paused outlets and fulfilment issues, not demand. The system pipeline runs route classification, memory/access checks, context loading, and metric reconciliation before generation, deliberately trading a slower first token for safer answers. Offline evaluation on 351 routing prompts hit 99.4% expected-route match, and a 150-case portfolio/brand answer-quality subset improved from 78.5 to 91.0 on a 0-100 rubric; these are launch-readiness offline metrics, not proof of business impact.
Table of contents
IntroductionWhy dashboards were easierThe route as a contractMemory had to stay narrowReconcile the metric before blaming the modelHow we evaluated itThe lesson we would reuseJoin usQuestions this post answers
Why did Grab add a routing step before letting an LLM answer account manager questions in Jarvis Pro?
An early prototype produced a fluent but wrong recommendation because it missed that a merchant's real issue was paused outlets and fulfilment failure, not demand, and pushing promotions would have made things worse. Routing first lets the system classify the task, load only relevant context, apply guardrails, and avoid jumping straight to generic recommendations like discounts. See how daily.dev surfaces engineering writeups like this on building safer LLM agent pipelines.
What evaluation results did Grab report for the Jarvis Pro routing system before launch?
Routing matched the expected safe route in 99.4% of 351 labelled offline evaluation prompts, and a 150-case portfolio and brand answer-quality subset improved from an average score of 78.5 to 91.0 on a 0-100 rubric covering template fit, metric use, diagnosis, and guardrail compliance. These are offline launch-readiness figures, not measures of commercial impact. Compare LLM evaluation approaches like this one on daily.dev when benchmarking your own agent pipeline.