Honeybadger describes how it built natural-language-to-query translation for its Error search and Insights (BadgerQL) features. Since BQL is a proprietary query language LLMs aren't trained on, the team relied on a system prompt rather than fine-tuning. To measure whether prompt changes actually helped, they extended their existing integration test suite with 'llm_case' blocks that send prompts to an LLM, run the generated BQL, and grade results by output rather than by comparing query text (since many different BQL queries can be equivalent). Starting from 0% pass rate with no system prompt, a hand-written prompt reached 71%, and attaching LLM guidance and examples directly to the data structures defining BQL functions (marking common ones as 'core') pushed the score to 88% against Claude Haiku 4.5. The resulting system prompt is published in their llms.txt for external agents and their hosted MCP server.
Table of contents
The system promptTest results, not query textImproving the promptSharing the contextConclusionQuestions this post answers
How do you evaluate whether an LLM correctly translated natural language into a custom query language when multiple query syntaxes can be equivalent?
Grade the results the generated query returns rather than comparing the query text itself. Honeybadger extended its integration test suite with 'llm_case' blocks containing a prompt; the LLM's generated BadgerQL is executed and passes if it returns the same expected events as the reference query, regardless of which valid syntax it used. daily.dev surfaces engineering writeups like this for teams building LLM-to-DSL translation pipelines.
How much does a well-structured system prompt improve LLM accuracy at generating a proprietary query language like BadgerQL?
Against Claude Haiku 4.5, accuracy went from 0% with no system prompt, to about 71% with a hand-written prompt, up to 88% after attaching structured examples and regex-like phrase patterns directly to the data structures defining each BadgerQL function, and marking common functions as 'core' so they rank higher in the prompt. developers tuning prompts for domain-specific languages can track this kind of case study on daily.dev.
What is BadgerQL and how is it different from Honeybadger's Error search syntax?
BadgerQL (BQL) is Honeybadger's own query language for Insights event data, designed to feel familiar to CloudWatch Insights and Splunk users, translated internally to ClickHouse SQL by a service called Opticon. Error search instead uses a simpler, constrained token syntax similar to Solr or basic Elasticsearch queries. daily.dev helps engineers evaluating custom query languages compare approaches like this one.
3.1K Impressions1 Comment