--- title: "How we teach LLMs to write BadgerQL" url: https://daily.dev/posts/how-we-teach-llms-to-write-badgerql-nk52e8sgh source_url: https://www.honeybadger.io/blog/teaching-llms-badgerql type: article source: "Honeybadger" published: 2026-08-18T20:33:31.567Z updated: 2026-08-18T20:33:58.344Z tags: ["llm", "testing", "prompt-engineering"] reading_time: 8 upvotes: 4 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How we teach LLMs to write BadgerQL **[Honeybadger](https://daily.dev/sources/honeybadger)** · 8 min read · 4 upvotes · 1 comments ## Summary Honeybadger describes how it built natural-language-to-query translation for its Error search and Insights (BadgerQL) features. Since BQL is a proprietary query language LLMs aren't trained on, the team relied on a system prompt rather than fine-tuning. To measure whether prompt changes actually helped, they extended their existing integration test suite with 'llm_case' blocks that send prompts to an LLM, run the generated BQL, and grade results by output rather than by comparing query text (since many different BQL queries can be equivalent). Starting from 0% pass rate with no system prompt, a hand-written prompt reached 71%, and attaching LLM guidance and examples directly to the data structures defining BQL functions (marking common ones as 'core') pushed the score to 88% against Claude Haiku 4.5. The resulting system prompt is published in their llms.txt for external agents and their hosted MCP server. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you evaluate whether an LLM correctly translated natural language into a custom query language when multiple query syntaxes can be equivalent? Grade the results the generated query returns rather than comparing the query text itself. Honeybadger extended its integration test suite with 'llm_case' blocks containing a prompt; the LLM's generated BadgerQL is executed and passes if it returns the same expected events as the reference query, regardless of which valid syntax it used. _daily.dev surfaces engineering writeups like this for teams building LLM-to-DSL translation pipelines._ ### How much does a well-structured system prompt improve LLM accuracy at generating a proprietary query language like BadgerQL? Against Claude Haiku 4.5, accuracy went from 0% with no system prompt, to about 71% with a hand-written prompt, up to 88% after attaching structured examples and regex-like phrase patterns directly to the data structures defining each BadgerQL function, and marking common functions as 'core' so they rank higher in the prompt. _developers tuning prompts for domain-specific languages can track this kind of case study on daily.dev._ ### What is BadgerQL and how is it different from Honeybadger's Error search syntax? BadgerQL (BQL) is Honeybadger's own query language for Insights event data, designed to feel familiar to CloudWatch Insights and Splunk users, translated internally to ClickHouse SQL by a service called Opticon. Error search instead uses a simpler, constrained token syntax similar to Solr or basic Elasticsearch queries. _daily.dev helps engineers evaluating custom query languages compare approaches like this one._ ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > I'd add adversarial fixtures where an overbroad filter returns the expected rows for the wrong reason. Result-only grading can pass when the dataset lacks the counterexample. ## Similar posts on daily.dev - [Local LLMs perform so much better when you teach them to ask before they answer](https://daily.dev/posts/local-llms-perform-so-much-better-when-you-teach-them-to-ask-before-they-answer-2bdnepe2i) · XDA Developers · 0 upvotes · 0 comments - [Your local LLM feels weak because you're treating it like a search engine](https://daily.dev/posts/your-local-llm-feels-weak-because-you-re-treating-it-like-a-search-engine-2vhydegdq) · XDA Developers · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/how-we-teach-llms-to-write-badgerql-nk52e8sgh)