More than 100x Faster & Cheaper LLM-Powered SQL Queries with Proxy Models
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Google Cloud researchers present proxy models, a technique to dramatically reduce the cost and latency of LLM-powered SQL functions in databases like BigQuery and AlloyDB. Instead of calling an LLM for every row, proxy models use lightweight classifiers (e.g., logistic regression) trained on Gemini embeddings to replace most LLM calls. The system automatically samples data, labels it with an LLM, trains a proxy, evaluates its quality, and decides whether to use the proxy or fall back to full LLM inference. Benchmarks across 11 datasets show proxy models achieve 90–116% of LLM accuracy while consuming ~400x fewer tokens and reducing query latency by 30–100x. The technique is already live in BigQuery and AlloyDB via the 'optimized mode' for AI.IF and AI.CLASSIFY functions.
