AutoJudge is a new method for accelerating LLM inference via task-specific lossy speculative decoding. Unlike standard speculative decoding that rejects any token mismatch between draft and target models, AutoJudge uses a lightweight logistic regression classifier—trained in a self-supervised manner on existing transformer hidden states—to distinguish 'important' mismatches (those that affect final answer quality) from 'unimportant' ones (harmless stylistic differences). By accepting unimportant mismatches, AutoJudge can accept up to 40–45 draft tokens per verification cycle, achieving 1.5–2× speedups over standard speculative decoding with only ~1–4% accuracy drops on math reasoning (GSM8K) and coding (LiveCodeBench) benchmarks. It integrates with vLLM, TensorRT-LLM, and TGI, and composes with EAGLE-2 for additional gains. No manual annotation is required. The work will be presented at NeurIPS 2025.
