G-Eval is a reference-free LLM evaluation framework from a 2023 Microsoft paper that scores open-ended text generation without needing gold-standard examples. It works through three mechanisms: a precisely defined rubric, a chain-of-thought that the model generates for itself before scoring, and probability-weighted scoring using token log probabilities rather than raw integer outputs. The probability weighting (taking the expectation over possible scores) solves two common failure modes: score collapse onto a dominant value and inability to distinguish close candidates. Key practical guidance includes caching the chain-of-thought per rubric, never using the same model family as both generator and judge (due to documented self-preference bias), calibrating rubrics against human-labeled examples, and falling back to multi-sample averaging when token log probabilities aren't available. G-Eval is inappropriate for closed-form checks like toxicity detection or JSON validation, where classifiers or parsers are faster and cheaper.
