G-Eval

• G-Eval is a SOTA, research-backed framework that uses LLM-as-a-judge to evaluate LLM outputs on any criteria using everyday language.
• G-Eval uses various techniques such as chain-of-thought, token weight summation, and form-filling paradigms to bypass pitfalls LLM judges are commonly vulnerable to.
• G-Eval can be further optimized by introducing rubrics, hardcoding criteria, and extended to a multi-turn use case.
• G-Eval can be used to evaluate AI agents, for both single-turn and multi-turn use cases.
• DeepEval allows anyone to implement G-Eval in under 5 lines of code ( docs here).

G-Eval is a framework that uses LLM-as-a-judge with chain-of-thoughts (CoT) to evaluate LLM outputs based on ANY custom criteria. It leverages an automatic chain-of-thought (CoT) approach to decompose your criteria and evaluate LLM outputs through a three-step process:

1. Evaluation Step Generation: an LLM first transforms your natural language criterion into a structured list of evaluation steps.
2. Judging: these steps are then used by an LLM judge to assess your application’s output.
3. Scoring: the resulting judgments are weighted by their log-probabilities to produce a final G-Eval score.

G-Eval was first introduced in the paper “NLG Evaluation using GPT-4 with Better Human Alignment”, and was originally developed as a superior alternative to traditional reference-based metrics like BLEU and ROUGE, which struggles with subjective and open-ended tasks that requires creativity, nuance, and an understanding of word semantics.