Panel of LLM evaluators (PoLL)

Also called Panel-of-Judges, the idea is to use multiple diverse LLMs to evaluate the generated output. This stems from the fact that LMMs are biased to evaluate their own outputs with the higher score.
Medium article summary of the original paper.

Paper abstract:

We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive

Figure 1: Top: Rankings of model performance change drastically depending on which LLM is used as the judge on KILT-NQ. Bottom: The Panel of LLm evaluators (PoLL) has the highest Cohen’s κ correlation with human judgements