LLM-AggreFact

• Resources
  • Leaderboard: llm-aggrefact.github.io
  • Paper: MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
    • EMNLP 2024
  • GitHub: Liyan06/MiniCheck
• From the leaderboard page:

A fact-checking benchmark that aggregates 11 of the most up-to-date publicly available datasets on grounded factuality (i.e., hallucination) evaluation.

• 11 grounded-factuality datasets, ~29K test samples, all with human-annotated labels:
  • AggreFact-CNN, AggreFact-XSum - summarization factuality
  • TofuEval-MediaS, TofuEval-MeetB - dialogue/meeting summarization
  • WiCE - claim verification over Wikipedia
  • Reveal - reasoning chain verification
  • ClaimVerify - search-augmented response verification
  • FactCheck-GPT - factuality of ChatGPT responses
  • ExpertQA - expert-domain QA
  • Lfqa - long-form QA
  • RAGTruth - RAG-specific hallucination labels
• Input format is uniform across all datasets: (document, claim) -> {0, 1} where 1 = claim is supported by document.

• Selected snapshot - see live board for current ranking:

• Model sizes on the board range from 0.4B to 405B, mixing dedicated fact-checkers (MiniCheck family, Granite Guardian) with general-purpose frontier LLMs (Claude, GPT, Llama).
• Key takeaway: a well-trained 7B specialized model is on par with or above the best frontier LLMs at this task, at a fraction of the inference cost.