RAG Evaluation

Evaluation or benchmarking methods employed in RAG architecture.

• Auto-RAG
  • AutoRAG
  • An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
  • 4k stars
• Evaluating Retrieval Augmented Generation using RAGAS
  • 2026-04-07 — Superlinked walkthrough of RAGAS: synthetic eval-set generation, faithfulness/retrieval metrics, and mapping to inference/serving concerns

• Redefining Retrieval Evaluation in the Era of LLMs
  • 2025-10-24 — proposes UDCG (Utility and Distraction-aware Cumulative Gain) with LLM-oriented positional discount; improves correlation with end-to-end answer accuracy by up to 36%

• Generative Benchmarking - https://research.trychroma.com/generative-benchmarking