Bespoke-MiniCheck-7B

• Resources
  • Model: bespokelabs/Bespoke-MiniCheck-7B
  • Library: Liyan06/MiniCheck
  • Paper: MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
    • EMNLP 2024
• SOTA on LLM-AggreFact at the time of release. From the model card:

Llama-3.1-Bespoke-MiniCheck-7B is the SOTA fact-checking model despite its small size.

• Binary grounding classifier:

The model takes as input a document and a sentence and determines whether the sentence is supported by the document: MiniCheck-Model(document, claim) -> {0, 1}

• Important constraint: multi-sentence claims must first be split via Claim Decomposition and scored sentence-by-sentence.

• Parameters: 7B (BF16)
• Base model: internlm/internlm2_5-7b-chat
• Synthetic data generator: meta-llama/Meta-Llama-3.1-405B-Instruct
• License: CC BY-NC 4.0 (commercial license via company@bespokelabs.ai)

• 35K total examples, mixing real NLI data with curated synthetic data:
  • 21K from ANLI (Adversarial NLI)
  • 7K synthetic “claim-to-document”
  • 7K synthetic “doc-to-claim”
• From the card:

While scaling up the model (compared to what is in MiniCheck) helped, many improvements come from high-quality curation, thus establishing the superiority of Bespoke Labs’s curation technology.

Based on our test on a single A6000 (48 VRAM), Llama-3.1-Bespoke-MiniCheck-7B with vLLM and MiniCheck-Flan-T5-Large have throughputs > 500 docs/min.

• Cheap enough to use as an online metric in production, not just an offline eval.

• Smaller / cheaper variants from the same MiniCheck family - useful as latency-sensitive fallbacks:
  • MiniCheck-Flan-T5-Large (0.8B)
  • MiniCheck-RoBERTa-Large (0.4B)
  • MiniCheck-DeBERTa-v3-Large (0.4B)