Agent Memory Benchmarks

• A memory benchmark gives a system a sequence of inputs over time, requires the system to write something to a store, and asks the system to retrieve from that store on a later turn to produce the right answer ( AI Memory Benchmarks in 2026).
• Distinct from LLM Benchmarks(NIAH, RULER, BABILong, InfiniteBench, LongBench) that test attention over a single fixed input rather than the multi-session write-and-retrieve loop.

Capacity and continuity are different problems. The same score does not measure both.

• LoCoMo
  • Very long-term conversational memory benchmark - ~300 turns and ~9,000 tokens spread across up to 35 sessions per conversation
  • Paper: Evaluating Very Long-Term Conversational Memory of LLM Agents
    • 2024
    • Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang
• LongMemEval
  • Chat-assistant memory benchmark with 500 curated questions, five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention
  • Two splits: LongMemEval_S (~115K tokens, ~40 sessions/user) and LongMemEval_M (~500 sessions/user)
  • Paper: LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
    • 2024
    • Wu, Wang, Yu, Zhang, Chang, Yu
• BEAM
  • 100 conversations spanning up to 10M tokens with 2,000 probing questions; tests ten memory capabilities including tracking facts/entities, updating information, resolving contradictions, temporal order, multi-hop reasoning, summarization
  • Two tracks: BEAM-1M and BEAM-10M
  • Paper: Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
    • Submitted 2025-10-31, ICLR 2026
    • Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell
  • Github: mohammadtavakoli78/BEAM
  • Dataset: Mohammadta/BEAM on HuggingFace
  • Companion framework: LIGHT - three complementary memory systems (long-term episodic, short-term working, scratchpad). Reports 3.5-12.7% accuracy improvement over strongest long-context baselines

• Agent Memory Benchmark
  • Public leaderboard tracking BEAM and related benchmarks across memory systems
  • BEAM leaderboard

• Cross-session continuity at production scale - LoCoMo tops at ~35 sessions, LongMemEval_M at ~500, BEAM at 10M tokens; no stable definition of “long enough” for year-long personal assistants
• Memory writes - almost every public benchmark grades only retrieval, not what is worth keeping
• Forgetting, eviction, consolidation - no widely adopted benchmark scores these dynamics directly; BEAM’s contradiction/update subscores get closest
• Per-user isolation - none of the public benchmarks test isolation under concurrent multi-user load
• Token economy under realistic budgets - leaderboards usually ignore tokens-per-query cost

• Blogs
  • AI Memory Benchmarks in 2026
    • Original article on mem0.ai
    • 2026-05-11
    • Walkthrough of LoCoMo, LongMemEval, BEAM and where evals fall short
  • Hindsight Is #1 on BEAM
    • Original article on vectorize.io
    • 2026-04-02
    • SOTA result on BEAM-10M and discussion of why the 10M tier matters
  • The Memory Layer for your AI Agents
    • Original article on mem0.ai
    • 2026-04-02
    • Mem0’s explainer on BEAM — generation pipeline, ten memory capabilities, LIGHT framework results, and nugget-based evaluation
• Papers
  • Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
    • 2025-10-31 (ICLR 2026)