How Retrieved Context Shapes Internal Representations in RAG

This paper from WISC investigates how retrieved documents in RAG systems shape the internal hidden representations of LLMs. Using four QA datasets and three LLMs, the authors classify retrieved documents into relevant, distracting, and random categories, then analyze last-prompt-token representations via PCA under controlled single- and multi-document settings. They share the following major findings:

1. Random documents cause the largest representation drift (even larger than relevant or distracting ones), and this drift is tightly correlated with abstention behavior.

2. Relevant documents barely shift representations at all, acting mainly as confidence boosters for queries the model can already answer rather than injecting genuinely new information, which explains why RAG struggles on hard queries.

3. In multi-document settings, even a single relevant document anchors the representation and suppresses noise from accompanying distractors or random docs.

4. Layer-wise analysis shows that models detect coarse semantic mismatches (random context) early on but can’t reliably separate relevant from distracting documents until much later layers.

5. Later layers progressively favor parametric knowledge over retrieved evidence, which stabilizes generation for easy queries but fundamentally limits RAG’s ability to override insufficient internal knowledge on harder ones.

The practical conclusion is that aggressive document filtering may be unnecessary when at least one relevant document is present, and current RAG architectures have a built-in ceiling on how much retrieved evidence can actually influence model behavior.