Perplexity

Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized sequence $X = (x_0, x_1, \ldots, x_t)$, then the perplexity of $X$ is,

$$
\text{PPL}(X) = \exp \left{ -\frac{1}{t} \sum_{i=1}^t \log p_\theta(x_i | x_{<i}) \right}
$$

where $\log p_\theta(x_i | x_{<i})$ is the log-likelihood of the $i$-th token conditioned on the preceding tokens $x_{<i}$ according to our model. Intuitively, it can be thought of as an evaluation of the model’s ability to predict uniformly among the set of specified tokens in a corpus. Importantly, this means that the tokenization procedure has a direct impact on a model’s perplexity, which should always be taken into consideration when comparing different models.

Source: ChatGPT¹

• It represents the effective number of choices the model has when generating the next token.
• Think of it as: “On average, how many equally probable options does the model consider at each step?”
• Lower perplexity = better model performance (less uncertainty),
• Higher perplexity = worse performance (more uncertainty).

• Focus on Local Prediction: Perplexity measures next-word prediction, which may yield locally coherent but contextually disjoint content.
• Limited in Handling Ambiguity and Creativity: It doesn’t capture a model’s ability to handle ambiguity or generate creative language.
• Vocabulary Dependency: Perplexity is sensitive to vocabulary size; unfamiliar terms can raise scores, affecting model evaluation.
• Overfitting Risk: Low perplexity on training data may not indicate strong generalization to diverse, real-world scenarios.

• Measures model fluency and coherence, indicating language understanding and task comprehension.
• Offers insight into model generalization; lower perplexity on unseen data shows better generalization.
• Provides a simple, cost-effective comparison metric for models, enabling quantitative evaluation.
• Serves as a key metric for model optimization, with lower perplexity indicating improved model quality.
• Assesses generated language quality, aiding in evaluating content applications using language models.