Which metric measures how surprised a language model is by a piece of text?

Get ready for the GARP Risk and AI Exam with flashcards and multiple choice questions. Each question comes with hints and explanations. Prepare for success!

Perplexity measures how surprised a language model is by a piece of text because it reflects the model’s uncertainty when predicting the next token. If the text aligns well with what the model has learned, the predicted probabilities for the actual next token are high, the loss is low, and perplexity is low. If the text is unusual or out of distribution, the model’s predicted probabilities are less accurate, the loss is high, and perplexity is high. This direct link between prediction uncertainty and text, often tied to how fluent or natural the text feels, is why perplexity is used as the fluency-related metric. Demographic parity is a fairness measure, agentic AI relates to autonomy and initiative of systems, and exact match metrics compare outputs to a reference exactly; none specifically capture the model’s surprise in predictions.