The Truth Lies Somewhere in the Middle (of the Generated Tokens)
Autoregressive language models produce a sequence of hidden states during generation, but it is unclear how this sequence should be collapsed into a single representation. We find that averaging hidden states across generated tokens yields representations that better capture the semantic content of the input than any individual token. We show five generation examples, each with its image, prompt, output, and PCA trajectory of embeddings from the Qwen3-14B language model. Embeddings are computed using either last-token pooling or mean pooling over generated tokens. Alignment is measured using the CKA metric by comparing the language kernel from these representations with a vision reference kernel from DINOv2 embeddings, using 1,024 image-caption pairs from the Wikipedia-based Image-Text dataset. The red star marks the embedding obtained by mean pooling 128 generated tokens. We find that mean-pooled token representations become increasingly aligned over the course of generation. Hidden s