Seeing Is Not Reasoning: How VLMs and Their Benchmarks Lean on Text

We stress-test 8 open-weight VLMs on 9 multimodal benchmarks under different input conditions to ask how much each benchmark actually depends on the image, and propose per-instance metrics — Modality Differential, Caption Substitution, Visual Dependence — to audit it.

📌 TL;DR We test 8 openly released VLMs on 9 VQA benchmarks under different input conditions, and propose three metrics ( Modality Differential , Caption Substitution , Visual Dependence ) to ask two questions: how much does each benchmark actually depend on the image , and how much do VLMs actually reason over it? Many widely used VQA benchmarks barely need the image. In roughly a third of the (model × benchmark) pairs, the VLM performs as well or better when the image is replaced by a short caption. Visual dependence decreases as VLMs scale up, since a larger backbone answers more from

saved by

• Lailah Nabegu

related reading

• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelsarxiv.org
• Less Detail, Better Answers: Degradation-Driven Prompting for VQAarxiv.org
• State of Vision-Language-Action (VLA) Research at ICLR 2026 – Moritz Reussmbreuss.github.io
• 2403.09611.pdfarxiv.org
• Circuit Tracing in Vision–Language Models:Understanding the Internal Mechanisms of Multimodal Thinkingarxiv.org
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• Video models are zero-shot learners and reasonersarxiv.org
• Part 1: The Map Was Wrong — Nemostationnemostation.com
• MMHal Bencharxiv.org
• [2301.13823] Grounding Language Models to Images for Multimodal Generationarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org