[2602.01132] Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation
Abstract:Tasks such as solving arithmetic equations, evaluating truth tables, and completing syllogisms are handled well by large language models (LLMs) in their standard form, but they often fail when the same problems are posed in logically equivalent yet obfuscated formats. To study this vulnerability, we introduce Logifus, a structure-preserving logical obfuscation framework, and, utilizing this, we present LogiQAte, a first-of-its-kind diagnostic benchmark with 1,108 questions across four reasoning tasks: (i) Obfus FOL (first-order logic entailment under equivalence-preserving rewrites), (ii) Obfus Blood Relation (family-graph entailment under indirect relational chains), (iii) Obfus Number Series (pattern induction under symbolic substitutions), and (iv) Obfus Direction Sense (navigation reasoning under altered directions and reference frames). Across all the tasks, evaluating six state-of-the-art models, we find that obfuscation severely degrades zero-shot performance, with performance dropping on average by 47% for GPT-4o, 27% for GPT-5, and 22% for reasoning model, o4-mini. Our findings reveal that current LLMs parse questions without deep understanding, highlighting the urgency of building models that genuinely comprehend and preserve meaning beyond surface form.
View PDF HTML (experimental) Abstract:Tasks such as solving arithmetic equations, evaluating truth tables, and completing syllogisms are handled well by large language models (LLMs) in their standard form, but they often fail when the same problems are posed in logically equivalent yet obfuscated formats. To study this vulnerability, we introduce Logifus, a structure-preserving logical obfuscation framework, and, utilizing this, we present LogiQAte, a first-of-its-kind diagnostic benchmark with 1,108 questions across four reasoning tasks: (i) Obfus FOL (first-order logic entailment under…
saved by
related reading
- Learning to reason with LLMs | OpenAIopenai.com
- the-illusion-of-thinking.pdfml-site.cdn-apple.com
- [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
- [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
- 2410.05229arxiv.org
- Prompt Injection as Role Confusionrole-confusion.github.io
- [2608.09867] Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
- [2406.02061] Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelsarxiv.org
- Stealing Reasoning Traces from Proprietary LLM APIsarxiv.org
- Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up - ACL Anthologyaclanthology.org
- Reasoning Models Reason Well, Until They Don'tarxiv.org
- Announcing ReasoningLens — Visualizing and Diagnosing LLM Reasoning at a Glancehuggingface.co