[2510.01634] CAT: Curvature-Adaptive Transformers for Geometry-Aware Learning

Abstract:Transformers achieve strong performance across diverse domains but implicitly assume Euclidean geometry in their attention mechanisms, limiting their effectiveness on data with non-Euclidean structure. While recent extensions to hyperbolic and spherical spaces show promise for hierarchical and cyclical patterns, respectively, they require committing to a single geometry a priori, reducing flexibility when data exhibits mixed geometric properties. We introduce the Curvature-Adaptive Transformer (CAT), a novel architecture that dynamically learns per-token routing across three geometric attention branches through a lightweight, differentiable gating mechanism. Unlike fixed-geometry approaches, CAT enables adaptive geometric specialization, routing tokens to the appropriate curvature based on their local relational structure. The routing network provides interpretable curvature preferences while each branch employs geometry-specific operations optimized for its respective manifold. On knowledge graph completion benchmarks (FB15k-237, WN18RR), CAT achieves approximately 10% improvements in MRR and Hits@10 over fixed-geometry baselines with minimal overhead (5% parameter increase, comparable inference time). These results demonstrate that learned geometric adaptation outperforms any single fixed geometry for complex relational reasoning, establishing CAT as a scalable and interpretable foundation for mixture-of-geometry architectures across language, vision, and multimodal domains.

# link_11fm4m5v5ed.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Ryan Y. Lin; Siddhartha Ojha; Nicholas Bai - Creator=arXiv GenPDF (tex2pdf:) - Custom.DOI=https://doi.org/10.48550/arXiv.2510.01634 - Custom.License=http://creativecommons.org/licenses/by/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2510.01634v1 - Producer=pikepdf 8.15.1 - Title=CAT: Curvature-Ada

saved by

• Vihan Tiwari

related reading

• Steering Along Manifolds to Control Neural Networksgoodfire.ai
• transformer_attention.pdfarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• When Models Manipulate Manifolds: The Geometry of a Counting Tasktransformer-circuits.pub
• [2207.02505] Pure Transformers are Powerful Graph Learnersarxiv.org
• [1810.00825] Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networksarxiv.org
• [2510.26745] Deep sequence models tend to memorize geometrically; it is unclear whyarxiv.org
• 1706.03762arxiv.org
• [2602.15799] The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safetyarxiv.org
• arxiv.org/pdf/2512.24880#page=3.56arxiv.org
• [2402.01258] Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapearxiv.org
• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org