[2602.05970] Inverse Depth Scaling From Most Layers Being Similar
Abstract:Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.
# link_13xioz9y4ru.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Yizhou Liu; Sara Kangaslahti; Ziming Liu; Jeff Gore - Creator=arXiv GenPDF (tex2pdf:a6404ea) - Custom.DOI=https://doi.org/10.48550/arXiv.2602.05970 - Custom.License=http://creativecommons.org/licenses/by/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2602.05970v2 - Producer=pikepdf 8.15.1 - Title=In
saved by
related reading
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- arxiv.org/pdf/2505.24832arxiv.org
- Quantifying the Necessity of Chain of Thought through Opaque Serial Deptharxiv.org
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
- [2607.05316] How Much is Left? LLMs Linearly Encode Their Remaining Output Lengtharxiv.org
- [2509.14786] Pre-training under infinite computearxiv.org
- Infinite Limits of Neural Networks - Kempner Institutekempnerinstitute.harvard.edu
- Large Language Diffusion Modelsarxiv.org
- [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
- [2607.13491] DeepLoop: Depth Scaling for Looped Transformersarxiv.org
- [2605.23857] Strong Teacher Not Needed? On Distillation in LLM Pretrainingarxiv.org