What the Models Learned in One Layer Deeper | Core Automation

What worked across 15,602 submissions to the One Layer Deeper competition.

The challenge Reasoning models can spend more compute by generating more tokens. Could they learn to use more steps inside the model instead? Our launch post asked whether a model could “productively use more computational depth at test time than it encountered during training.” The public problem is repeated modular squaring: Repeated modular squaring z = x % N for _ in range(T): z = z * z % N Each squaring depends on the previous result. Increasing T lengthens that chain without requiring longer inputs or answers. The aim was to learn the computation from examples, then run it for…

saved by

• Ishaan Panigrahi

related reading

• Why Inference Matters 12x More Than Training | YC Paper Clubyoutube.com
• Auto-research with codex: How I achieved a 232x Faster Kernel over baseline with Codex in GPU Mode's qr_v2 problemsankalp.bearblog.dev
• (Some of) The Models, They Just Don't Want to Learn | Tildeblog.tilderesearch.com
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• Lil'Loglilianweng.github.io
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Investigating the learning coefficient of modular addition: hackathon project — LessWronglesswrong.com
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Learning to reason with LLMs | OpenAIopenai.com
• Can LLMs Be Computers?percepta.ai
• [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
• 2402.12875arxiv.org