Can LLMs Be Computers? | Percepta

We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.

TL;DR Language models can solve tough math problems at research grade but struggle on simple computational tasks that involve reasoning over many steps and long context. Even multiplying two numbers or solving small Sudokus is nearly impossible unless they rely on external tools. But what does it take for an LLM itself to be as reliable and efficient as a computer? We answer this by literally building a computer inside a transformer. We turn arbitrary C code into tokens that the model itself can execute reliably for millions of steps in seconds. Here is how it works when solving an…

saved by

• Denys
• Ray Arayilakath

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• How LLM Inference Worksarpitbhayani.me
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Looped Transformers as Programmable Computersarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Thinking like Transformersrush.github.io
• How to make LLMs go fastvgel.me
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• [2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelsarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org