Can LLMs Be Computers? | Percepta

We build a computer inside a transformer — executing arbitrary C programs for millions of steps with exponentially faster inference via 2D attention heads.

TL;DR Language models can solve tough math problems at research grade but struggle on simple computational tasks that involve reasoning over many steps and long context. Even multiplying two numbers or solving small Sudokus is nearly impossible unless they rely on external tools. But what does it take for an LLM itself to be as reliable and efficient as a computer? We answer this by literally building a computer inside a transformer. We turn arbitrary C code into tokens that the model itself can execute reliably for millions of steps in seconds. Here is how it works when solving an…

saved by

• Denys
• Ray Arayilakath

related reading

• As Rocks May Think | Eric Jangevjang.com
• Transformer Circuits Threadtransformer-circuits.pub
• 2402.12875arxiv.org
• Thinking like Transformersrush.github.io
• Composer2.pdfcursor.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Memory makes computation universal, remember?thinks.lol
• 2305.18654arxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• Looped Transformers as Programmable Computersarxiv.org
• LLM Resourcesforrestbicker.com
• Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelsarxiv.org