Tensor-Transformer Variants are Surprisingly Performant — LessWrong

I've been researching tensor networks as a more interpretable architecture, but whenever I tell people this, they always ask "But is it any good?" …

x Tensor-Transformer Variants are Surprisingly Performant — LessWrong AI Frontpage 87 Tensor-Transformer Variants are Surprisingly Performant by Logan Riggs 12th Jan 2026 5 min read 16 87 I've been researching tensor networks as a more interpretable architecture, but whenever I tell people this, they always ask "But is it any good?" So I trained multiple 500M parameter LLMs on fineweb, showing the tensor variant needed ~4% more batches of data to match CE-loss. There's a few caveats, so my personal estimate is around 15% worst to 10% better. Details below. The Architecture Replacing MLP w/ a B

saved by

• Asher P
• Sudarsh K
• Emanuel Ruzak

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• I am worried about near-term non-LLM AI developments — LessWronglesswrong.com
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• transformer_attention.pdfarxiv.org
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Softmax Linear Unitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io