TensorStream - Perceptron

Generative causal transformers are 1D computers. All instructions need to be serialized into a 1D token stream. For text-only data it's simple because humans process it the same way. All text-only data is stored/processed as 1D token streams. The problem gets challenging with multistream/multimodal data e.g. videos. Not only do videos have separate time-aligned audio and vision streams but also can have multiple metadata streams like transcriptions, narration, or even actions for robotics/video game data. This presents a challenge - we need some protocol for forming this token stream. The problem of preparing this data is 2-stage. The first is about organizing the raw data into a causal chain. The second is about storing that data in a way it can flexibly be used and ultimately converted into an input to a transformer model. We solve this problem with a data structure called TensorStream and a library built around it. First we read in the raw data into packets we call Events which are

Perceptron September 23, 2025 TensorStream Structuring Multimodal Data for Causal Transformers September 23, 2025 TensorStream Structuring Multimodal Data for Causal Transformers September 23, 2025 TensorStream Structuring Multimodal Data for Causal Transformers Problem Generative causal transformers are 1D computers. All instructions need to be serialized into a 1D token stream. For text-only data it's simple because humans process it the same way. All text-only data is stored/processed as 1D token streams. The problem gets challenging with multistream/multimodal data e.g. videos. Not only do

saved by

• Pranav
• Elizabeth Qiu
• Karan MJ
• Cheikh Fiteni

related reading

• Transformers from Scratche2eml.school
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformers from Scratchbrandonrohrer.com