TiDAR: Think in Diffusion, Talk in Autoregression | alphaXiv

Large Language Models (LLMs) face a fundamental efficiency problem during text generation. Traditional autoregressive (AR) models like GPT-4 and Llama generate text one token at a time, severely underutilizing modern GPU hardware. This sequential generation creates a bottleneck where powerful parallel processors sit largely idle, generating only a single token per forward pass. Figure 1: Analysis showing "free token slots" on GPUs where additional tokens can be processed with minimal latency increase, revealing untapped computational capacity in current LLM inference. The research community has explored two main approaches to address this inefficiency: diffusion language models (dLMs) that can generate multiple tokens in parallel, and speculative decoding methods that use smaller "draft" models to propose token sequences. However, both approaches face significant limitations. Diffusion models typically sacrifice quality for speed, while speculative decoding relies on separate models an

Understanding the GPU Efficiency Challenge Large Language Models (LLMs) face a fundamental efficiency problem during text generation. Traditional autoregressive (AR) models like GPT-4 and Llama generate text one token at a time, severely underutilizing modern GPU hardware. This sequential generation creates a bottleneck where powerful parallel processors sit largely idle, generating only a single token per forward pass. Figure 1: Analysis showing "free token slots" on GPUs where additional tokens can be processed with minimal latency increase, revealing untapped computational capacity in…

saved by

• Lev Chizhov

related reading

• Diffusion will be everywherevarunneal.github.io
• Large Language Diffusion Modelsarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Speculative Decoding - philkravphilkrav.com
• Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusionarxiv.org
• Kuleshov Group | How to Build a Diffusion Language Modelkuleshov-group.github.io
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Large Language Diffusion Modelsml-gsai.github.io