TiDAR: Think in Diffusion, Talk in Autoregression | alphaXiv

Large Language Models (LLMs) face a fundamental efficiency problem during text generation. Traditional autoregressive (AR) models like GPT-4 and Llama generate text one token at a time, severely underutilizing modern GPU hardware. This sequential generation creates a bottleneck where powerful parallel processors sit largely idle, generating only a single token per forward pass. Figure 1: Analysis showing "free token slots" on GPUs where additional tokens can be processed with minimal latency increase, revealing untapped computational capacity in current LLM inference. The research community has explored two main approaches to address this inefficiency: diffusion language models (dLMs) that can generate multiple tokens in parallel, and speculative decoding methods that use smaller "draft" models to propose token sequences. However, both approaches face significant limitations. Diffusion models typically sacrifice quality for speed, while speculative decoding relies on separate models an

saved by

• Lev Chizhov