Looking back at speculative decoding

research.google · 1,711 words · saved by 3 readers

We strive to create an environment conducive to many different types of research across many different time scales and levels of risk. Our researchers drive advancements in computer science through both fundamental and applied research. We regularly open-source projects with the broader research community and apply our developments to Google products. Publishing our work allows us to share ideas and work collaboratively to advance the field of computer science. We make products, tools, and datasets available to everyone with the goal of building a more collaborative ecosystem. Supporting the next generation of researchers through a wide range of programming. Participating in the academic research community through meaningful engagement with university faculty. Connecting with the broader research community through events is essential for creating progress in every aspect of our work. December 6, 2024 Yaniv Leviathan, Distinguished Engineer, Matan Kalman, Software Engineer, and Yossi Ma

Large language models (LLMs) are at the center of the recent rapid progress in artificial intelligence (AI). While groundbreaking, a challenge for user-facing products is that due to their size, these large models are slow at inference (i.e., output generation), which may result in an undesirably slow user experience. In 2022 we published "Fast Inference from Transformers via Speculative Decoding", which introduced a technique called speculative decoding that can reduce the inference times for LLMs significantly. The algorithm speeds up generation from autoregressive models by computing…

Looking back at speculative decoding

saved by

related reading