Speculative decoding

aarnphm.xyz · 5,558 words · saved by 1 readers

Idea: “draft-and-verify” using smaller models to generate a head tokens (quick explanation from karpathy) Intuitively: In a sense, we are verify these in parallel instead of autoregressive decoding. A few techniques such as ngrams, EAGLE are supported in vLLM Extrapolation Algorithm for Greater Language-model Efficiency Motivation: Difference between EAGLE-1 and EAGLE-3 distribution skew EAGLE does not involve any fine-tuning of the target model, therefore preservation of outputs distributions by EAGLE is theoretically guaranteed for both greedy and non-greedy sampling. This is not the case with Lookahead and Medusa. Observations: autoregressive on feature-level 1 is simpler than token-level, given that there are more regularity. uncertainty in sampling process hinders the performance of predicting the next feature. feature-level are high-dimensional and continuous, meaning sampling “am” or “always” will results in different feature sequences. EAGLE address this by inputs the token seq

Speculative execution for LLMs is an excellent inference-time optimization. It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might… https://t.co/FiwTwqsfho — Andrej Karpathy (@karpathy) 31 août 2023 Intuition: we generate a small set of lookahead tokens, albeit 2-5 tokens with smaller speculators uses the larger Transformers to “verify” the input sequences + draft tokens (then replace tokens that aren’t valid from rejection sampler) In a sense, we are v

Speculative decoding

related reading