KV cache strategies

The key-value (KV) vectors are used to calculate attention scores. For autoregressive models, KV scores are calculated every time because the model predicts one token at a time. Each prediction depends on the previous tokens, which means the model performs the same computations each time. A KV cache stores these calculations so they can be reused without recomputing them. Efficient caching is crucial for optimizing model performance because it reduces computation time and improves response rates. Refer to the Caching doc for a more detailed explanation about how a cache works. Transformers offers several Cache classes that implement different caching mechanisms. Some of these Cache classes are optimized to save memory while others are designed to maximize generation speed. Refer to the table below to compare cache types and use it to help you select the best cache for your use case. This guide introduces you to the different Cache classes and shows you how to use them for generation. T

Cache strategies · Hugging Face Transformers documentation Cache strategies Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transformers Trans

saved by

• LIME YAO

related reading

• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Optimizing inference · Hugging Facehuggingface.co
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Overview · Hugging Facehuggingface.co
• All About Transformer Inferencejax-ml.github.io
• Transformer inference trickssubstack.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Cross-Model KV Cache Transfer in LLM Families:A Closed-Form Linear Mapping for Prefill Reusearxiv.org
• Continuous batching from first principleshuggingface.co
• Multi Query Attention (MQA) and Grouped-Query Attention (GQA)tinkerd.net