KV cache strategies

The key-value (KV) vectors are used to calculate attention scores. For autoregressive models, KV scores are calculated every time because the model predicts one token at a time. Each prediction depends on the previous tokens, which means the model performs the same computations each time. A KV cache stores these calculations so they can be reused without recomputing them. Efficient caching is crucial for optimizing model performance because it reduces computation time and improves response rates. Refer to the Caching doc for a more detailed explanation about how a cache works. Transformers offers several Cache classes that implement different caching mechanisms. Some of these Cache classes are optimized to save memory while others are designed to maximize generation speed. Refer to the table below to compare cache types and use it to help you select the best cache for your use case. This guide introduces you to the different Cache classes and shows you how to use them for generation. T

Cache strategies · Hugging Face Transformers documentation Cache strategies Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transformers Trans

saved by

• LIME YAO

related reading

• API Reference — TensorRT LLMnvidia.github.io
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Multi Query Attention (MQA) and Grouped-Query Attention (GQA)tinkerd.net
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor modelfergusfinn.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How to make LLMs go fastvgel.me
• TurboQuant: Redefining AI efficiency with extreme compressionresearch.google
• [2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarityarxiv.org