Methods and tools for efficient training on a single GPU
huggingface.co · 646 words · saved by 1 readers
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Transformers provides multiple inference optimization techniques to make models fast, affordable, and accessible. Options include alternative attention mechanisms for reduced memory traffic, code compilation for faster execution, and optimized kernels for throughput. Stack these techniques for maximum performance. Memory and speed are closely related but not the same. Shrinking your memory footprint makes a model “faster” because there is less data to move around. Pure speed optimizations don’t always reduce memory and sometimes increase usage. Choose the appropriate optimization based on…
related reading
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- Together AI | The AI Native Cloudtogether.ai
- Hugging Face – The AI community building the future.huggingface.co
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Llama 2 · Hugging Facehuggingface.co
- Optimizing inference · Hugging Facehuggingface.co
- Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blogdeveloper.nvidia.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Hugging Face · GitHubgithub.com
- All About Transformer Inferencejax-ml.github.io
- KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co