LLM.int8() and Emergent Features — Tim Dettmers
When I attended NAACL, I wanted to do a little test. I had two pitches for my LLM.int8() paper. One pitch is about how I use advanced quantization methods to achieve no performance degradation transformer inference at scale that makes large models more accessible. The other pitch talks about emergent outliers in transformers and how […]
When I attended NAACL, I wanted to do a little test. I had two pitches for my LLM.int8() paper. One pitch is about how I use advanced quantization methods to achieve no performance degradation transformer inference at scale that makes large models more accessible. The other pitch talks about emergent outliers in transformers and how they radically change what transformers learn and how they function. From that, I learned that quantization research is like printers. Nobody cares about printers. Nobody likes printers. But everybody is happy if printers do their job. How that job is done for you
related reading
- LLM.int8()arxiv.org
- The Annotated Transformernlp.seas.harvard.edu
- Attention Is Off By One – Evan Millerevanmiller.org
- Quantization from the ground upngrok.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Efficient LLM inferencefinbarrtimbers.substack.com
- Non_Interactive – Software & MLnonint.com
- Compression and Intelligencegreene.sh
- Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- Transformers Laid Outgoyalpramod.github.io