Ketulkumar Polara
0 followers · 248 views
on the atlas · 3
- How to derive chunking strategy for specific use case for RAG1 savers
- LLM Inference Performance Engineering: Best Practices | Databricks Blog6 savers
- Transformer Inference Arithmetic | kipply's blog17 savers
highlights · 4
IT has demonstrated the ability to enhance model performance with minimal downstream training data, making it a quick learner compared to traditional supervised fine-tuning approaches
How to derive chunking strategy for specific use case for RAG"prefill", where the tokens in the input prompt are processed in parallel, and "decoding", where text is generated one 'token' at a time in an autoregressive manner.
LLM Inference Performance Engineering: Best Practices | Databricks BlogWe do 2 ⋅ � 2⋅P flops of operations, which can be intuited by the fact that we matmul through all the parameters.
Transformer Inference Arithmetic | kipply's blogOur latency calculations are mostly about the flops vs memory boundedness. If we have a small number of multiplies to do per parameter, then maybe we'll be throttled by memory bandwidth. Flops are increased by both batch size and number of parameters, while memory is only increased by number of parameters.
Transformer Inference Arithmetic | kipply's blog