Inference characteristics of Llama · Cursor

Llama-2-70B is an alluring alternative to GPT-3.5, but if looking for a cheap language model, it may not be worth it to deviate from OpenAI’s API. When considering price and latency: You should not serve Llama-2 for completion-heavy workloads Instead, Llama is best for prompt-dominated tasks, such as classification. Llama-2 may also make sense when: Otherwise, GPT-3.5 should be cheaper and faster. A quick disclaimer, one reason to use Llama over GPT-3.5 is finetuning 1. But in this post, we only explore cost and latency. I don’t compare Llama-2 to GPT-4, as it is closer to a 3.5-level model. Benchmark performance also supports this claim: I’ll prove these assertions by comparing the cost of serving Llama-2-70B with GPT-3.5-turbo given roughly similar latencies. We serve Llama on 2 80-GB A100 GPUs, as that is the minumum required to fit Llama in memory (with 16-bit precision) 3. On 2-A100s, we find that Llama has worse pricing than GPT-3.5 for completion tokens. We speculate competitive

Blog / research Llama-2-70B is an alluring alternative to GPT-3.5, but if looking for a cheap language model, it may not be worth it to deviate from OpenAI’s API. When considering price and latency: You should not serve Llama-2 for completion-heavy workloads Instead, Llama is best for prompt-dominated tasks , such as classification. Llama-2 may also make sense when: Your workload has no prompt tokens (unintuitive but we’ll explain later) You are performing batch processing jobs Otherwise, GPT-3.5 should be cheaper and faster. A quick disclaimer, one reason to use Llama over GPT-3.5 is finetuni

saved by

• Enrique Moran

related reading

• Inference characteristics of Llama · Cursorcursor.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• LLM Inference Economics from First Principlestensoreconomics.com
• How is LLaMa.cpp possible?finbarr.ca
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Optimizing inference · Hugging Facehuggingface.co
• How LLM Inference Worksarpitbhayani.me
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io