flâneur

LLM Optimization via Synthetic Distillation

anarchyai.substack.com · 1,085 words · saved by 1 readers

The rise of Large Language Models (LLMs) has led to new learning techniques like K-shot prompting, where a model that is given just a few question-and-answer examples can create new answers for novel questions. This works best on large models, but running them can be expensive and slow. Here we will demonstrate how to use Model Distillation to use the outputs of a larger model to train a smaller model in order to achieve comparable performance with less computational weight. Understanding Model Distillation Model distillation transfers knowledge from a larger "teacher" model to a smaller, efficient "student" model, aiming for the student to mimic the teacher's outputs with fewer resources. In our examples, we’ll use the ChatGPT-4 as the teacher, and the 70-million parameter pythia-70m model as the student. We’ll also compare the performance with the 3-billion parameter open-llama model as a student. In principle, you can use any two models. Let’s say we’re going to use our LLMs to help

The rise of Large Language Models (LLMs) has led to new learning techniques like K-shot prompting, where a model that is given just a few question-and-answer examples can create new answers for novel questions. This works best on large models, but running them can be expensive and slow. Here we will demonstrate how to use Model Distillation to use the outputs of a larger model to train a smaller model in order to achieve comparable performance with less computational weight. Understanding Model Distillation Model distillation transfers knowledge from a larger "teacher" model to a smaller,…

saved by

related reading