Deft Fixing LLM Writing with Distribution Fine-Tuning
Rosmine's research paper on Distribution Fine-Tuning: the measurement stack, SFT failure modes, DFT results, model comparisons, methods, data, and limitations.
Slop. It's not just annoying — it's exhausting. You're absolutely right to be annoyed by it, and in this blog I will delve into a solution. You've probably noticed most models have their favorite words or phrases they overuse, like "—", "it's not X, it's Y", or "delve". Before investigating the solution, I first address the metrics I use to measure output quality. Instead of measuring "quality" itself, which is not well defined, I measure similarity to human writing samples. Metrics N-Gram Token Distribution L2 Distance This metric captures word choice similarity, and is useful for…
saved by
related reading
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
- Large Language Diffusion Modelsarxiv.org
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- [2302.08582] Pretraining Language Models with Human Preferencesarxiv.org
- Fine-Tuning Llama-2: Tailoring Models to Unique Applicationsanyscale.com
- Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com