flâneur

A Guide to Structured Outputs Using Constrained Decoding

aidancooper.co.uk · 4,189 words · saved by 1 readers

We often want specific outputs when interacting with generative language models. This is especially true in programming domains, where a generated output may become a direct input to a function. But sometimes, no matter how explicit you are with your instructions, generative language models will get too creative, deviate off task, or simply succumb to their urge to yap. Fortunately, there are techniques that ensure language models only return outputs that conform to your requirements. This article serves as a practitioner's guide for perhaps the most powerful of these techniques: constrained decoding. We'll cover what structured generation and constrained decoding are, how they work, best practices, useful patterns, and pitfalls to avoid. This article is a living document being continuously updated as the field evolves. Terminology such as "structured generation" and "constrained decoding" do not have consensus definitions, and their scopes are evolving over time — the discussion here

Introduction We often want specific outputs when interacting with generative language models. This is especially true in programming domains, where a generated output may become a direct input to a function. But sometimes, no matter how explicit you are with your instructions, generative language models will get too creative, deviate off task, or simply succumb to their urge to yap . Fortunately, there are techniques that ensure language models only return outputs that conform to your requirements. This article serves as a practitioner's guide for perhaps the most powerful of these techniques:

related reading