Planning with Large Language Models for Code Generation
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process. Although the programs they generate achieve high token-matching-based scores, they often fail to compile or generate incorrect outputs. The main reason is that conventional Transformer decoding algorithms may not be the best choice for code generation. In this work, we propose a novel Transformer decoding algorithm, Planning-Guided Transformer Decoding (PG-TD), that uses a planning algorithm to do lookahead search and guide the Transformer to generate better programs. Specifically, instead of simply optimizing the likelihood of the generated sequences, the Transformer makes use of a planner that generates complete programs and tests them using public test cases. The Transformer can therefore make more informed decisions and output tokens that will eventually lead to higher-quality programs. We also design a mechanism that shares information between
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process. Although the programs they generate achieve high token-matching-based scores, they often fail to compile or generate incorrect outputs. The main reason is that conventional Transformer decoding algorithms may not be the best choice for code generation. In this work, we propose a novel Transformer decoding algorithm, Planning-Guided Transformer Decoding (PG-TD), that uses a planning algorithm to do lookahead search and guide the Transformer to generate…
saved by
related reading
- [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- [2112.02969] Jigsaw: Large Language Models meet Program Synthesisarxiv.org
- Efficient Guided Generation for Large Language Modelsarxiv.org
- [2605.20873] PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelsarxiv.org
- Program Synthesis with Large Language Modelsarxiv.org
- Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
- Large Language Diffusion Modelsarxiv.org
- Looking back at speculative decodingresearch.google
- ZeroCoder: Can LLMs Improve Code Generation Without Ground-Truth Supervision?alphaxiv.org
- Large Language Models for Compiler Optimizationarxiv.org