✳flâneur — a map of the web's best reading

[2107.03374] Evaluating Large Language Models Trained on Code

arxiv.org · 839 words · saved by 1 readers

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

[2107.03374] Evaluating Large Language Models Trained on Code Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2107.03374 (cs) [Submitted on 7 Jul 2021 ( v1 ), last revised 14 Jul 2021 (this version, v2)] Title: Evaluating Large Language Models Trained on Code Authors: Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan , Harri Edwards , Yuri Burda , Nicholas Joseph , Greg Brockman , Alex Ray , Raul Puri , Gret

Explore this link on the map →

[2107.03374] Evaluating Large Language Models Trained on Code

related reading