StarCoder: A State-of-the-Art LLM for Code

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Introducing StarCoder StarCoder and StarCoderBase are Large Language Models for Code (Code LLMs) trained on permissively licensed data from GitHub, including from 80+ programming languages, Git commits, GitHub issues, and Jupyter notebooks. Similar to LLaMA, we trained a ~15B parameter model for 1 trillion tokens. We fine-tuned StarCoderBase model for 35B Python tokens, resulting in a new model that we call StarCoder. We found that StarCoderBase outperforms existing open Code LLMs on popular programming benchmarks and matches or surpasses closed models such as code-cushman-001 from OpenAI…

related reading

• Coding Models Are Doing Too Much | whnrehiew.github.io
• GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
• GitHub - zai-org/CodeGeeX: CodeGeeX: An Open Multilingual Code Generation Model (KDD 2023)github.com
• The open source AI coding agentopencode.ai
• LiCoEval: Evaluating LLMs on License Compliance in Code Generationarxiv.org
• Introducing FrontierCode | Cognitioncognition.ai
• [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
• ZeroCoder: Can LLMs Improve Code Generation Without Ground-Truth Supervision?alphaxiv.org
• Introducing FrontierCode | Cognitioncognition.com
• Here’s how I use LLMs to help me write codesimonwillison.net
• Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebasedatabricks.com
• Introducing Marin: An Open Lab for Building Foundation Models | Marinmarin.community