RedPajama, a project to create leading open-source models, starts by reproducing LLaMA training dataset of over 1.2 trillion tokens — TOGETHER
RedPajama is a project to create a set of leading, fully open-source models. Today, we are excited to announce the completion of the first step of this project: the reproduction of the LLaMA training dataset of over 1.2 trillion tokens.
Foundation models such as GPT-4 have driven rapid improvement in AI. However, the most powerful models are closed commercial models or only partially open. RedPajama is a project to create a set of leading, fully open-source models. Today, we are excited to announce the completion of the first step of this project: the reproduction of the LLaMA training dataset of over 1.2 trillion tokens. The most capable foundation models today are closed behind commercial APIs, which limits research, customization, and their use with sensitive data. Fully open-source models hold the promise of removing thes
saved by
related reading
- GitHub - openlm-research/open_llama: OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama datasetgithub.com
- Google "We Have No Moat, And Neither Does OpenAI"semianalysis.com
- Introducing Marin: An Open Lab for Building Foundation Models | Marinmarin.community
- Accelerating LLaMA with Fabric: A Comprehensive Guide to Training and Fine-Tuning LLaMA - Lightning AIlightning.ai
- Stanford CRFMcrfm.stanford.edu
- Mosaic LLMs: GPT-3 quality formosaicml.com
- A developer’s guide to open source LLMs and generative AIgithub.blog
- Llama 2 · Hugging Facehuggingface.co
- The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
- Large Language Diffusion Modelsarxiv.org
- Releasing Alpaca-30Babuqader.substack.com
- GitHub - open-compass/opencompass: OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.github.com