Answer.AI - You can now train a 70b language model at home

Today, we’re releasing Answer.AI’s first project: a fully open source system that, for the first time, can efficiently train a 70b large language model on a regular desktop computer with two or more standard gaming GPUs (RTX 3090 or 4090). This system, which combines FSDP and QLoRA, is the result of a collaboration between Answer.AI, Tim Dettmers (U Washington), and Hugging Face’s Titus von Koeller and Sourab Mangrulkar. This system will help the open source community release better models. Teknium, the creator of the extremely popular OpenHermes models and datasets, with over half a million downloads, said: “With this capability we can take huge models to new heights locally, and gigantic, hundreds of billions of parameter models are now accessible by small labs.” At Answer.AI we made this our first project because it’s a key foundation of our north star: helping make useful AI available to everyone. Just being able to use other people’s models is not enough. We want everyone to be ab

You can now train a 70b language model at home – Answer.AI Summary Today, we’re releasing Answer.AI’s first project: a fully open source system that, for the first time, can efficiently train a 70b large language model on a regular desktop computer with two or more standard gaming GPUs (RTX 3090 or 4090). This system, which combines FSDP and QLoRA, is the result of a collaboration between Answer.AI, Tim Dettmers (U Washington), and Hugging Face’s Titus von Koeller and Sourab Mangrulkar. This system will help the open source community release better models. Teknium, the creator of the extremely

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Mosaic LLMs: GPT-3 quality formosaicml.com
• Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org
• 2305.14314arxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Efficient LLM Finetuning with Unsloth | Modal Docsmodal.com
• How is LLaMa.cpp possible?finbarr.ca
• GitHub - openlm-research/open_llama: OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama datasetgithub.com
• RedPajama, a project to create leading open-source models, starts by reproducing LLaMA training dataset of over 1.2 trillion tokenstogether.xyz
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• >10x More Efficient Pretraining — Magicmagic.dev
• Can LLMs Be Computers?percepta.ai