MosaicBERT: Pretraining BERT from Scratch for $20

With the MosaicBERT architecture + training recipe, you can now pretrain a competitive BERT-Base model from scratch on the MosaicML platform for $20. We’ve released the pretraining and finetuning code, as well as the pretrained weights.

MosaicBERT: Pretraining BERT from Scratch for $20 | Databricks Blog Skip to main content With the MosaicBERT architecture + training recipe, you can now pretrain a competitive BERT-Base model from scratch on the MosaicML platform for $20. We’ve released the pretraining and finetuning code, as well as the pretrained weights. Give our Github Repo a star here! Download the MosaicBERT pretrained weights on the Hugging Face Hub! BERT models—used for everything from sentiment analysis to text summarization—have been the workhorse of modern natural language processing (NLP) since their introduction i

related reading

• 1810.04805arxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Mosaic LLMs: GPT-3 quality formosaicml.com
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Transfer Learninglena-voita.github.io
• >10x More Efficient Pretraining — Magicmagic.dev
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• TOD-Bertarxiv.org
• Recent Advances in Language Model Fine-tuningruder.io