Finally, a Replacement for BERT

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

TL;DR This blog post introduces ModernBERT, a family of state-of-the-art encoder-only models representing improvements over older generation encoders across the board, with a 8192 sequence length, better downstream performance and much faster processing. ModernBERT is available as a slot-in replacement for any BERT-like models, with both a base (149M params) and large (395M params) model size. Click to see how to use these models with transformers ModernBERT will be included in v4.48.0 of transformers. Until then, it requires installing transformers from main: pip install…

related reading

• The Annotated Transformernlp.seas.harvard.edu
• Generalized Language Models | Lil'Loglilianweng.github.io
• microgptkarpathy.github.io
• 1810.04805arxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• Mamba: The Easy Wayjackcook.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io