sapientinc/HRM-Text: HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning. ·
github.com · 1,465 words · saved by 1 readers
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
HRM-Text: Efficient Pretraining Beyond Scaling 🌟 Pretrain a foundation model from scratch with ~$1000. 🌠 HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning. It offers a full pretraining framework, making foundation model pretraining accessible with 130-600x less compute and 150-900x less data. It is built upon a hierarchical recurrent architecture, PrefixLM sequence packing, FlashAttention 3 kernels, PyTorch FSDP2 training, evaluation, and checkpoint conversion tooling. Join 1200+ HRM Developers on Our Discord…
saved by
related reading
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
- Let's build GPT: from scratch, in code, spelled out.youtube.com
- microgptkarpathy.github.io
- An Open Base Model for Conversational Speech | blogsi.inc
- GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automaticallygithub.com
- GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.github.com
- Llama 2 · Hugging Facehuggingface.co
- GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
- Let's reproduce GPT-2 (124M)youtube.com
- The Annotated Transformernlp.seas.harvard.edu
- Optimizing inference · Hugging Facehuggingface.co