[2509.14252] LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
Abstract:Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: {\em can language training methods learn a few tricks from the vision ones?} The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: this https URL.
Preprint LLM-JEPA: L ARGE L ANGUAGE M ODELS M EET J OINT E MBEDDING P REDICTIVE A RCHITECTURES Hai Huang Yann LeCun Randall Balestriero Atlassian NYU Brown University hhuang3@atlassian.com…
saved by
related reading
- The Annotated JEPA | Elements of a Vector Spaceelonlit.com
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- GenAI Handbookgenai-handbook.github.io
- What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelsle-wm.github.io
- 2403.09611.pdfarxiv.org
- A History of Large Language Modelsgregorygundersen.com
- Generalized Language Models | Lil'Loglilianweng.github.io
- Large Language Model: world models or surface statistics?thegradient.pub
- The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
- What language do language models speak?tcz.hu