[2509.14252] LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

Abstract:Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: {\em can language training methods learn a few tricks from the vision ones?} The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: this https URL.

Preprint LLM-JEPA: L ARGE L ANGUAGE M ODELS M EET J OINT E MBEDDING P REDICTIVE A RCHITECTURES Hai Huang Yann LeCun Randall Balestriero Atlassian NYU Brown University hhuang3@atlassian.com…

saved by

• Arya Somu

related reading

• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• The Annotated JEPA | Elements of a Vector Spaceelonlit.com
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Large Language Diffusion Modelsarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org
• [2605.06546] Efficient Pre-Training with Token Superpositionarxiv.org