ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

TL;DR

ER-JEPA improves joint-embedding predictive learning in language models by adding an episodic replay path. This enables learning from both current and stored training pairs, providing additional supervision. ER-JEPA outperforms LLM-JEPA across multiple datasets.

Why It Matters

ER-JEPA enhances the accuracy and stability of language model predictions, which is crucial for real-world applications. Improved language understanding can lead to better performance in tasks like text generation and question answering. This can have significant impacts on industries like customer service and content creation.

Key Findings

• ER-JEPA consistently outperforms LLM-JEPA
• Episodic replay improves token prediction and representation alignment
• ER-JEPA achieves better results across multiple datasets