AVA-Encoder: Towards Agent-Native Video Representation Learning

TL;DR

AVA-Encoder introduces a framework that converts videos into structured knowledge graphs for use by intelligent agents, enabling reasoning and editing through text and media assets. It uses a dual-loop optimization process to refine encoding policies and knowledge graph representations, improving video reconstruction quality. The approach achieves significant gains over existing methods and reduces reliance on extensive prompt engineering.

Why It Matters

This work enables creative agents to understand and manipulate cinematic content more effectively, bridging the gap between high-quality human-made videos and AI-driven video generation. By providing a standardized, agent-friendly video representation, it supports scalable, interpretable, and editable video processing in real-world applications.

Key Findings

• AVA-Encoder transforms videos into knowledge graphs with structured text, media assets, and typed relationships, enabling direct agent interaction.
• The framework achieves a 20.7 percentage point improvement over the strongest baseline in video reconstruction accuracy.
• The pseudo-trained encoding policy outperforms a manually tuned policy while using 74.3% fewer system-prompt tokens in a policy-only setting.