Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

TL;DR

The paper proposes VideoTreeSearch, a framework for grounded long video question answering that uses self-correcting search over an adaptive temporal tree. This approach enables agents to recover from mistakes and outperforms prior methods on several benchmarks. The framework also transfers to general long-video QA tasks.

Why It Matters

The ability to accurately answer questions about long videos has significant implications for applications such as video search, recommendation systems, and content analysis. Effective grounded long video question answering can also improve human-computer interaction and enhance accessibility. This technology has the potential to revolutionize the way we interact with video content.

Key Findings

• VideoTreeSearch outperforms prior agentic methods on Grounded LVQA benchmarks
• The framework enables self-correcting search and recovery from mistakes
• The approach transfers to general long-video QA tasks