GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

TL;DR

GameHorizon Suite introduces a comprehensive framework for evaluating AI models in video games across multiple temporal horizons. It includes an automated annotation system, a large-scale dataset with aligned video, actions, and instructions, and a dual-track benchmark for offline and online evaluation. The suite enables standardized, reproducible assessment of models across diverse games and tasks.

Why It Matters

It establishes a unified standard for measuring AI performance in complex, long-horizon gameplay scenarios, addressing limitations in prior benchmarks. This supports more reliable comparisons across model architectures and advances research in multimodal reasoning, planning, and action control.

Key Findings

• GameHorizon-Data contains 5,000 hours of gameplay from 21 AAA titles, annotated with multi-horizon language instructions by expert players.
• The benchmark reveals a clear hierarchy of task difficulty and exposes model-specific weaknesses in long-horizon planning and step-level execution.
• Offline evaluation scores correlate moderately with online performance, indicating that some models generalize well while others fail at specific decision points.