DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

TL;DR

DreamStream is a generative, closed-loop driving simulator that improves sim-to-real fidelity by preserving policy-relevant scene features through a distilled autoregressive video model guided by traffic layout. It introduces FDπ, a metric based on features from real-world end-to-end policies, to better assess simulation quality than traditional perceptual metrics. The platform enables the Navhard-CL benchmark, which tests policies under adversarial conditions and reveals critical failure modes.

Why It Matters

Accurate simulation is essential for safe deployment of end-to-end driving systems, but prior simulators fail to preserve decision-critical scene features. DreamStream addresses this by aligning simulation with actual policy behavior, enabling more reliable evaluation and discovery of safety-critical flaws. This advances the development of robust autonomous driving systems through better testing infrastructure.

Key Findings

• DreamStream's autoregressive video model, distilled from a large pretrained model using traffic layout guidance, maintains policy-relevant features like scene layout and object dynamics while varying visual appearance.
• FDπ, a new metric based on Fréchet distance over scene-context features extracted from public E2E policies, better captures sim-to-real fidelity than standard perceptual metrics like FID.
• Navhard-CL, built on DreamStream, introduces interactive, adversarial scenarios and weather variations, exposing policy weaknesses such as scorer bias and poor recovery behavior that prior benchmarks miss.