Pixel-space video generation with U-JiT

TL;DR

The paper introduces U-JiT, a UNet-like transformer model for direct pixel-space video generation, avoiding the need for pretrained autoencoders. It achieves state-of-the-art results on Kinetics-600, outperforming both latent and pixel-space baselines in FVD and FID across multiple resolutions. The model demonstrates improved scalability and generative quality as spatiotemporal compression increases.

Why It Matters

Eliminating the need for autoencoders enables fully end-to-end training and avoids reconstruction artifacts, which is crucial for high-fidelity video synthesis. This approach opens new pathways for training generative video models with greater flexibility and fidelity, especially at higher resolutions.

Key Findings

• U-JiT achieves a 27% improvement in FVD over prior state-of-the-art models on the frames-to-video task at 128 resolution.
• Performance gains increase with resolution and compression rate, with improvements over both latent and pixel-space models growing up to 64×64×8 spatiotemporal compression.
• On class-to-video generation, U-JiT reduces FID by 27% compared to latent models and 61% compared to pixel-space models, with corresponding FVD reductions of 9% and 56%.