Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference

TL;DR

This paper proposes a method to optimize memory usage in AI models by permuting operator orders, achieving exact realization and certified optimality. The approach outperforms traditional methods, especially in branching networks. It also explores selective recomputation with per-operator caps.

Why It Matters

The proposed method has practical significance as it enables efficient deployment of AI models on small devices with limited memory. Optimizing memory usage can lead to improved performance and reduced latency. This is crucial for real-time inference applications.

Key Findings

• exact realization of modeled peak memory
• certified optimality with forward and reverse trap lower bounds
• selective recomputation with per-operator caps is strongly NP-hard