Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections

TL;DR

The paper provides exact geometric descriptions of minimal-norm solutions for univariate two-layer ReLU networks in binary classification. It shows that skip connections preserve optimal function-space solutions while eliminating suboptimal critical points in parameter space. With unpenalized biases, solutions are piecewise-affine with kinks at label changes; with penalized biases, solutions are sparse and have one kink per same-label interval.

Why It Matters

Understanding the structure of optimal classifiers in simple neural networks helps explain why certain architectures generalize better and informs the design of more robust training procedures. The findings suggest that skip connections can fundamentally improve optimization landscapes without altering the best possible model, which is valuable for both theory and practice.

Key Findings

• Minimal-norm interpolators with unpenalized biases are continuous piecewise-affine functions with kinks at every label switch and correct convexity.
• When biases are penalized, the unique minimizer has exactly one kink in each intermediate segment of consecutive same-label points, making it a sparse positive-margin classifier.
• Adding a skip connection preserves the optimal function but ensures all KKT points are globally optimal, eliminating suboptimal critical points in parameter space.