Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

TL;DR

The paper derives precise asymptotic expressions for generalization error in kernel ridge regression under anisotropic Gaussian data with power-law decaying covariance. It reveals how the decay exponent α reshapes learning curves: weak anisotropy preserves some high-dimensional features, while strong anisotropy reduces effective dimension and alters variance and bias behavior. For single-index targets, alignment with principal directions determines learning efficiency under anisotropy.

Why It Matters

Understanding how data geometry affects kernel methods is crucial for designing robust models in high-dimensional settings. The results provide a theoretical foundation for predicting performance in real-world data with structured covariance, such as in neuroscience or image data. This enables better hyperparameter tuning and model selection under anisotropic conditions.

Key Findings

• The generalization error depends critically on the power-law decay exponent α of the input covariance, which governs the effective dimensionality.
• For α < 1, variance peaks at integer sample complexities but is damped as α increases; bias transitions occur at fractional sample complexities when targets align with principal directions.
• For α > 1, the effective dimension becomes constant, variance becomes sample-size independent under ridgeless interpolation, and bias exhibits abrupt transitions depending on target decay rate.