The Symmetry of Differential Neural Tangent Kernel under the Orthogonal Group

TL;DR

This paper studies the symmetry properties of the differential neural tangent kernel (DNTK) under orthogonal group actions for both two-layer and deep neural networks. It establishes that the infinite-width DNTK inherits symmetries from the network architecture and activation functions, leading to zonal symmetry on the unit sphere and identical spectral properties for operators in the same group orbit.

Why It Matters

Understanding the symmetry of DNTK helps explain the generalization and convergence behavior of overparameterized neural networks, especially in physics-informed settings. The derived spectral decay rates enable better theoretical analysis of learning dynamics and model performance.

Key Findings

• The infinite DNTK exhibits invariance under orthogonal group actions for both shallow and deep networks, regardless of activation type or bias presence.
• Integral operators induced by group orbit elements share the same spectrum and have reproducing kernel Hilbert spaces related by orthogonal transformations.
• The fixed-point DNTK under orthogonal symmetry shows zonal symmetry on the unit sphere, enabling derivation of eigenvalue decay rates for RePU-activated shallow networks with zero bias initialization.