TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

TL;DR

TabPack is a method for building efficient ensembles of multilayer perceptrons for tabular data that automatically samples diverse hyperparameters during training. It selects strong models on the fly without requiring manual tuning, enabling strong performance with default settings. The approach reduces both computational cost and effort compared to traditional hyperparameter optimization.

Why It Matters

TabPack significantly lowers the barrier to achieving high performance in tabular deep learning by eliminating the need for extensive hyperparameter tuning. This makes state-of-the-art results accessible on consumer hardware, accelerating experimentation and deployment. It enables faster iteration and broader adoption of deep learning for tabular tasks.

Key Findings

• TabPack trains multiple MLPs in parallel with randomly sampled hyperparameters from predefined ranges.
• It dynamically selects ensemble members during training based on performance, avoiding post-hoc selection.
• With default configurations, TabPack matches or exceeds the performance of tuned baselines on medium-to-large datasets.