TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

TL;DR

TACO is a memory-efficient optimizer for fine-tuning large language models that reduces optimizer state storage by over 170 times compared to AdamW8bit while maintaining accuracy and training speed. It achieves this by using a ternary, column-wise sparse update rule based on the largest magnitude entry per column, preserving first-order gradients and enabling full-parameter fine-tuning of 30-32B models on a single 80GB H100 GPU.

Why It Matters

This enables training larger language models on consumer-grade hardware without sacrificing performance, lowering the barrier to entry for researchers and developers. It also improves training efficiency by drastically reducing memory overhead, making large-scale fine-tuning more accessible and sustainable.

Key Findings

• TACO reduces persistent optimizer state memory by 174× compared to AdamW8bit on OPT-13B, from 27.7 GB to 0.16 GB.
• Peak training memory is reduced by 2.9×, from 80.6 GB to 27.5 GB, allowing full-parameter fine-tuning of 30-32B models on a single 80GB H100 GPU.
• TACO maintains accuracy and runtime comparable to AdamW while using a sparse, low-precision update strategy that preserves first-order gradient information.