Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

TL;DR

The paper introduces BOTTLED, a benchmark evaluating LLM agents' ability to create efficient, reusable solutions for repetitive tasks by minimizing cost while maintaining quality. Results show that strong zero-shot performance doesn't guarantee effective bottling, but successful bottling can reduce costs by hundreds of times while preserving most of the original model's accuracy. Opus 5 achieves 82% of its zero-shot F1 at 657× lower cost on one task, outperforming dedicated low-cost models in efficiency.

Why It Matters

Automating the creation of low-cost, reusable solutions for large-scale repetitive tasks is crucial for deploying LLMs economically. This work provides a framework to assess and improve agents' ability to optimize resource use, enabling scalable AI systems without sacrificing performance. It highlights the gap between raw capability and practical efficiency, guiding future agent design.

Key Findings

• Strong zero-shot task performance does not reliably predict success in bottling, with significant variation across models even with similar zero-shot scores.
• Many bottling attempts underperform both zero-shot baselines and small-model distillation approaches, even when using the same token budget.
• Successful bottling can reduce cost by up to 657× while retaining 82% of zero-shot performance, and outperform specialized models like Jev in cost-efficiency.