JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs

TL;DR

JOVE is a framework that dynamically assigns LLMs to subtasks in a reasoning pipeline and selects which intermediate results to verify, balancing immediate execution costs with long-term learning gains. It uses online optimization and learning to adapt to uncertain LLM performance, reducing both cost and latency while maintaining high accuracy. The system achieves sublinear regret in learning task-specific LLM quality and outperforms baselines across multiple benchmarks.

Why It Matters

It enables efficient and reliable execution of complex reasoning tasks using heterogeneous LLMs, making large-scale AI reasoning more practical and cost-effective. By intelligently managing verification and allocation, JOVE reduces resource usage without sacrificing accuracy, which is crucial for real-world deployment.

Key Findings

• JOVE jointly optimizes executor selection and verification choices using mixed-integer linear programming with an information-gain bonus.
• Online learning updates LLM quality estimates based on verification feedback, achieving sublinear regret in quality learning.