FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

TL;DR

FlowMaker automates the selection of optimal generative AI workflow variants—comprising task sequences, model choices, and deployment settings—by dynamically adapting to cluster load and resource availability. It uses a constraint-guided pruning algorithm to make fast, on-the-fly decisions that balance cost, latency, and throughput.

Why It Matters

Deploying generative AI workflows efficiently at scale is challenging due to the interplay of model selection, workflow design, and infrastructure constraints. FlowMaker enables real-time optimization that significantly improves system utilization and reduces operational costs, making large-scale AI services more viable.

Key Findings

• Joint optimization of workflow structure, model assignment, and deployment configuration leads to substantial improvements in goodput and cost efficiency.
• On-the-fly reuse of deployed large language models, guided by current load and resource availability, is essential for minimizing cost without sacrificing performance.
• FlowMaker's load-aware pruning algorithm enables rapid decision-making under dynamic conditions, outperforming existing systems like Murakkab in both cost reduction and throughput.