Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox

TL;DR

This paper introduces Foyer, a self-hosted personal inbox agent with a rigorous evaluation protocol using human-annotated ground truth before training. It compares typed decision models, classical baselines, and generative LLMs on multilingual email routing, revealing that accuracy alone is misleading—cost-aware thresholds and contact graph context are critical for real-world performance.

Why It Matters

The study exposes the flaws in standard email agent evaluation, showing that high accuracy doesn't translate to better actionability. It demonstrates that contact-level context and cost-sensitive thresholds are essential for practical deployment, guiding future design of personal AI assistants.

Key Findings

• Production keyword rules perform poorly (28% on message type), with the most prominent field carrying no useful information.
• TF-IDF + logistic regression achieves the highest accuracy (~79-90%) with ~200 labels, but fails on rare classes and underperforms on actionability.
• Sender relationship is best inferred from contact history (98.6% accuracy), not single messages—highlighting the need for contact graph integration over prompt engineering.