Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustness

TL;DR

This study evaluates how model design, adaptation methods, and deployment settings impact log anomaly detection using large language models. It finds that adaptation strategies significantly affect performance, scaling benefits vary by dataset, and low-bit quantization maintains accuracy while reducing resource use. Robustness tests show sensitivity to different noise types, revealing trade-offs between accuracy and reliability.

Why It Matters

Organizations deploying LLMs for log monitoring need practical guidance on balancing detection quality, computational cost, and resilience to real-world data imperfections. This work provides evidence-based recommendations for model selection and configuration beyond simple accuracy metrics.

Key Findings

• Different adaptation strategies lead to large variations in detection performance, with some methods significantly outperforming others across datasets.
• Increasing model size improves detection accuracy on some datasets but yields diminishing returns or no gains on others, indicating dataset-specific scaling behavior.
• Low-bit quantization preserves detection accuracy in most tested configurations while drastically reducing inference cost, and models show varying sensitivity to structural, semantic, and label noise.