vLLM

High-throughput open-source inference and serving engine for LLMs.

vLLM is an open-source library for fast LLM inference and serving, known for PagedAttention memory management that dramatically improves throughput. It supports continuous batching, tensor/pipeline parallelism, quantization, and an OpenAI-compatible server, and is a de facto standard for self-hosted model serving.