VLLM

vLLM is an open-source library and server for high-throughput inference and serving of large language models (LLMs), originally developed in the Sky Computing Lab at UC Berkeley and now maintained by a community of over 2000 contributors from many dozens of academic institutions and companies.¹ Its central technique is PagedAttention, an attention algorithm that manages the key-value (KV) cache with near-zero memory waste by treating GPU memory like an operating system's virtual memory, allowing significantly larger batch sizes and high serving throughput.^{2 • 3}

The KV cache problem and PagedAttention

High-throughput LLM serving requires batching many requests at once, but the KV cache, the memory holding each request's attention keys and values, is large and grows and shrinks dynamically as sequences lengthen. When managed inefficiently, this memory is wasted by fragmentation and redundant duplication, which limits batch size.²

PagedAttention addresses this by borrowing the classical virtual memory and paging techniques of operating systems: attention keys and values are stored in non-contiguous paged blocks rather than one contiguous reservation per request.² The result is near-zero waste in KV cache memory and flexible sharing of cache blocks within and across requests, which further reduces memory usage.² Sharing matters for complex sampling algorithms: PagedAttention's memory sharing cuts the memory usage of parallel sampling and beam search by up to 55%.⁵

Scheduling. vLLM also uses continuous batching, in which new requests are injected into the running batch as others finish rather than waiting for a whole batch to complete; a vLLM engineering post by Cade Daniel and colleagues describes this as enabling 23x throughput in LLM inference while reducing p50 latency.¹ Chunked prefill and prefix caching are supported alongside it.¹ When the server hits KV cache limits, vLLM preempts requests to free cache space; in the V1 architecture the default preemption mode is RECOMPUTE rather than SWAP, because recomputation has lower overhead there.⁶

Measured performance

The original paper's evaluation showed vLLM improving the throughput of popular LLMs by 2-4x at the same latency compared with the then state-of-the-art systems FasterTransformer and Orca, with the improvement more pronounced for longer sequences, larger models, and more complex decoding algorithms.² The 2023 announcement blog compared vLLM against HuggingFace Transformers and HuggingFace Text Generation Inference (TGI) on requests asking for three parallel output completions, reporting 8.5x-15x higher throughput than Transformers and 3.3x-3.5x higher than TGI.⁵

Later independent work has examined the comparison with TGI in more detail. A benchmark study using LLaMA-2 models from 7B to 70B parameters found vLLM achieving up to 24x higher throughput than TGI under high-concurrency workloads, while TGI showed lower tail latencies for interactive single-user scenarios; the authors concluded that vLLM excels in high-throughput batch processing and TGI in latency-sensitive interactive applications with moderate concurrency.⁷ A survey of LLM serving adoption in open-source software found vLLM the most visible framework in popularity and adoption.⁸

Research continues on alternatives to the paging approach itself: the vAttention project proposes dynamic memory management without PagedAttention and reports throughput improvements of up to 1.23x over PagedAttention-based kernels of FlashAttention and FlashInfer, while supporting various attention kernels out of the box.⁹

Architecture and interfaces

vLLM exposes two primary interfaces. The offline interface is the LLM class for programmatic batch inference; the online interface is an API server started with vllm serve, which handles HTTP requests against an OpenAI-compatible API, performs input processing such as tokenization and multi-modal data loading, and streams results back to clients.⁴ Anthropic Messages API and gRPC support are also provided.¹

The V1 engine uses a multi-process architecture to separate concerns and maximize throughput, with the API server process distinct from the engine core that handles model inference and asynchronous request processing.⁴ Memory use is controlled by the gpu_memory_utilization setting, under which vLLM pre-allocates GPU cache; on a device with 100 GiB total and 50 GiB free after loading weights, the default marks 90% of the 50 GiB free as usable, leaving a 5 GiB margin regardless of total device memory.¹⁰

Features and hardware support

The feature set includes quantization schemes such as FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, and GGUF; optimized attention kernels including FlashAttention, FlashInfer, FlashMLA, and Triton; speculative decoding with n-gram, suffix, EAGLE, and DFlash methods; and tensor, pipeline, data, expert, and context parallelism for distributed inference.¹ Structured outputs are generated with xgrammar or guidance, and efficient multi-LoRA support covers dense and MoE layers.¹

Supported model families span decoder-only LLMs such as Llama, Qwen, and Gemma; mixture-of-expert models such as Mixtral, DeepSeek-V3, and Qwen-MoE; hybrid attention and state-space models such as Mamba; multi-modal models such as LLaVA, Qwen-VL, and Pixtral; and embedding, reward, and classification models.¹ Beyond NVIDIA and AMD GPUs and x86/ARM/PowerPC CPUs, hardware plugins target Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, and MetaX GPU.¹ vLLM does not support Windows natively.¹¹

Ecosystem

vLLM sits at the center of a family of companion projects, including AIBrix for AI infrastructure on Kubernetes, GuideLLM for performance evaluation, LLM Compressor for model quantization, Production Stack for deployment, Speculators for speculative decoding, and vLLM Omni for omni-modality models.¹² Hugging Face Inference Endpoints offers managed serving of models via vLLM with an OpenAI Completion API interface.¹³ The project originated in the Sky Computing Lab at UC Berkeley, whose co-director Joseph E. Gonzalez is a professor of electrical engineering and computer science there; the lab reports vLLM as the top open-source project on GitHub for 2025.^{1 • 14}

References

1. Welcome to vLLM, vLLM documentation
2. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv.
3. vLLM Overview, NVIDIA Docs
4. vLLM architecture overview, vllm-project/vllm GitHub
5. vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, vLLM Blog (June 2023)
6. Optimization and Tuning, vLLM documentation v0.12.0
7. Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI, arXiv
8. LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs, arXiv
9. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention, arXiv
10. Other AI accelerators, vLLM documentation
11. GPU installation, vLLM documentation
12. vLLM project site
13. Hugging Face Inference Endpoints, vLLM documentation
14. News, UC Berkeley Sky Computing Lab

Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › AI companies, people, and products › AI products and assistants › AI agents and developer tools

Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 11, 2026; Oct 15, 2026 · Last review: Oct 15, 2026

© 2026 EdgeChat AI, a subsidiary of Biostate AI. Free to use with credit under the Edgepedia Community License. Developers: read Edgepedia by API or MCP. Embed a reference card.