{ "id": "epnzq5gd2t", "slug": "vllm", "title": "VLLM", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai" }, { "id": "technology.computing.modern_ai.industry", "label": "AI companies, people, and products", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry" }, { "id": "technology.computing.modern_ai.industry.ai_products_assistants", "label": "AI products and assistants", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry.ai_products_assistants" }, { "id": "technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools", "label": "AI agents and developer tools", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools" } ], "geo": [ { "id": "geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools", "label": "United States · 2021 and later: AI agents and developer tools", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools", "path": [ { "id": "geo.us", "label": "United States", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us" }, { "id": "geo.us.t2021", "label": "United States · 2021 and later", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021" }, { "id": "geo.us.t2021.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology" }, { "id": "geo.us.t2021.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing" }, { "id": "geo.us.t2021.technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai" }, { "id": "geo.us.t2021.technology.computing.modern_ai.industry", "label": "AI companies, people, and products", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.industry" }, { "id": "geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants", "label": "AI products and assistants", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants" }, { "id": "geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools", "label": "AI agents and developer tools", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools" } ] } ], "excerpt": "vLLM is an open-source library and server for high-throughput inference and serving of large language models, developed at UC Berkeley and known for its PagedAttention algorithm.", "snippet": "vLLM is an open-source library and server for high-throughput inference and serving of large language models, developed at UC Berkeley and known for its PagedAttention algorithm.", "node": "technology.computing.modern_ai.industry.ai_products_assistants.ai-agents-and-developer-tools", "markdown": "# VLLM\n\n**vLLM** is an open-source library and server for high-throughput inference and serving of large language models (LLMs), originally developed in the Sky Computing Lab at UC Berkeley and now maintained by a community of over 2000 contributors from many dozens of academic institutions and companies.[1](https://docs.vllm.ai/en/stable/) Its central technique is [PagedAttention](https://www.edgepedia.org/pagedattention), an attention algorithm that manages the key-value (KV) cache with near-zero memory waste by treating GPU memory like an operating system's virtual memory, allowing significantly larger batch sizes and high serving throughput.[2](https://arxiv.org/abs/2309.06180) • [3](https://docs.nvidia.com/deeplearning/frameworks/vllm-release-notes/overview.html)\n\n| Key fact | Detail |\n|---|---|\n| What it is | Open-source LLM inference and serving engine with an OpenAI-compatible API server, started via `vllm serve`[1](https://docs.vllm.ai/en/stable/) • [4](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) |\n| Core algorithm | PagedAttention stores attention keys and values in non-contiguous paged memory, achieving near-zero KV cache waste and flexible cache sharing within and across requests[2](https://arxiv.org/abs/2309.06180) |\n| Original results | Throughput of popular LLMs improved 2-4x at the same latency versus FasterTransformer and Orca; gains grow with longer sequences, larger models, and more complex decoding[2](https://arxiv.org/abs/2309.06180) |\n| 2023 blog benchmarks | 8.5x-15x higher throughput than HuggingFace Transformers and 3.3x-3.5x higher than TGI for requests with three parallel output completions[5](https://vllm-project.github.io/2023/06/20/vllm.html) |\n| Model coverage | 200+ HuggingFace model architectures, including decoder-only LLMs, mixture-of-expert models, hybrid attention and state-space models, multi-modal models, and embedding models[1](https://docs.vllm.ai/en/stable/) |\n| Hardware | NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs, plus plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Apple Silicon, and others[1](https://docs.vllm.ai/en/stable/) |\n| Publication | \"Efficient Memory Management for Large Language Model Serving with PagedAttention\", SOSP 2023 (arXiv September 2023)[2](https://arxiv.org/abs/2309.06180) |\n\n## The KV cache problem and PagedAttention\n\nHigh-throughput LLM serving requires batching many requests at once, but the [KV cache](https://www.edgepedia.org/kv-cache), the memory holding each request's attention keys and values, is large and grows and shrinks dynamically as sequences lengthen. When managed inefficiently, this memory is wasted by fragmentation and redundant duplication, which limits batch size.[2](https://arxiv.org/abs/2309.06180)\n\nPagedAttention addresses this by borrowing the classical virtual memory and paging techniques of operating systems: attention keys and values are stored in non-contiguous paged blocks rather than one contiguous reservation per request.[2](https://arxiv.org/abs/2309.06180) The result is near-zero waste in KV cache memory and flexible sharing of cache blocks within and across requests, which further reduces memory usage.[2](https://arxiv.org/abs/2309.06180) Sharing matters for complex sampling algorithms: PagedAttention's memory sharing cuts the memory usage of parallel sampling and beam search by up to 55%.[5](https://vllm-project.github.io/2023/06/20/vllm.html)\n\n**Scheduling.** vLLM also uses continuous batching, in which new requests are injected into the running batch as others finish rather than waiting for a whole batch to complete; a vLLM engineering post by Cade Daniel and colleagues describes this as enabling 23x throughput in LLM inference while reducing p50 latency.[1](https://docs.vllm.ai/en/stable/) [Chunked prefill](https://www.edgepedia.org/chunked-prefill) and prefix caching are supported alongside it.[1](https://docs.vllm.ai/en/stable/) When the server hits KV cache limits, vLLM preempts requests to free cache space; in the V1 architecture the default preemption mode is RECOMPUTE rather than SWAP, because recomputation has lower overhead there.[6](https://docs.vllm.ai/en/v0.12.0/configuration/optimization/)\n\n## Measured performance\n\nThe original paper's evaluation showed vLLM improving the throughput of popular LLMs by 2-4x at the same latency compared with the then state-of-the-art systems [FasterTransformer](https://www.edgepedia.org/fastertransformer) and Orca, with the improvement more pronounced for longer sequences, larger models, and more complex decoding algorithms.[2](https://arxiv.org/abs/2309.06180) The 2023 announcement blog compared vLLM against HuggingFace Transformers and HuggingFace Text Generation Inference (TGI) on requests asking for three parallel output completions, reporting 8.5x-15x higher throughput than [Transformers](https://www.edgepedia.org/transformers) and 3.3x-3.5x higher than TGI.[5](https://vllm-project.github.io/2023/06/20/vllm.html)\n\nLater independent work has examined the comparison with TGI in more detail. A benchmark study using LLaMA-2 models from 7B to 70B parameters found vLLM achieving up to 24x higher throughput than TGI under high-concurrency workloads, while TGI showed lower tail latencies for interactive single-user scenarios; the authors concluded that vLLM excels in high-throughput batch processing and TGI in latency-sensitive interactive applications with moderate concurrency.[7](https://arxiv.org/abs/2511.17593) A survey of LLM serving adoption in open-source software found vLLM the most visible framework in popularity and adoption.[8](https://arxiv.org/abs/2608.03036)\n\nResearch continues on alternatives to the paging approach itself: the vAttention project proposes dynamic memory management without PagedAttention and reports throughput improvements of up to 1.23x over PagedAttention-based kernels of [FlashAttention](https://www.edgepedia.org/flashattention) and [FlashInfer](https://www.edgepedia.org/flashinfer), while supporting various attention kernels out of the box.[9](https://arxiv.org/abs/2405.04437)\n\n## Architecture and interfaces\n\nvLLM exposes two primary interfaces. The offline interface is the `LLM` class for programmatic batch inference; the online interface is an API server started with `vllm serve`, which handles HTTP requests against an OpenAI-compatible API, performs input processing such as tokenization and multi-modal data loading, and streams results back to clients.[4](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) Anthropic Messages API and gRPC support are also provided.[1](https://docs.vllm.ai/en/stable/)\n\nThe V1 engine uses a multi-process architecture to separate concerns and maximize throughput, with the API server process distinct from the engine core that handles model inference and asynchronous request processing.[4](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md) Memory use is controlled by the `gpu_memory_utilization` setting, under which vLLM pre-allocates GPU cache; on a device with 100 GiB total and 50 GiB free after loading weights, the default marks 90% of the 50 GiB free as usable, leaving a 5 GiB margin regardless of total device memory.[10](https://docs.vllm.cc/en/latest/getting_started/installation/ai_accelerator.html)\n\n## Features and hardware support\n\nThe feature set includes quantization schemes such as FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, and GGUF; optimized attention kernels including FlashAttention, FlashInfer, FlashMLA, and Triton; speculative decoding with n-gram, suffix, EAGLE, and DFlash methods; and tensor, pipeline, data, expert, and [context parallelism](https://www.edgepedia.org/context-parallelism) for distributed inference.[1](https://docs.vllm.ai/en/stable/) [Structured outputs](https://www.edgepedia.org/structured-outputs) are generated with xgrammar or guidance, and efficient multi-LoRA support covers dense and MoE layers.[1](https://docs.vllm.ai/en/stable/)\n\nSupported model families span decoder-only LLMs such as Llama, Qwen, and Gemma; mixture-of-expert models such as Mixtral, DeepSeek-V3, and Qwen-MoE; hybrid attention and state-space models such as [Mamba](https://www.edgepedia.org/mamba-deep-learning-architecture); multi-modal models such as LLaVA, Qwen-VL, and Pixtral; and embedding, reward, and classification models.[1](https://docs.vllm.ai/en/stable/) Beyond NVIDIA and AMD GPUs and x86/ARM/PowerPC CPUs, hardware plugins target Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, and MetaX GPU.[1](https://docs.vllm.ai/en/stable/) vLLM does not support Windows natively.[11](https://docs.vllm.cc/en/latest/getting_started/installation/gpu.html)\n\n## Ecosystem\n\nvLLM sits at the center of a family of companion projects, including AIBrix for AI infrastructure on [Kubernetes](https://www.edgepedia.org/kubernetes), GuideLLM for performance evaluation, LLM Compressor for model quantization, Production Stack for deployment, Speculators for speculative decoding, and vLLM Omni for omni-modality models.[12](https://vllm.ai/) Hugging Face Inference Endpoints offers managed serving of models via vLLM with an OpenAI Completion API interface.[13](https://docs.vllm.ai/en/latest/deployment/frameworks/hf_inference_endpoints/) The project originated in the Sky Computing Lab at UC Berkeley, whose co-director Joseph E. Gonzalez is a professor of electrical engineering and computer science there; the lab reports vLLM as the top open-source project on GitHub for 2025.[1](https://docs.vllm.ai/en/stable/) • [14](https://sky.cs.berkeley.edu/news/)\n\n## References\n\n1. [Welcome to vLLM, vLLM documentation](https://docs.vllm.ai/en/stable/)\n2. [Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv.](https://arxiv.org/abs/2309.06180)\n3. [vLLM Overview, NVIDIA Docs](https://docs.nvidia.com/deeplearning/frameworks/vllm-release-notes/overview.html)\n4. [vLLM architecture overview, vllm-project/vllm GitHub](https://github.com/vllm-project/vllm/blob/main/docs/design/arch_overview.md)\n5. [vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, vLLM Blog (June 2023)](https://vllm-project.github.io/2023/06/20/vllm.html)\n6. [Optimization and Tuning, vLLM documentation v0.12.0](https://docs.vllm.ai/en/v0.12.0/configuration/optimization/)\n7. [Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI, arXiv](https://arxiv.org/abs/2511.17593)\n8. [LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs, arXiv](https://arxiv.org/abs/2608.03036)\n9. [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention, arXiv](https://arxiv.org/abs/2405.04437)\n10. [Other AI accelerators, vLLM documentation](https://docs.vllm.cc/en/latest/getting_started/installation/ai_accelerator.html)\n11. [GPU installation, vLLM documentation](https://docs.vllm.cc/en/latest/getting_started/installation/gpu.html)\n12. [vLLM project site](https://vllm.ai/)\n13. [Hugging Face Inference Endpoints, vLLM documentation](https://docs.vllm.ai/en/latest/deployment/frameworks/hf_inference_endpoints/)\n14. [News, UC Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/news/)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › AI companies, people, and products › AI products and assistants › AI agents and developer tools*\n\n*Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 11, 2026; Oct 15, 2026 · Last review: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/vllm", "markdown_url": "https://www.edgepedia.org/vllm.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"VLLM\", Edgepedia (EdgeChat), https://www.edgepedia.org/vllm. Edgepedia Community License 1.0.", "credit_md": "\"[VLLM](https://www.edgepedia.org/vllm)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/vllm](https://www.edgepedia.org/vllm). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"VLLM\", Edgepedia (EdgeChat), https://www.edgepedia.org/vllm. Edgepedia Community License 1.0.", "speakable": "vLLM is an open-source library and server for high-throughput inference and serving of large language models, developed at UC Berkeley and known for its PagedAttention algorithm." }