{
 "id": "epv80g3r7b",
 "slug": "sglang",
 "title": "SGLang",
 "updated": "2026-10-15",
 "topic_path": [
  {
   "id": "technology",
   "label": "Technology and the built world",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology"
  },
  {
   "id": "technology.computing",
   "label": "Computing and digital systems",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing"
  },
  {
   "id": "technology.computing.modern_ai",
   "label": "Modern AI: foundation models, generative AI, and the AI industry",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai"
  },
  {
   "id": "technology.computing.modern_ai.methods",
   "label": "Foundation-model methods and training",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.methods"
  },
  {
   "id": "technology.computing.modern_ai.methods.inference_efficiency",
   "label": "Inference, serving, and efficiency of foundation models",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.methods.inference_efficiency"
  }
 ],
 "geo": [
  {
   "id": "geo.us.t2021.technology.computing.modern_ai.methods",
   "label": "United States · 2021 and later: Foundation-model methods and training",
   "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.methods",
   "path": [
    {
     "id": "geo.us",
     "label": "United States",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us"
    },
    {
     "id": "geo.us.t2021",
     "label": "United States · 2021 and later",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021"
    },
    {
     "id": "geo.us.t2021.technology",
     "label": "Technology and the built world",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology"
    },
    {
     "id": "geo.us.t2021.technology.computing",
     "label": "Computing and digital systems",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing"
    },
    {
     "id": "geo.us.t2021.technology.computing.modern_ai",
     "label": "Modern AI: foundation models, generative AI, and the AI industry",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai"
    },
    {
     "id": "geo.us.t2021.technology.computing.modern_ai.methods",
     "label": "Foundation-model methods and training",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2021.technology.computing.modern_ai.methods"
    }
   ]
  }
 ],
 "excerpt": "SGLang is an open-source serving framework for large language and multimodal models, introduced in December 2023 by Lianmin Zheng and colleagues at UC Berkeley and Stanford.",
 "snippet": "SGLang is an open-source serving framework for large language and multimodal models, introduced in December 2023 by Lianmin Zheng and colleagues at UC Berkeley and Stanford.",
 "node": "technology.computing.modern_ai.methods.inference_efficiency",
 "markdown": "# SGLang\n\n**SGLang** is an open-source serving framework for large language models and multimodal models, originally built as a system for efficiently executing structured language model programs, meaning applications that chain multiple generation calls with control flow and structured inputs and outputs. It was introduced in December 2023 by Lianmin Zheng and colleagues at UC Berkeley and Stanford, published at NeurIPS 2024, and is distributed under the Apache-2.0 license from the sgl-project repository.<sup>[1](https://arxiv.org/abs/2312.07104)</sup><sup> • </sup><sup>[2](https://mast.stanford.edu/pubs/sglang_efficient_execution_of_structured_language_model_programs/)</sup><sup> • </sup><sup>[3](https://github.com/sgl-project)</sup>\n\n| Key fact | Detail |\n|---|---|\n| First release | arXiv paper dated 12 December 2023; NeurIPS 2024 publication<sup>[1](https://arxiv.org/abs/2312.07104)</sup><sup> • </sup><sup>[2](https://mast.stanford.edu/pubs/sglang_efficient_execution_of_structured_language_model_programs/)</sup> |\n| Core design | A frontend language with primitives for generation and parallelism control, plus a runtime with RadixAttention for KV cache reuse and compressed finite state machines for structured output decoding<sup>[1](https://arxiv.org/abs/2312.07104)</sup> |\n| Reported throughput | Up to 6.4x higher throughput than state-of-the-art inference systems on agent control, logical reasoning, few-shot benchmarks, JSON decoding, RAG pipelines, and multi-turn chat<sup>[1](https://arxiv.org/abs/2312.07104)</sup> |\n| Cache hierarchy | RadixAttention caches prefix KV caches in GPU memory; HiCache extends this to host memory (L2) and distributed storage (L3) with backends such as Mooncake, 3FS, NIXL, and AIBrix KVCache<sup>[4](https://github.com/Schilings/sglang/blob/main/docs/advanced_features/hicache_design.md)</sup> |\n| Hardware | NVIDIA and AMD GPUs, plus TPU support through SGL-JAX<sup>[5](https://mlai.qa/blog/vllm-vs-sglang-vs-tensorrt-llm/)</sup> |\n| Commercialization | Spun out as RadixArk in January 2026 at roughly a $400 million valuation in a round led by Accel; Ying Sheng is co-founder and CEO<sup>[6](https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes/)</sup> |\n| Notable users | Companies including xAI and Cursor; deployed in Chatbot Arena to serve open-weight models<sup>[6](https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes/)</sup><sup> • </sup><sup>[7](https://par.nsf.gov/servlets/purl/10524135)</sup> |\n\n## Origin and design\n\nSGLang originated in 2023 inside the UC Berkeley lab of [Ion Stoica](https://www.edgepedia.org/ion-stoica), a professor at UC Berkeley and co-founder of [Databricks](https://www.edgepedia.org/databricks).<sup>[6](https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes/)</sup> The original paper framed the problem as one of programming and execution: LLM applications increasingly need multiple generation calls, advanced prompting, control flow, and structured inputs and outputs, but efficient systems for running such programs were lacking. The system therefore has two parts. The frontend simplifies programming with primitives for generation and parallelism control, and programs can be traced and compiled for further optimization. The runtime accelerates execution with RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding.<sup>[1](https://arxiv.org/abs/2312.07104)</sup><sup> • </sup><sup>[8](https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf)</sup>\n\nAccording to the paper, when running both LLMs and models that handle multiple modalities, the system reached throughputs as much as 6.4 times greater than the most advanced competing inference engines, measured over workloads such as controlling agents, reasoning logically, benchmarks for learning from few examples, decoding JSON, RAG pipelines, and chats spanning multiple turns.<sup>[1](https://arxiv.org/abs/2312.07104)</sup> SGLang was subsequently deployed in Chatbot Arena to serve open-weight models.<sup>[7](https://par.nsf.gov/servlets/purl/10524135)</sup>\n\n## RadixAttention and the cache hierarchy\n\nIn LLM inference, the prefill phase converts input sequences into a key-value (KV) cache used for decoding. When multiple requests share the same prefix, that [KV cache](https://www.edgepedia.org/kv-cache) is identical, so caching and reusing it avoids redundant computation. RadixAttention leverages idle GPU memory to cache and reuse prefix KV caches in a radix tree, where each node corresponds to the KV cache of a consecutive span of tokens and a path from root to leaf represents a request's prefix; shared prefixes reuse the same nodes.<sup>[4](https://github.com/Schilings/sglang/blob/main/docs/advanced_features/hicache_design.md)</sup>\n\n**HiCache** extends this idea beyond GPU memory. Inspired by the three-level cache design of CPUs, it treats GPU memory as L1, host memory as L2, and distributed storage as L3. The L1 and L2 caches are private to each inference instance, while the L3 cache is shared across the cluster, and the system integrates distributed cache backends including Mooncake, 3FS, NIXL, and AIBrix KVCache. Metadata is organized in a HiRadixTree that records where each span's KV cache is stored, and prefetching from L3 can be tuned with best_effort, wait_complete, or timeout termination strategies.<sup>[4](https://github.com/Schilings/sglang/blob/main/docs/advanced_features/hicache_design.md)</sup> The project's blog reports multi-turn prefix cache hit rates near 98% with a Mooncake L3 tier on DeepSeek-V4-Flash.<sup>[9](https://www.sglang.io/blog/unified-radix-cache)</sup>\n\n## Runtime features\n\nThe current framework describes itself as a high-performance serving framework for large language and multimodal models, with a feature set that includes a zero-overhead CPU scheduler with continuous batching, paged attention, chunked prefill, prefill-decode disaggregation, speculative decoding, structured output, quantization (FP4/FP8/INT4/AWQ/GPTQ), multi-LoRA batching, and tensor, pipeline, expert, and data parallelism.<sup>[10](https://docs.sglang.com.cn/)</sup><sup> • </sup><sup>[3](https://github.com/sgl-project)</sup> It serves OpenAI-compatible endpoints.<sup>[11](https://www.sglang.io/)</sup> A native Rust serving layer with an OpenAI-compatible API server and prefill-decode disaggregation support shipped in the v0.5.19 release, which also made a unified radix tree the default cache for every model and added a persistent \"Lean\" attention kernel for AMD MI300X and MI355X GPUs.<sup>[12](https://github.com/sgl-project/sglang/releases?ref=codersera.com)</sup>\n\n## Position among inference engines\n\nBy 2025, inference software had consolidated around three open-source frameworks: vLLM, SGLang, and NVIDIA TensorRT-LLM.<sup>[13](https://artificialanalysis.ai/downloads/state-of-ai/2025/2025-Year-End-Artificial-Analysis-State-of-AI-Highlights-Report.pdf)</sup> All three now ship the same baseline features, including continuous batching, paged KV cache, chunked prefill, prefix caching, speculative decoding, structured output, and multi-LoRA support, so the practical differences lie in prefix-reuse design, hardware breadth, and governance.<sup>[5](https://mlai.qa/blog/vllm-vs-sglang-vs-tensorrt-llm/)</sup> SGLang is characterized as strongest on prefix-heavy and agentic workloads and on day-0 support for newly released frontier models, while vLLM spans the widest set of hardware targets and [TensorRT-LLM](https://www.edgepedia.org/tensorrt-llm) is NVIDIA-only.<sup>[5](https://mlai.qa/blog/vllm-vs-sglang-vs-tensorrt-llm/)</sup> An empirical study of open-source repositories found vLLM the most visible framework in popularity and adoption among the five serving frameworks studied, with SGLang among the most widely used.<sup>[14](https://arxiv.org/abs/2608.03036)</sup>\n\n**Benchmark interpretation.** Published head-to-head numbers vary widely with model, precision, prompt length, concurrency, and engine version, and independent comparisons disagree with each other by double-digit percentages depending on those variables.<sup>[15](https://ai-solutions.wiki/comparisons/vllm-vs-tensorrt-llm-vs-sglang/)</sup> SGLang did not submit to the audited [MLPerf Inference](https://www.edgepedia.org/mlperf-inference) v6.0 round, so no audited three-way comparison exists.<sup>[5](https://mlai.qa/blog/vllm-vs-sglang-vs-tensorrt-llm/)</sup> One documented pitfall is that benchmark gains can be an artifact of prefix sharing: a staging trace that replayed a captured production day showed SGLang 29% ahead of vLLM in tokens per second, but the gap collapsed to about 3% in production once per-tenant prompts replaced thousands of identical system prompts, because the benchmark had effectively measured the radix cache rather than the engine.<sup>[16](https://shirokoff.ca/blog/vllm-sglang-tensorrt-llm-compared)</sup>\n\n## RadixArk and commercialization\n\nIn January 2026 the project spun out as RadixArk, a commercial company valued at about $400 million in a funding round led by Accel, with earlier angel investment from figures including Intel CEO Lip-Bu Tan. Ying Sheng, a key SGLang contributor formerly at xAI and Databricks, became co-founder and CEO. RadixArk continues to develop SGLang as an open-source engine, is building Miles, a specialized framework for reinforcement learning, and has begun charging fees for hosting services.<sup>[6](https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes/)</sup>\n\n## References\n\n1. [Lianmin Zheng et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. arXiv.](https://arxiv.org/abs/2312.07104)\n2. [SGLang: Efficient Execution of Structured Language Model Programs, Stanford MAST Lab](https://mast.stanford.edu/pubs/sglang_efficient_execution_of_structured_language_model_programs/)\n3. [sgl-project on GitHub](https://github.com/sgl-project)\n4. [HiCache System Design and Optimization, SGLang documentation](https://github.com/Schilings/sglang/blob/main/docs/advanced_features/hicache_design.md)\n5. [vLLM vs SGLang vs TensorRT-LLM 2026: Inference Engine Comparison, mlai.qa](https://mlai.qa/blog/vllm-vs-sglang-vs-tensorrt-llm/)\n6. [Sources: Project SGLang spins out as RadixArk with $400M valuation as inference market explodes, TechCrunch (21 January 2026)](https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes/)\n7. [SGLang: Efficient Execution of Structured Language Model Programs (NSF PAR copy)](https://par.nsf.gov/servlets/purl/10524135)\n8. [SGLang paper, NeurIPS 2024 proceedings](https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf)\n9. [Unified Radix Cache in SGLang, SGLang Blog](https://www.sglang.io/blog/unified-radix-cache)\n10. [SGLang documentation](https://docs.sglang.com.cn/)\n11. [SGLang website](https://www.sglang.io/)\n12. [Releases, sgl-project/sglang on GitHub](https://github.com/sgl-project/sglang/releases?ref=codersera.com)\n13. [2025 Year-End Artificial Analysis State of AI Highlights Report](https://artificialanalysis.ai/downloads/state-of-ai/2025/2025-Year-End-Artificial-Analysis-State-of-AI-Highlights-Report.pdf)\n14. [LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs. arXiv.](https://arxiv.org/abs/2608.03036)\n15. [vLLM vs TensorRT-LLM vs SGLang, AI Solutions Wiki](https://ai-solutions.wiki/comparisons/vllm-vs-tensorrt-llm-vs-sglang/)\n16. [vLLM vs SGLang vs TensorRT-LLM: How to Actually Choose, shirokoff.ca](https://shirokoff.ca/blog/vllm-sglang-tensorrt-llm-compared)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Inference, serving, and efficiency of foundation models*\n\n*Initially written Sep 17, 2026 · Reviewed: Oct 15, 2026 · Edited: Oct 11, 2026; Oct 15, 2026 · Last review: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n",
 "same_as": [],
 "url": "https://www.edgepedia.org/sglang",
 "markdown_url": "https://www.edgepedia.org/sglang.md",
 "license": {
  "name": "Edgepedia Community License 1.0",
  "url": "https://www.edgepedia.org/edgepedia/license",
  "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.",
  "spdx": "LicenseRef-Edgepedia-Community-1.0"
 },
 "credit": "\"SGLang\", Edgepedia (EdgeChat), https://www.edgepedia.org/sglang. Edgepedia Community License 1.0.",
 "credit_md": "\"[SGLang](https://www.edgepedia.org/sglang)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/sglang](https://www.edgepedia.org/sglang). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).",
 "credit_html": "\"<a href=\"https://www.edgepedia.org/sglang\">SGLang</a>\", Edgepedia (EdgeChat), <a href=\"https://www.edgepedia.org/sglang\">https://www.edgepedia.org/sglang</a>. <a href=\"https://www.edgepedia.org/edgepedia/license\">Edgepedia Community License 1.0</a>.",
 "speakable": "SGLang is an open-source serving framework for large language and multimodal models, introduced in December 2023 by Lianmin Zheng and colleagues at UC Berkeley and Stanford."
}