# TensorRT-LLM

TensorRT-LLM is NVIDIA's open-source library for optimizing inference of large language models (LLMs) and visual generative models on NVIDIA GPUs, first released publicly on 16 August 2023 when its GitHub repository was created.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> It is a serving and optimization stack, not a model: it takes trained models and makes them run faster and cheaper on NVIDIA hardware, through custom GPU kernels, quantization, and scheduling techniques.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup><sup> • </sup><sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup>

| Fact | Detail |
|---|---|
| Creator | NVIDIA<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> |
| First public release | 16 August 2023 (repository creation)<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> |
| Architecture | PyTorch-native, with a high-level Python LLM API and a modular Python runtime<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup><sup> • </sup><sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> |
| Hardware scope | NVIDIA GPUs only, from desktop to data center<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> |
| License | Non-standard "Other" license, not a standard OSI license<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> |
| Ecosystem size (Sept 2026) | ~14,400 stars, ~2,600 forks, ~1,550 open issues<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> |
| API stability | From v1.0, a 3-month deprecation migration period, consistent with semantic versioning<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> |

## What TensorRT-LLM is

NVIDIA describes TensorRT-LLM as an open-source library for high-performance, real-time LLM inference optimization on NVIDIA GPUs, whether on a desktop or in a data center.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> The repository describes it as providing state-of-the-art optimizations, including custom kernels for common inference operations such as attention, GEMMs (general matrix multiplications), and MoE (mixture-of-experts) layers, plus algorithmic runtime optimizations such as prefill-decode disaggregation, wide expert parallelism, and speculative decoding.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup>

Architected on PyTorch, it exposes a high-level Python LLM API that supports deployments from a single GPU to multi-GPU or multi-node setups, and it integrates with [NVIDIA Dynamo](https://www.edgepedia.org/nvidia-dynamo) and Triton Inference Server for serving.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> It is distinct from the models it serves and from NVIDIA the company; this article covers the library as a product.

## How it works

Custom kernels and quantization. The library ships hand-optimized CUDA kernels for attention, GEMM, and MoE operations.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup> On the precision side, it supports FP8, FP4/NVFP4, INT4 AWQ, and INT8 SmoothQuant quantization.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup> According to NVIDIA's documentation, on H100 and later GPUs FP8 quantization can double performance and halve memory consumption compared to 16-bit floating point, with minimal impact on model accuracy; this is a vendor claim, and no independent accuracy evaluation appears in this record.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup> FP4/NVFP4 has native support on [NVIDIA B200](https://www.edgepedia.org/nvidia-b200) (Blackwell) GPUs with optimized FP4 kernels, and NVFP4 is supported only on Blackwell hardware.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup>

Batching and scheduling. In-flight batching dynamically manages request execution, processing context (prefill), and generation (decode) phases together for maximum GPU utilization and reduced latency, alongside paged [KV cache](https://www.edgepedia.org/kv-cache) with block reuse and chunked prefill.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup> CUDA Graphs capture sequences of CUDA operations as a single graph, reducing CPU-side kernel-launch overhead, which NVIDIA notes is particularly impactful in PyTorch-based inference where Python host-side code can be a bottleneck; the company reports that CUDA Graph padding demonstrates up to a 22% end-to-end throughput increase on certain models and hardware.<sup>[4](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)</sup> The Overlap Scheduler launches GPU work for decoding step n+1 without waiting for CPU processing of step n, hiding CPU-bound latency behind GPU computation at the cost of one extra decoding step; it is enabled by default.<sup>[4](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)</sup>

[Speculative decoding](https://www.edgepedia.org/speculative-decoding) and parallelism. The library supports speculative decoding with multiple algorithms including EAGLE, MTP, and NGram, later extended to EAGLE-3 and [multi-token prediction](https://www.edgepedia.org/multi-token-prediction), plus LoRA multi-adapter support, guided decoding, and disaggregated serving (beta) that separates context and generation phases across different GPUs for optimal resource utilization.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup> For large models it supports multi-GPU and multi-node inference with tensor, pipeline, data, expert, context, and wide expert parallelism, the last of these (Wide-EP) including load balancing for large-scale MoE models.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup>

## Launch and version history

The dated launch evidence in this record is the GitHub repository creation on 2023-08-16, which marks the public open-source release.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup>

Version 1.0 marked a consolidation point: NVIDIA describes it as bringing a modular Python runtime, PyTorch-native model authoring, and a stable production LLM API.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> From v1.0 onward the project follows a deprecation policy giving a 3-month migration period during which deprecated APIs continue to work before removal, consistent with semantic versioning.<sup>[1](https://github.com/nvidia/tensorrt-llm)</sup>

## By the numbers: vendor claims, no independent measurements

Every performance figure below is vendor-reported; this record contains no independent benchmark, leaderboard, or third-party evaluation of TensorRT-LLM.

- **FP8 on H100**: NVIDIA's documentation states FP8 quantization can double performance and halve memory consumption versus 16-bit floating point, with minimal accuracy impact.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup>
- **CUDA Graph padding**: up to a 22% end-to-end throughput increase on certain models and hardware.<sup>[4](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)</sup>
- **v1.0**: NVIDIA claims a "record-setting 8X AI inference performance improvement"; no independent measurement is available.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup>
- **Blackwell Ultra**: NVIDIA claims up to 50x better performance and 35x lower cost for agentic AI, a vendor claim tied to its GB300 NVL72 platform.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup>


## Hardware support and the 2025–2026 arc

TensorRT-LLM runs on NVIDIA GPUs only, from desktop to data center; there is no non-NVIDIA hardware support in this record.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> Within the NVIDIA lineup, the precision story tracks the hardware generations: FP8 on Hopper (H100) and later, and NVFP4 natively on Blackwell (B200) with optimized FP4 kernels, supported only on Blackwell.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup>

Through 2025 and 2026 the feature set expanded toward large-scale agentic workloads: disaggregated serving in beta separating context and generation across GPUs, wide expert parallelism for large MoE models, and advanced speculative decoding including EAGLE-3 and multi-token prediction.<sup>[3](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)</sup><sup> • </sup><sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup> NVIDIA reports that cloud providers including Microsoft, CoreWeave, and Oracle Cloud Infrastructure are deploying GB300 NVL72 systems at scale for low-latency and long-context use cases such as agentic coding and coding assistants, using Dynamo and TensorRT-LLM; this is a vendor claim, and deployment scale is not independently verified.<sup>[2](https://developer.nvidia.com/tensorrt-llm)</sup>

## References

1. [NVIDIA/TensorRT-LLM GitHub repository](https://github.com/nvidia/tensorrt-llm)
2. [TensorRT LLM | NVIDIA Developer](https://developer.nvidia.com/tensorrt-llm)
3. [Overview — TensorRT LLM (official documentation)](https://nvidia.github.io/TensorRT-LLM/latest/overview.html)
4. [Architecture Overview — TensorRT LLM](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)

---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Inference, serving, and efficiency of foundation models*

*Initially written Sep 17, 2026 · Reviewed: Sep 19, 2026 · Edited: Sep 19, 2026; Oct 11, 2026 · Last review: Sep 19, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license