56 articles
MLPerf Inference
MLPerf Inference is a benchmark suite published by MLCommons that measures machine learning inference performance across edge and datacenter systems, first run in October 2019.
MLPerf Mobile
MLPerf Mobile is an open-source benchmark suite from MLCommons that measures how fast phones and laptops run AI inference tasks, and how accurate the results are.
MLX
MLX is an open-source array framework for machine learning on Apple silicon, released by Apple in December 2023, with Python and Swift APIs for running and fine-tuning LLMs locally.
Model pruning (LLMs)
Model pruning is a compression technique for large language models that removes weights, neurons, attention heads, or layers so the model needs less memory and compute at inference.
Nucleus sampling
Nucleus sampling, also called top-p sampling, is a language model decoding method that samples from the smallest set of top tokens whose probabilities sum to p.
NVIDIA Dynamo
NVIDIA Dynamo is an open-source distributed inference-serving framework from NVIDIA, launched in March 2025 as the successor to Triton Inference Server, coordinating GPU fleets running large language models.
NVIDIA NIM
NVIDIA NIM is an NVIDIA inference microservice product, launched in March 2024, that packages pretrained AI models as optimized Docker containers for clouds, data centers, and RTX workstations.
NVIDIA Triton Inference Server
NVIDIA Triton Inference Server is open-source software that serves trained AI models from any major framework, including PyTorch, TensorFlow, and TensorRT, on GPU or CPU infrastructure in the cloud, data center, or edge.
On-device LLM inference
On-device LLM inference is the practice of running transformer models directly on consumer hardware like smartphones. This technology is driven by privacy, latency, and cost.
Open WebUI
Open WebUI, formerly Ollama WebUI, is a self-hosted web interface for large language models, connecting to Ollama, vLLM, and OpenAI-compatible servers while bundling chat, RAG, and multi-user management.
PagedAttention
PagedAttention is an attention algorithm for large language model serving that manages the KV cache in fixed-size blocks like virtual memory, proposed at UC Berkeley in 2023.
Petals
Petals is an open-source system for running and fine-tuning large language models collaboratively over the Internet, BitTorrent-style, created by researchers at Yandex Research, the University of Washington, and Hugging Face.
Prefill-decode disaggregation
Prefill-decode disaggregation is a serving architecture that runs prompt processing and token generation for large language models on separate hardware, transferring the KV cache between them.
Prefix/context caching
Prefix/context caching is an inference technique that stores and reuses key-value states for shared prompt prefixes, cutting prefill cost and latency; adopted by major providers in 2024–2025, it discounts cache-hit tokens by 50 to 90%.
Quantization (LLM inference)
Quantization in LLM inference maps a model's weights and activations from 16-bit floating point to 8-bit or 4-bit integers, so the model uses less memory and runs faster.
Quantization (machine learning)
Quantization in machine learning represents a neural network's weights, activations, and caches in lower-precision formats such as 8-bit or 4-bit integers, shrinking memory use and speeding up inference.
Semantic caching
Semantic caching is a caching technique for large language model applications: prompts are compared as embeddings, and similar queries receive a cached response, cutting cost and latency.
SGLang
SGLang is an open-source serving framework for large language and multimodal models, introduced in December 2023 by Lianmin Zheng and colleagues at UC Berkeley and Stanford.
SmoothQuant
SmoothQuant is a training-free post-training quantization method enabling 8-bit weight and activation (W8A8) inference for large language models, introduced by MIT and NVIDIA researchers in November 2022.
SparseGPT
SparseGPT is a one-shot pruning method for large language models, introduced in 2023, that prunes GPT-family models to at least 50% sparsity in a single pass without retraining.
Speculative decoding
Speculative decoding is an inference method for language models in which a smaller drafter proposes several tokens that the large target model verifies in a single pass, speeding generation.
Tensor parallelism
Tensor parallelism (TP) is a model parallelism technique that splits individual layers across GPUs, each holding a slice of weight tensors, popularized by NVIDIA's Megatron-LM in 2019.
TensorRT-LLM
TensorRT-LLM is NVIDIA's open-source library for optimizing large language model inference on NVIDIA GPUs, first released on 16 August 2023, using custom GPU kernels, quantization, and scheduling techniques.
Text Generation Inference (TGI)
Text Generation Inference (TGI) is a Hugging Face server for large language models, first released in 2022 as the first high-throughput serving framework for transformer models and archived in 2026.
Token merging
Token merging (ToMe) is a technique that speeds up transformers by merging similar tokens between layers without retraining, introduced by Meta AI researchers in October 2022.
Weight-only quantization
Weight-only quantization is a model compression technique that stores neural network weights in 4-bit or 8-bit integers while keeping activations and computation in high precision.