vLLM

Overview

vLLM is an open-source large model inference and serving engine. Its core uses PagedAttention technology to manage the attention key-value cache, reducing GPU memory fragmentation through a paging mechanism, and significantly improving throughput with continuous batching. It supports tensor parallelism, pipeline parallelism, speculative decoding, multiple quantization schemes, and LoRA adaptation, provides an OpenAI-compatible HTTP server, and can be deployed on various hardware such as NVIDIA GPUs, AMD GPUs, and TPUs. With its high throughput and ease of integration, vLLM is used as an underlying engine by many cloud providers and inference platforms. On September 24, 2026, vLLM announced support for lossless text watermarking based on the Gumbel-max algorithm, integrating watermarking capability into the sampling pipeline of Model Runner v2, maintaining output diversity while remaining compatible with speculative decoding, and providing traceable identification capability for generated content.

Key Features

• PagedAttention Key-Value Cache Management: Manages the attention key-value cache in pages, reducing GPU memory fragmentation and significantly improving inference throughput together with continuous batching.
• OpenAI-Compatible Server: Provides an OpenAI-compatible HTTP server, making it easy for existing applications to migrate and integrate seamlessly.
• Multi-Hardware and Parallel Support: Supports tensor parallelism, pipeline parallelism, speculative decoding, multiple quantization schemes, and LoRA adaptation, and can be deployed on various hardware such as NVIDIA GPUs, AMD GPUs, and TPUs.
• Lossless Text Watermarking: On September 24, 2026, lossless watermarking based on the Gumbel-max algorithm was added, integrated into the Model Runner v2 sampling pipeline. It is implemented through three changes: fused GPU kernel, dual-key scheme, and context deduplication (PR 54053, 56122, 56233), is compatible with speculative decoding, and maintains output diversity.

Use Cases

• Cloud providers and inference platforms use it as an underlying inference engine to support high-concurrency large model services.
• Enterprises deploy OpenAI-compatible private large model API services.
• Batch text generation scenarios that require high throughput and low GPU memory fragmentation.
• Compliance and content review scenarios that require traceable identification of generated content.
• Model inference deployment in multi-hardware environments, including NVIDIA GPUs, AMD GPUs, and TPUs.

Pros

• Open-source engine with an active community, adopted by many cloud providers and inference platforms.
• The combination of PagedAttention and continuous batching significantly improves throughput and reduces GPU memory fragmentation.
• Provides an OpenAI-compatible server with low integration cost.
• Supports tensor parallelism, pipeline parallelism, speculative decoding, multiple quantization schemes, and LoRA adaptation, offering strong extensibility.
• Can be deployed on various hardware such as NVIDIA GPUs, AMD GPUs, and TPUs, providing flexible hardware choices.
• Adds lossless text watermarking, compatible with speculative decoding while maintaining output diversity, providing traceable identification capability.

Pricing

vLLM is an open-source project. For specific pricing and commercial support information, please refer to the official website.

Summary

vLLM is an open-source large model inference and serving engine that achieves high throughput with PagedAttention and continuous batching, provides an OpenAI-compatible server, supports multiple parallelism, quantization, and LoRA adaptation options, and can be deployed on various hardware. In September 2026, lossless text watermarking was added, compatible with speculative decoding while maintaining output diversity, providing traceable identification for generated content.

Version History

• An online fitted acceptance estimator brings Adaptive Verification to draft models like EAGLE and MTP (2026-10-14): On October 14, 2026, vLLM published an online fitted acceptance estimator. Adaptive Verification needs to decide how much of a draft to accept at each step, which usually depends on a confidence head built into the draft model, and draft models such as EAGLE and MTP do not have one. The new approach fits an estimator online, predicting acceptance length without changing the model architecture, so these draft models can use adaptive verification too and cut verification overhead while keeping the output distribution intact.
• Push-based KV transfer lands, improving time to first token by up to 3x in disaggregated inference (2026-10-14): On October 14, 2026, vLLM published a push-based KV cache transfer connector built on NIXL. In prefill-decode disaggregated deployments the prefill node now pushes KV to the decode node directly, removing a round trip of request and pull, which vLLM reports improves time to first token by up to 3x for disaggregated inference.
• Bi-directional KV transfer lands: TTFT up to 1.6x to 3.3x lower for multi-turn agentic workloads (2026-10-12): On October 12, 2026, vLLM published bi-directional KV cache transfer, written by Sunita Nadampalli of AWS. Under prefill-decode disaggregation, the previous implementation only moved KV cache one way, from prefill to decode. For multi-turn workloads that meant the prefill node had to recompute KV projections for previous response tokens, and redundant work grew with the conversation, raising time to first token. Bi-directional transfer lets a decode node return previously computed KV to a prefill node on later turns, coordinated by a KV-cache-aware router and governed by a configurable KV recompute threshold. In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances connected with AWS Elastic Fabric Adapter, TTFT dropped by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results depending on workload and configuration. The feature was first built and validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic capability.
• DeepSeek-V4.1-Flash inference optimizations (2026-10-07): Inferact and the vLLM community finished optimizing for DeepSeek-V4.1-Flash within three weeks of its release, reaching a 1.9x speedup at low concurrency and 5.3x higher throughput under a 150 TPS constraint, which markedly improves both responsiveness and concurrency in agent workloads
• vLLM adds distortion-free text watermarking based on Gumbel-max (2026-09-24): Watermarking is integrated into the Model Runner v2 sampling pipeline through fused GPU kernels, a dual-key scheme and context deduplication, staying compatible with speculative decoding while preserving output diversity and giving generated content a traceable marker.
• vLLM releases vllm-metal, supporting concurrent serving on Apple Silicon, providing more balanced performance under concurrent agent workloads. (2026-09-22): vLLM releases vllm-metal, supporting concurrent serving on Apple Silicon, providing smoother TTFT under concurrent agent workloads.