vLLM 0.32.0

Original release notes

v0.32.0

Highlights

This release features 814 commits from 363 contributors (121 new)!

Structured decisions and generic parsers: the new POST /v1/systemone endpoint answers choice, noul and score questions over a structured state (#59299), is served by default (#60677, #60696), and also runs on Cloudflare/clef-omni (#61064), LiquidAI/d1-3B (#60676) and DiffusionGemma (#60466). The new hf tool and reasoning parser (--tool-call-parser hf, --reasoning-parser hf) reads the checkpoint's response_template, so models that ship it need no model-specific parser (#58604, #58602), in both the Python and Rust frontends (#59005).
Low-precision KV cache everywhere: NVFP4 KV cache now runs on SM8x and SM12x (#46963), SM90 (#60620) and with XQA decode on SM12x (#60452), and on MiniMax-M3 sparse attention on SM100 (#59300); GLM-5.x defaults to an FP8 KV cache on SM100, halving KV memory per rank with 2.3-5.5% more throughput at a 7-11% TTFT cost in the PR's benchmark (#60140, #60281); fp8_ds_mla on SM90 gives GLM-5.3-Flash up to 31% more output throughput (#59246); ROCm gains the 4-bit ultraquant_4bit KV cache with about 2x the density of FP8 (#57057).
Model Runner V2: a GPU n-gram speculator with variable-length drafts (#40704), Elastic EP (#53934), pipeline parallelism with prefill context parallelism for MLA models (#59139), a stock torch.compile mode (#59079), and MRV2 is now the default on ROCm with the model-specific MRV1 exceptions removed (#56360).
Faster startup and serving: vllm download-kernels installs FlashInfer's precompiled kernels (#58765); vllm preload --preload-autotune keeps the FlashInfer autotune table warm across restarts (#60085); a shared-memory tensor arena broadcasts large multimodal tensors to workers with one copy (#51207); tokenizers and processors load from the local snapshot (#60825); and with --api-server-count above 1 each connection goes to the least-loaded API worker, for 43-85% more req/s on short requests (Qwen3-0.6B, 32 input / 1 output tokens) in the PR's benchmark (#60723).
Model support: new EmbeddingGemma 2 omni-modal embeddings (#60254), MiniCPM-V 4.7 (#58674) and Nemotron 3.5 ASR (#59827); video input in the Transformers modeling backend (#57441); GLM-5.3 sparse MLA index conversions reused across layers (3.5-3.9x on that kernel, #59464) and fused multi-step draft decode (13.3% at concurrency 1, #57443); HiSparse per-request residency caching for up to 58% more throughput on GLM-5.3-NVFP4 on B200 with long, fully GPU-resident contexts at high concurrency (#60083).
Security: fixes for GHSA-4hhp-h66f-j5j7 (chat template resource exhaustion, #50300), GHSA-q97c-p368-75c4 (tokenizer max_token_id, #59491) and GHSA-frvg-qw88-hww9 (P2P ZMQ socket exhaustion, #51504); CORS no longer allows every origin by default (#61009); untrusted images are limited to raster formats (#60022); shared-memory multimodal cache handles are signed (#59357); and JSON schemas nested deeper than 64 levels are rejected (#60036).
Breaking changes: Python 3.10 dropped (#60402); default CORS origins restricted (#61009); transformers>=5.16.1,<5.20.0 and tokenizers>=0.23.2 (#57387, #59796); torch.compile sequence parallelism and AsyncTP passes removed (#60517); cascade attention removed (#61126); mamba_cache_mode="all" removed (#58997); max_tokens no longer reserved from the context window (#57035); Responses API reasoning stream events renamed (#59652).

Release Artifacts

Python Wheels

| Platform | Install | |---|---| | PyPI (CUDA 13.0) | pip install vllm | | PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto | | ROCm | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.32.0/rocm723 | | XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.32.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |

Docker Images

| Platform | Docker Image | |---|---| | CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.32.0 | | CUDA 12.9 | docker pull vllm/vllm-openai:v0.32.0-cu129 | | CUDA 13.0 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-ubuntu2404 | | CUDA 12.9 + Ubuntu 24.04 | docker pull vllm/vllm-openai:v0.32.0-cu129-ubuntu2404 | | ROCm | docker pull vllm/vllm-openai-rocm:v0.32.0 | | CPU | docker pull vllm/vllm-openai-cpu:v0.32.0 | | XPU | docker pull vllm/vllm-openai-xpu:v0.32.0 |

Other Artifacts

Pre-built release artifacts are available in the Assets section at the bottom of this page, including:

Source distribution tarball
CUDA 12.9 Python wheels for x86_64 and arm64
CUDA 13.0 Python wheels for x86_64 and arm64
CPU Python wheels for x86_64, arm64, and macOS
XPU Python wheel for x86_64

Model Support

New models: EmbeddingGemma 2 (google/embeddinggemma-2), an omni-modal text/image/audio/video embedding model, with unused modality encoders skipped through --limit-mm-per-prompt (#60254, #60289); MiniCPM-V 4.7 (#58674); Nemotron 3.5 ASR (nvidia/nemotron-3.5-asr-streaming-0.6b) complete-audio transcription; native realtime streaming is not yet supported (#59827); Cloudflare/clef-omni (#61064) and LiquidAI/d1-3B (#60676) on /v1/systemone; PLaMo 3 reasoning and tool parsers (#56575); and the zerank-2 reranker score template (#60415).
Transformers modeling backend: video inputs (#57441), and Glm, Arcee, CWM, Mellum (#59679), GPT-NeoX, Phi, Seed-OSS and Jais2 (#59701) now served through it.
New model capabilities: LoRA for BERT, RoBERTa and XLM-RoBERTa embedding models (#61033); Granite as an EAGLE3/DFlash target (#60855); Qwen3-ASR EAGLE3 (#52824); Sarvam MLA EAGLE3/DSpark with PP (#55902); GLM-5.3-Flash with the DFlash2 drafter (#56983); and MiniMax-M3 EAGLE3 at PP > 1 on MRV2 (#57197) with MTP sharing target embeddings under PP (#58648).
GLM-5.3 and GLM-5.3-Flash: sparse MLA index conversion reused across layers (#59464), fused multi-step draft decode (#57443), FP8 KV cache default on SM100 (#60140, #60281), fp8_ds_mla on SM90 (#59246), qlnorm skipped for MHA (4.4-7.7% TTFT, #58845), long-context indexer prefill sharded across TP (up to 25% lower TTFT at 1M context, #54951), DCP for the kpool indexer (#59211, #60032), KDA prefill checkpoints with FlashKDA (#56960), FP8 KV on SM90 sparse MLA (#55222), --mamba-ssm-cache-dtype honored for KDA state (#59147), a generic packed KV layout (#57169), MTP index sharing restored (#61079), and fixes for vision-tower crashes (#59126), non-square image encoder budgets (#59565), DP startup (#56974), video timestamps (#55647), DFlash2/EAGLE3 aux hidden states (#60763) and null-block writes (#59528).
DeepSeek-V4 and V4.1: decoder-side SWA bounded replay (#58132) with replay layers in CUDA graphs and trimmed in PIECEWISE graphs (compilation_config.decoder_replay_trim_threshold, #59532) plus last KV-source layer trimming (+3.6%, #59894), packed FP8 Engram exchange across DP (#59923), faster CPU-offloaded Engram lookups (#59327), fused DeepSeek V4 + MTP drafting (#58463), encoder CUDA graph packing for V4.1 and Kimi-K3 (11% lower mean TTFT, #58925), EPLB for DSpark drafts (#57549), and fixes for NVFP4 checkpoints with 0% draft acceptance (#59655), MegaMoE shared-expert finalize ordering (#59927), worst-case vision dummy inputs (#57898, #59271), compressor ring null-block writes (#58560), Triton 3.9 indexer quantization (#60716) and decoder-replay CUDA graphs with FlashInfer EP MoE backends (#61041).
Qwen: Qwen3.8-Flash-Next (Qwen4Exp) merged QSA projections in one GEMM (1.45-1.68x, #59533), fused QSA pre-indexer (#57097), sequence parallelism through --enable-hc-sp (#56322), SM100 low-latency decode GEMM plans (up to 2.7x vs cuBLAS on small-M, #59214), --kv-cache-dtype-skip-layers honored (#60023), and PLE loading for INC (#59990), Quark (#59443) and pre-SM89 GPUs (#60142); Qwen3-VL/Qwen3.5/Qwen3.8 M-RoPE cache 4x smaller and Qwen3.5 YaRN matching Transformers (#59853); Qwen3-Omni M-RoPE fixes (#58890, #59842); Qwen3-ASR on the upstream processor (#57146); Qwen3-VL unknown video fps (#60501) and per-video options with the multimodal cache (#55203).
Kimi-K3 and MiniMax-M3: Kimi-K3 FlashInfer fused speculative KDA on SM100 (#54255), fused vision processor (10-20x preprocessing at 1080p-8K, #59278), fused K/V pack on the DCP prefill path (#56881), and DSpark KV cache back to 4 groups (#59733); MiniMax-M3 MSA prefill with an E4M3 query (26-37% faster with FP8 KV, #60684), FP8 indexer cache on non-SM100 GPUs (#59081), Triton indexer tuning (#56151, #59481), and indexer score fixes (#60194, #54597).
Multimodal: device-side input normalization for Qwen3-VL/Qwen3.5/Qwen4Exp (#51289), GLM-V and GLM-OCR (#55389), Llama Nemotron VL (#57928) and Kimi K2.5/K3 (#59278), kept fused under compile_mm_encoder (#59195); encoder inputs sorted by modality (6.5-8.2% more req/s, #60196); Mistral 3 preprocessing 8% faster (#57531); Pixtral compile_mm_encoder (#57168); and fixes for Llama 4 / Mistral Large 3 EAGLE drafters (regression since v0.28.0, #57633), Phi-4 MM audio OOM (#60614), DeepSeek-OCR zero-sum pixels (#59417), MiMo-V2.5/V2.6 with fastsafetensors (#58142), MiMo-V2.6 FP8 KV cache (#58128), Voxtral realtime (#45022), Nemotron Super vision LayerNorm (#61401), K-EXAONE config and MTP RoPE (#60187), Nemotron Parse tied lm_head (#60272), LongCat-Flash MTP loading (#60080) and Ling-3.0-flash-VL (#59026).

Engine Core

Model Runner V2: GPU n-gram speculator (#40704), Elastic EP with drain-then-commit scaling (#53934), PP with PCP for MLA models (#59139), STOCK_TORCH_COMPILE mode (#59079), dynamic K in autoregressive speculators (#57053), MTP sampling-mask replay (#59359) and per-module LM heads (#58921), non-blocking H2D copies on WSL2 (#60215), and fixes for int64 index offsets at long context (#60736), warmup (#61013), GDN prefill checkpoints (#59536) and MRV2+PP+MTP align state (#60210).
Speculative decoding: fused multi-step draft decode on FlashInfer trtllm-gen (#58371), FlashInfer ReplaySSM for Mamba2 MTP (#52928), in-graph metadata for DFlash/DSpark drafts (#60692), and correctness fixes for ngram on hybrid recurrent models (#56531), Mamba prefill checkpoints with MTP and prefix caching (#59759, #60797), stop-string trimming (#47616), draft model config (#61240), adaptive block verification (#60945), DSpark bonus slots (#59105) and draft models on the Transformers backend (#59293).
Hybrid models and prefix caching: Mamba2 internal prefill checkpoints for NemotronH, Granite-4.0-H, Falcon-H1 and Zamba2 (#57329), stale block hashes dropped when streaming sessions truncate (#59103), and fixes for quantized Mamba-2 with TP (#58083), spec decode page sizes (#59975), prefix-cache bounds (#55601, #60533) and sm90 Mamba2 NaNs (#61718).
Sleep and RL: sleep releases workspace scratch (up to 4.5 GiB/GPU, #59156) and KV-init state (about 4.6 GiB/GPU, #59158), opt-in sleep_mode_offload_cudagraph (#59160, #59523); token-level weight-version spans on outputs (#59582, #61070); weight-transfer communicators released without stopping inference (#61060); --sleep-preserve-parameter-names 'draft.*' (#61046); modelexpress weight-transfer backend (#58399); POST /weight_checker (#51350); weight-transfer metrics (#55781); --enable-nccl-comm-suspend releasing the MNNVL workspace (#59360); and sleep(level=2) keeping FP8 KV scales (#57163).
HiSparse: per-request residency cache (up to 58% at high concurrency with GPU-resident contexts, #60083), lower scheduler overhead (#57930), GPU residency owned by HiSparse (#60647), less staging memory (#60429), host cache gauges (#58949, #59485), and fixes for decode token limits (#60682) and TP hangs (#60941).
Startup and memory: shared-memory tensor arena (--enable-shm-tensor-arena, on by default, #51207), Fast Start with PP (#55477), daemon-held weights counted against gpu_memory_utilization (#57298), autotune preload (#60085), local tokenizer snapshots (#60825), sparse_mla_prefill_workspace_factor saving several GiB at 1M context (#61023), cached memory reclaimed during weight loading (#57516), fewer Triton sampler warmup compiles (#58605), sharded state loader with PP (#52110), and reload_weights with Run:ai streamer (#58124).
Kernels and runtime: MoE weighted-sum tuning (1.17-2.37x, #59731), flash-maxsim kernels for ColBERT/ColPali scoring (#40337), fallback detokenization with 69% less CPU at 32K tokens (#60497), faster KV block free queue (#60269), Triton kernels off tl.make_block_ptr (#60502), fusion passes restored under PyTorch 2.15 (#57299) and static FP8 quant fusion on recent torch (#61108).
Batch invariance: custom all-reduce under VLLM_BATCH_INVARIANT=1 (#58623), deterministic split-K LoRA shrink (#59377), every attention backend checked (#60271), SM120 TP configs (#58495), and ROCm support (#52231).
Observability: vllm:prompt_tokens_cached_by_source_total (#56318), vllm:num_requests_kv_fetch_by_stage (#58874), vllm:admission_rejections_total (#55812), vllm:watermark_requests (#58715), --custom-histogram-buckets (#48867), a JSON log formatter via LoggingConfig.formatter (#58739), and per-session profiling controls (#57875).

Large Scale Serving

MoE communication: FlashInfer CuteDSL MegaMoE for more models via --moe-backend flashinfer_cutedsl (#54049), FP8 combine for the FlashInfer one-sided all2all (+4% at DEP8, #57995), NCCL symmetric-memory MoE reduce-scatter (9.6-16% lower latency, #49194), DeepEPv2 hybrid NVLink+RDMA mode on by default (#57991), Kimi-K3 shared experts with DeepEP v2 (#56875), and opt-in EPLB migration batching (#52641).
Context parallelism: decode requests sharded across PCP ranks (#52162) and TokenSpeed MLA with block-interleaved DCP for heterogeneous NIXL disaggregation (#59462).
KV connectors: NIXL host-buffer copies coalesced (#54483), mixed block sizes (#58860), peer metadata recovery (#55471) and Qwen3.8-Flash-Next MTP acceptance in P/D (#59997); Mooncake coalesced hybrid/MLA RDMA regions (#57952) and bootstrap retries (#55923); MoRIIO Kimi-K3 DSpark READ (#57700); DeepSeek-V4.1 P/D skipping bounded replay for loaded KV (#59197); and KVCR pool-and-index API (#59899).