vLLM 0.26.0

vLLM v0.26.0 Release Notes

Highlights

This release features 411 commits from 212 contributors (61 new)!

• New Inkling model family with a full support stack: base modeling (#48799), piecewise CUDA graph support (#48822), Hopper FA4 relative attention (#48858), MTP=1 speculative decoding (#48869), LoRA (#48884), and standard ModelOpt NVFP4 quantization (#48990).
• DeepSeek-V4 performance push across vendors: a specialized routing kernel (2.94% E2E TPOT, #48660), fused_topk_bias (1.5-2x kernel, #47463), and redundant repeat/copy removal (1.8% E2E TPOT, #48137), plus ROCm two-stage compressor for HCA prefill (#47718), sparse decode/prefill optimizations (#48519, #48788, #46275), and DSpark speculative decoding on AMD (#47419) and XPU (#47677).
• fp32 lm_head for generation models via head_dtype (#48390), extended to the LoRA path (#48525) and given a ROCm torch.mm fast path (#48688), improving accuracy for generation heads.
• Flexible attention backends: the attention backend can now be selected per KV-cache group (#48012), and sliding-window support is now an explicit backend capability (#48011) - improving support for hybrid models.
• KV offloading & tiered secondary storage matured substantially: offloading metrics (#45958, #47666, #47679), tier-owned event handling (#46544, #47923), object-store secondary tier with workload identity (#47063, #47274, #48150), DP-replica-aware tiering (#47987), and encoder-cache (EC) connectors including CPU offloading (#42433, #47423).
• Rust frontend gained multimodal video (#47959) and audio (#48554), a Seed-OSS tool parser (#47741), and a native vllm-bench port (#48107).
• Transformers 5.13.0 (#47867) with more models migrated to the Transformers modeling backend: Olmo/Olmo2 (#48100), MistralLarge3 (#48153), and HunyuanVL (#47872).

Model Support

• New models: Inkling family (#48799, #48822, #48858, #48869, #48884, #48990), BertForMaskedLM (#48463), RobertaForTokenClassification / XLMRobertaForTokenClassification (#47991), LongCat-Flash-Lite n-gram embedding (#47857), Cosmos3 Edge Reasoner (#48291) and Cosmos3-Super registration (#48211), TranslateGemma-12b-it (#41599).
• Transformers backend migrations: Olmo/Olmo2 (#48100), MistralLarge3 to AutoWeightsLoader (#48153), HunyuanVL native transformers processor for transformers 5.13 (#47872).
• GLM5.2: migrate MoE sequence-parallel support to the non-torch-compiled path (#47881).
• LoRA: FlashInfer MoE LoRA for BF16 models (#48632), LoRA for tower/connector in LlavaNextVideo (#48594), fp32 lm_head on the LoRA path (#48525), optimized TrtLlmLoRAExperts (#48759).
• Multimodal: automatic fallback to ViT data parallelism when TP is unavailable (#49046).
• Fixes: correct pooling scores for chunked prefill under torch.compile (#48901).

Engine Core

• fp32 lm_head for generation models via head_dtype (#48390); lower memory for capturing large CUDA graph sizes (#48483); opt-in persistence and reuse of the memory-profiling result across boots (#47388); improved InstantTensor loading (#46868).
• Attention: select a different attention backend per KV-cache group (#48012); sliding-window as an explicit backend capability (#48011); KV-cache layout refactor packing K/V into the content dim across backends (#44455); MRV2 virtual-batch PCP for MLA (#46570).
• Speculative decoding: runtime draft weight update (#46725), hybrid (SWA + full attention) DFlash drafters (#47914), SWA support for qwen-eagle3 (#47568), Gemma4-12B DSpark draft model (#47216), DSv4 DSpark on AMD (#47419), separate kv_cache_dtype for speculative_config (#48787).
• KV offloading: basic offloading metrics (#45958), split CPU cache usage into read/write gauges (#47666) and tiering-lookup-delay into sync/async histograms (#47679), tier-owned event handling and BlockStored events (#46544, #47923), object-store secondary tier with workload identity (#47063, #47274, #48150), DP-replica-aware tiering (#47987), blocks_per_chunk config for heterogeneous KV groups (#48878), P2P default host/port env vars (#47636).
• Caching: partial prefix-cache hit for hybrid models (#46384), selective hybrid cache retention (#47782), report prefix-cache-reused blocks in full report mode (#45261).
• Reasoning: optimize TPOT for thinking budget when used with speculative decoding (#46662).
• RLHF: stateful trainer-send abstractions (#48042).
• Fixes: host memory leak from undrained new_block_ids (#44490), DSv3.2 + MTP + sequence-parallel accuracy (#48036).

Hardware & Performance

• DeepSeek-V4: specialized routing kernel (2.94% E2E TPOT, #48660), fused_topk_bias 1.5-2x (#47463), redundant repeat/copy removal (1.8% TPOT, #48137).
• MoE router GEMMs: BF16x3 router GEMM (#47973), FP32 router GEMV (#48335), generic CuteDSL LL BF16 router GEMM (#42562); TRTLLM BF16 MoE modular kernel (#45182); write FlashInfer combine into final output (#47156).
• Qwen: fuse more RMSNorm + all-reduce in Qwen3.5 (#46998), replace MoE all-reduce with reduce-scatter (#47006), Qwen3.5 H20 optimization (#48350), expand Triton warmup coverage (#47546).
• MLA: dense MHA path for short sparse-MLA sequences (#47327); MiniMax-M3 long-context decode indexer on sm100 (#48582).
• Kernels: CUDA kernel for ReLUSquaredActivation / relu^2 (#39058), Helion kernel lazy registration (#48264), vectorize _copy_mamba_state_block to uint64 (#48110), stop upcasting logits to fp32 in the sampler (#48641).
• ROCm: fp32 head_dtype torch.mm fast path (#48688), DSv4 two-stage compressor kernel (#47718), sparse decode/prefill optimizations (#48519, #48788, #46275), DSv3.2 sparse MLA KV-split heuristic (#46832) and MTP CUDA-graph mode (#45149), MXFP8 GEMM for MiniMax-M3 (#46117), AITER sparse paged attention + spec decode for MiniMax-M3 (#47287, #47984), MiniMax-M2 fused QK-norm + all-reduce via AITER (#44849), HybridW4A16 linear kernel (#40977), Qwen3-30B-A3B QK-Norm+RoPE+KV runtime fusion (#42749).
• XPU: batch-invariant kernels (#41934), HND KV layout support (#47975), DSpark spec decode for DSv4 (#47677), nightly/release image publishing (#47880, #48126).
• CPU: DFlash speculative decoding for GDN models on CPU (#46090), s390x NUMA topology (#40714), native macOS arm64 CPU wheel builds (#48289); POWER VSX math function optimization (#47321) and IBM Power docker builds using prebuilt wheels (#46017).
• Distributed fusion: FlashInfer MNNVL all-reduce RMS quant fusion (#48064).
• Build/autotune: arm64 Blackwell SM10x/SM110 image builds (#48041); skip CuTeDSL fp4_gemm autotuning by default (#48268).

Large Scale Serving & Distributed

• Decode Context Parallel (DCP): hybrid attention support (#40996), DCP + Eagle for Tokenspeed MLA backends (#48180).
• PD disaggregation: NIXL pipeline-parallel prefill in push mode (#45880).
• Encoder-cache connectors: EC transfer params (#42433) and CPU-offloading EC connector (#47423).

Quantization

• Humming w[2-7]a[4,8] weight-only inference with compressed-tensors (#46390); int4 quantization for the emulation MoE backend (#48451); INT2 XPU weight-only quant linear (#47521).
• NVFP4/MXFP4: nvfp4_per_token online MoE quantization (#48538), CuTe-DSL FlashInfer MXFP4 quantization (#48417); bounded peak memory when repacking FP4 MoE weights for Marlin (#47851) and for NVFP4 MoE weight loading (#46276).
• MLA: kv_cache_dtype_skip_layers support (#47309).
• ROCm: HybridW4A16 linear kernel (#40977).

API & Frontend

• Rust frontend: multimodal video (#47959) and audio (#48554), Seed-OSS tool parser (#47741), native vllm-bench port (#48107), continue_final_message handling with renderer sentinel (#47844).
• OpenAI compatibility: bad_words in /v1/completions (#46793), expose logprob_token_ids on Python OpenAI endpoints (#43463), include_reasoning param for non-Harmony models (#44301), populate num_cache_creation_tokens on Messages responses (#48535).
• Endpoint plugins framework (#47454); /abort_requests on the RLHF dev API router (#47173); Deepstream video decoding backend (#42424); overlap preprocessing and computation for pooling models in offline inference (#47699).
• UX: human-readable integers for more CLI args (#47608), CuTeDSL compilation progress bar (#48881), expanded GPU profiler config scope/annotations (#37524), log worker exit code when a process dies unexpectedly (#38641).
• Stability/correctness: handle grammar compilation failures without crashing the engine (#47312), fix logprobs token-string collision from SentencePiece spaces (#48674).

Security

• Replace diskcache to eliminate pickle deserialization (#44549).
• Fix a concurrent sparse-invariant race that bypassed CVE remediation (#48583).
• Add resource-bounds validation to derender endpoints (#47260); sanitize server file paths from validation error responses (#46415); bound the completion prompt list to prevent unbounded engine fan-out (#47845); guard lm-format-enforcer regex compilation with a timeout (#47595).

Dependencies

• Transformers 5.13.0 (#47867), FlashInfer 0.6.14 (#47669), NIXL 1.3.1 (#47559), tpu-inference v0.24.0 (#47835), nvidia-cutlass-dsl 4.6.0 (#47442), vllm_xpu_kernels v0.1.11.1 (#48942).
• FlashAttention 3 pinned to the torch stable-ABI commit (#47995); ABI-stable FlashMLA build (#48174).

Deprecations & Removals

• Models removed: TeleChat (#47989), Persimmon and Fuyu (#48096).

New Contributors

• @adhi29 made their first contribution in https://github.com/vllm-project/vllm/pull/48262
• @adsridhar made their first contribution in https://github.com/vllm-project/vllm/pull/48291
• @alexxu-roblox made their first contribution in https://github.com/vllm-project/vllm/pull/48025
• @amd-ethany made their first contribution in https://github.com/vllm-project/vllm/pull/46117
• @AndyDai-nv made their first contribution in https://github.com/vllm-project/vllm/pull/48507
• @aoright made their first contribution in https://github.com/vllm-project/vllm/pull/47797
• @ap9272 made their first contribution in https://github.com/vllm-project/vllm/pull/43896
• @avalliappan-nvidia made their first contribution in https://github.com/vllm-project/vllm/pull/47460
• @brijrajk made their first contribution in https://github.com/vllm-project/vllm/pull/44349
• @Debasish-87 made their first contribution in https://github.com/vllm-project/vllm/pull/48878
• @devalshahamd made their first contribution in https://github.com/vllm-project/vllm/pull/37524
• @DiegoCao made their first contribution in https://github.com/vllm-project/vllm/pull/47216
• @drakosha made their first contribution in https://github.com/vllm-project/vllm/pull/46972
• @edwinlim0919 made their first contribution in https://github.com/vllm-project/vllm/pull/47495
• @ErenAta16 made their first contribution in https://github.com/vllm-project/vllm/pull/48333
• @Functionhx made their first contribution in https://github.com/vllm-project/vllm/pull/48153
• @gangula-karthik made their first contribution in https://github.com/vllm-project/vllm/pull/48594
• @Gavin-Morris-04 made their first contribution in https://github.com/vllm-project/vllm/pull/48293
• @giuseppegrossi made their first contribution in https://github.com/vllm-project/vllm/pull/48159
• @GongLei-HW made their first contribution in https://github.com/vllm-project/vllm/pull/45261
• @guoriyue made their first contribution in https://github.com/vllm-project/vllm/pull/48211
• @hugo-cen made their first contribution in https://github.com/vllm-project/vllm/pull/48330
• @ibondarenko1 made their first contribution in https://github.com/vllm-project/vllm/pull/43117
• @iyastreb made their first contribution in https://github.com/vllm-project/vllm/pull/48209
• @jacklin78911-collab made their first contribution in https://github.com/vllm-project/vllm/pull/47744
• @jhu960213 made their first contribution in https://github.com/vllm-project/vllm/pull/42749
• @joanvelja made their first contribution in https://github.com/vllm-project/vllm/pull/44371
• @KKothuri made their first contribution in https://github.com/vllm-project/vllm/pull/48390
• @kl527 made their first contribution in https://github.com/vllm-project/vllm/pull/47464
• @krishy91 made their first contribution in https://github.com/vllm-project/vllm/pull/47991
• @langzhao-netizen made their first contribution in https://github.com/vllm-project/vllm/pull/43463
• @lishunyang12 made their first contribution in https://github.com/vllm-project/vllm/pull/49015
• @mahadrehmann made their first contribution in https://github.com/vllm-project/vllm/pull/48098
• @ManaEstras made their first contribution in https://github.com/vllm-project/vllm/pull/47872
• @mosya415 made their first contribution in https://github.com/vllm-project/vllm/pull/48846
• @mwoodson made their first contribution in https://github.com/vllm-project/vllm/pull/48523
• @MynameFelix made their first contribution in https://github.com/vllm-project/vllm/pull/41811
• @nicklasfrahm made their first contribution in https://github.com/vllm-project/vllm/pull/45313
• @passtoor-agi made their first contribution in https://github.com/vllm-project/vllm/pull/48849
• @pierDipi made their first contribution in https://github.com/vllm-project/vllm/pull/47063
• @ruikangliu made their first contribution in https://github.com/vllm-project/vllm/pull/47309
• @Sahil170595 made their first contribution in https://github.com/vllm-project/vllm/pull/45207
• @samnordmann made their first contribution in https://github.com/vllm-project/vllm/pull/47156
• @shawntsai made their first contribution in https://github.com/vllm-project/vllm/pull/47801
• @sungbin1015 made their first contribution in https://github.com/vllm-project/vllm/pull/46793
• @tanish-malekar made their first contribution in https://github.com/vllm-project/vllm/pull/39058
• @tomerg-nvidia made their first contribution in https://github.com/vllm-project/vllm/pull/47021
• @tsvikas made their first contribution in https://github.com/vllm-project/vllm/pull/47296
• @vanshbhatia-amd made their first contribution in https://github.com/vllm-project/vllm/pull/47767