Hugging Face Transformers Release 5.16.0

Release v5.16.0

New Model additions

Qwen4-Exp

Qwen4-Exp builds on Qwen3.5's hybrid text and multimodal architecture with three key components: GatedResidual (GR), Qwen Sparse Attention (QSA), and Per-Layer Embedding (PLE).

GR is a Qwen-developed residual architecture that combines Hyper-Connection with GatedNorm. It mixes multiple residual streams with fine-grained elementwise gating before each attention and Mixture-of-Experts (MoE) block, then controls how much of the block output is injected back into each stream.

QSA uses multiple query heads to score compressed key blocks, selects the most relevant contiguous token blocks, and keeps the incomplete trailing block uncompressed. This block-level selection reduces indexing overhead and improves memory locality for long sequences. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate linear and sparse attention, substantially improving inference efficiency for long-context workloads.

PLE enriches selected decoder layers with layer-specific lexical features derived from hashed token n-grams and a dilated depthwise convolution.

Links: Documentation

• Add Qwen4Exp model (#48337) by @Cyrilvallez in #48337

GraniteSpeech5

Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the Granite Speech family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder.

Architecturally, it extends the Granite Speech conformer CTC encoder with:

1. Frame stacking + block-wise time subsampling: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop.

1. Block attention with Shaw's relative positional embeddings: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections.

1. Self-conditioned CTC: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction.

Links: Documentation

• Add Granite Speech 5.0 - (#48288) by @eustlb in #48288

Step3p7

Step-3.7-Flash was proposed in Step 3.7 Flash by StepFun. It is a 198B-parameter sparse Mixture-of-Experts vision-language model, pairing a 196B-parameter MoE language backbone with a 1.8B-parameter vision encoder for native image understanding.

StepFun hasn't published a technical report for Step-3.7-Flash, so the details below are drawn from the released checkpoint's configuration rather than a paper.

• Sparse MoE decoder: all but the first 3 decoder layers route through a MoE block of 288 routed experts (top-8 per token) plus a single shared expert. The router scores experts with a sigmoid and a learned per-expert bias instead of an auxiliary load-balancing loss, the same strategy as DeepSeek-V3.
• Gated attention: each attention layer adds an extra projection whose sigmoid output gates the attention output per head, before the output projection - the same *Gated Attention* mechanism used in Qwen3-Next. A subset of layers use fewer heads and a sliding window instead of full attention.
• Multi-token prediction: some checkpoints ship extra decoder layers trained for multi-token prediction, which [~GenerationMixin.generate] can use for speculative decoding via use_mtp=True.
• Vision encoder: a SigLIP-style ViT with 2-D rotary position embeddings and a learned per-layer scale on the attention and MLP branches. Its output is downsampled 4x by two stride-2 convolutions before a linear projector maps it into the text model's hidden size.
• Dynamic image tiling: instead of a fixed tile grid, the image processor picks its tiling window from each image's own aspect ratio, producing one downscaled global view plus zero or more local high-resolution crops per image.

Links: Documentation

• [new model] step 3.7 (#46658) by @itazap in #46658

CohereCompass

CohereCompass is the base architecture for small, specialized (vision-)language models trained by Cohere.

Links: Documentation

• Add CohereCompass modeling (#47878) by @calpt in #47878

ESMC and ESMFold2

ESMC and ESMFold2 are new state-of-the-art protein language and folding models from BioHub. ESMC is trained with a masked language modeling objective, and it can be easily transferred to sequence and token classification tasks for proteins. Checkpoints exist in various sizes, from 300M parameters up to 6B parameters. It works as a drop-in replacement for older ESM-2 and ESM-3 models, with significantly higher accuracy.

ESMFold2 is a state-of-the-art protein folding model which produces high accuracy predictions. It uses an iterated diffusion approach that is significantly different from the original ESMFold, offering huge improvements in accuracy for more complex structures.

Links: Documentation ESMC, Documentation ESMFold2

• Port ESMC and ESMFold2 to Transformers (#46419) by @Rocketknight1 in #46419

Breaking changes

The legacy tensor-parallel implementation has been replaced with a DTensor-native backend, so users relying on the previous TP API for inference or training must migrate to the new DTensor-based interface.

• TP dtensor API inference + training (#47579) by @3outeille

attn_implementation="sdpa" dispatch is now properly supported for wav2vec2_conformer, wav2vec2-bert, and SeamlessM4T/v2 models, which may change initialization behavior for users who previously worked around this limitation.

• [wav2vec2] Support attn_implementation=sdpa dispatch (#46196) by @YangKai0616

FuyuProcessor no longer returns the image_patch_indices output, so any code that depends on this field must be updated to remove references to it.

• :rotating_light: Leftover processors (#47924) by @zucchini-nlp

Cache

Several cache-related bugs were fixed in this release, including an off-by-one error in the sliding window cache, Whisper speculative decoding cache corruption, CpmAnt use-cache failures, Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches, and compressed-tensors loading for KV-cache-only quantized models. Documentation was also added for cache token removal using negative values, and per-layer cache configuration support (allowing models to use different cache settings per layer) was introduced.

• Cpmant fix use cache (#48013) by @jiqing-feng in [#48013]
• [docs] Cache crop (#47950) by @stevhliu in [#47950]
• Revert "Support per-layer cache configuration and attention-mask selection" (#48175) by @Cyrilvallez in [#48175]
• Support per-layer cache configuration and attention-mask selection (#47901) by @eladsegal in [#47901]
• Fix Qwen2.5-Omni / Qwen3-Omni-MoE generation with a compilable cache (#47872) by @jiqing-feng in [#47872]
• Fix sliding window cache index off-by-one on wraparound (#47708) by @hameedibrh in [#47708]
• [Whisper] Fix speculative decoding: UnboundLocalError, cache corruption, and speed regression (#48000) by @ydshieh in [#48000]
• Fix compressed-tensors loading for KV-cache-only quantized models (#47904) by @kylesayrs in [#47904]

Generation

This release fixes several generation bugs across multiple models, including Whisper speculative decoding issues (UnboundLocalError, cache corruption, speed regression, and left-padded batch position IDs), broken image generation in Emu3, garbage output in OLMo/GPTNeoX, and Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches. Additionally, logit distributions for candidate generators using sampling are now aligned by returning logits after applying logit processors.

• [Whisper] Fix speculative decoding: preserve cleared suppress tokens through super().generate() (#48108) by @ydshieh in [#48108]
• [Whisper] Fix decoder position IDs for left-padded batches in longform generation (#48028) by @ydshieh in [#48028]
• [serge] Fix 2 integration tests for model generation failing with import_or_config (other (2)) (#48061) by @sergereview[bot] in [#48061]
• Align logit distributions for CandidateGenerators using sampling (#48007) by @Cyrilvallez in [#48007]
• [GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988) by @ydshieh in [#47988]
• [emu3] Black Labrador is back! Fix image generation broken since #37033 (#47948) by @ydshieh in [#47948]

Attention

Several attention-related bug fixes were made in this release, including correcting a SigLIP2 documentation typo, fixing Flash/SDPA attention dispatch tests for xcodec2 and ROCm RDNA GPUs, resolving a GPT2 cross-attention mask being silently discarded, and enabling SDPA support declaration in TimmWrapper. Per-layer cache configuration and attention-mask selection support was also introduced, allowing models with heterogeneous layer configurations to use distinct sliding_window, attention_chunk_size, and number_of_conv_states values per layer.

• doc: Fix typo in SigLIP2 Flash Attention code example (#48197) by @VimalN2005 in [#48197]
• [xcodec2] Fix flex attention and flash dispatch tests (#48244) by @jiqing-feng in [#48244]
• Fix ROCm SDPA-flash skip guard that crashes on RDNA GPUs (#47965) by @Abdennacer-Badaoui in [#47965]
• [GPT2] Fix encoder_attention_mask being silently discarded in cross-attention (#47946) by @DavidJohnQuinlan in [#47946]
• Declare sdpa support in TimmWrapper (#47939) by @jiqing-feng in [#47939]

Quantization

Quantization improvements include adding NVFP4 quantization support via HF kernels (enabling on-the-fly BF16 weight quantization with ~50% memory reduction), and fixing several bugs: reverting a regression in is_quantization_compressed that caused incorrect module layouts for packed-format checkpoints, fixing CLIP weight initialization failures with quantized checkpoints, and restoring KV-cache quantization setup for KV-cache-only quantized models.

• Revert "[Quantization]: Refactor is_quantization_compressed for format-based detection" (#48072) by @subin9 in [#48072]
• feat: add nvfp4 quantization (#47883) by @drbh in [#47883]
• [DeepSeekV2] Fix integration tests OOM: use device_map=auto instead of 8-bit quantization (#47991) by @ydshieh in [#47991]
• Fix CLIP _init_weights when a child module carries quantized weights (#47921) by @Bluear7878 in [#47921]

Parallelization

Introduced a naive pipeline parallel inference engine supporting tied/untied weight embeddings with seamless generate() integration, while restoring backward compatibility for the tensor-parallel API with a deprecation cycle for tp_plan in from_pretrained(). Additionally fixed a model parallel bug in the BLT model affecting beam search.

• Restore BC for the tensor-parallel API (#48300) by @ArthurZucker in [#48300]
• fix bug for blt model parallel bug (#48327) by @kaixuanliu in [#48327]
• Pipeline parallel naive inference (#47289) by @3outeille in [#47289]

Kernels

Kernel support was improved with documentation updates highlighting supported models, a fix for export crashes on kernel-decorated functions by adding a is_torchdynamo_exporting guard, and the default Flash Attention 2 hub kernel version was bumped to v3 to resolve compatibility issues with newer PyTorch versions.

• [docs] Kernel supported models (#48258) by @stevhliu in [#48258]
• [Fix] Export crashes on kernel-decorated function (#47808) by @remi-or in [#47808]
• Bump default flash-attn2 hub kernel version to v3 (#47863) by @jiqing-feng in [#47863]

Bugfixes and improvements

• Fix video-llama modular conversion (#48336) by @zucchini-nlp in [#48336]
• CI: gate the hunyuan-moe slow test (#48330) by @tarekziade in [#48330]
• Add a regression test for force_accelerate_hooks signature preservation (#48260) by @wtdcode in [#48260]
• Add an opt-in per-frame pixel cap (cap_pixels_per_frame) to the Qwen3-VL video processor (#48071) by @dkrisman in [#48071]
• Fix scores type in stopping criteria docstrings (#47676) by @qgallouedec in [#47676]
• Docstring check didn't match some file - fix it (#48121) by @zucchini-nlp in [#48121]
• Add shared ImageProcessingTester (#47745) by @guarin in [#47745]
• Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211) by @ydshieh in [#48211]
• [docs] Fix failing doctests (#47687) by @stevhliu in [#47687]
• Fix build_2d_sinusoidal_position_embedding on MPS (#47897) by @guarin in [#47897]
• [Gemma4] Investigate flaky test_generation_beyond_sliding_window_1_eager (#48236) by @ydshieh in [#48236]
• Let gradient checkpointing skip layers with every_n_layers (#48200) by @qgallouedec in [#48200]
• Disable daily nightly CI (#48292) by @remi-or in [#48292]
• gs (#48288) by @eustlb in [#48288]
• CI: fix muse OOMs (#48284) by @tarekziade in [#48284]
• Fix BayesianDetectorModel.from_pretrained() by calling post_init() (#48254) by @woojinpaik in [#48254]