llama.cpp 0.6.0

Overview

llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) and MTP speculative decoding for Qwen4Exp, ships a new /v1/systemone server API for decision models (laya, julia-1, lev, openjev, kev, nimble), overhauls the Web UI with a Hugging Face Hub data layer and model download pipeline, adds a Metal tensor-API flash attention kernel for F16 KV, sparse flash attention for quantized K/V on Vulkan, and updates ggml to v0.26.0.

Highlights

• New llama_batch_ext extended batch API with llama_process(), supporting mixed token/embedding batches and per-token "state" embeddings for MTP and deepstack models #24669
• New models: GLM-5.3-Flash (GLM5-Next), a 320B text+vision hybrid model #27773, and the Clef decision model, fully supported with both text and vision #29831 #29969
• Qwen4Exp: high-quality support is now available, with MTP speculative decoding (~1.5x decode speedup on DGX Spark) and various correctness fixes #29761 #29751
• llama and server: new /v1/systemone API supporting five decision models - laya, julia-1, lev, openjev (+vision), kev #29818
• Metal: new tensor API flash attention kernel for F16 KV #29570
• Metal: new few-row MMA mat-mul kernels for speculative and batched decoding, up to ~3x faster mat-mul on Apple GPUs #29869
• New llama_prefetch_rows() using MADVISE-based prefetching of PLE tensors in Qwen4Exp and Gemma4 #29599

API changes

• include/llama.h: new llama_batch_ext batch API with llama_embd, llama_process() and llama_process_type #24669, new llama_get_causal_attn() #28876, session formats bumped to LLAMA_SESSION_VERSION 11 and LLAMA_STATE_SEQ_VERSION 4
• include/llama-cpp.h: added llama_batch_ext_ptr and deleter for the new extended batch API #24669
• tools/mtmd/mtmd.h: mtmd_get_memory_usage() now returns an mtmd_memory_usage struct with image_max_tokens and use_non_causal #29773
• tools/server: new /v1/systemone endpoint for decision models #29818 and /v1/embeddings now accepts typed vision/audio/video content #29556

New models

• GLM-5.3-Flash (GLM5-Next): 320B KDA/DSA hybrid text+vision model with mHC and MoE #27773
• Clef decision model, fully supported with both text and vision #29831 #29969
• Ling 3.0 VL, folded into the BailingMoeV3 architecture #29151
• Nimble decision model #29844
• Registered Lfm2BidirectionalForMaskedLM for LFM2.5-Encoder-230M/350M #29862
• Added classifier_pooling support for rerankers #29627

Core changes

• Migrated examples, speculative decoding, mtmd and server to the new llama_batch_ext API #29385 #29601; batches now accept both embd and raw tokens #29622
• Added llama_prec_policy and a model-driven W4A4 (NVFP4/MXFP4) mul_mat path #24364
• Qwen4Exp: halved indexer score memory #29825, optimized mask constructions #29824, re-enabled the -sm tensor #28569; GLM5-Next: unique scatter rows for dead indexer slots #29745
• KV cache: fixed restoring mismatched KV cache rotation #28498, fixed K/V and recurrent state cleanup after failed restores #27530 and an invalid assert in recurrent memory #29799
• Speculative decoding: probabilistic sampling for simple draft and MTP #27694, fixed n-gram drafts rejected at temp > 0 after truncation #29924, preserved original batch order for layer inputs #29019, stop accepting draft tokens at EOG #29638
• k-pool models: fixed unexpected graph reallocation #29958 and clamped kpool re-pool bound to existing pools #29805
• Fixed tensor split for fused qkv with uneven K/V head sizes #29294, gather recurrent states once so the reserve covers every split #29856, and properly handle KV on training #28520
• Context: do not re-reserve the scheduler when toggling causal_attn #28751
• DFlash drafts: write Gemma embedding scale during conversion #29802 and add dflash support for MiMo #29650

Multi-modality changes

• Cap max_image to n_ubatch for non-causal models #29773
• Fixed the mel preprocessor in LFM2 audio #29403
• Migrated input processing to llama_batch_ext #29385

Server changes

• New /v1/systemone decision-model API with dedicated decision pipeline #29818, extended to the nimble decision model #29844
• Support vision input for Clef #29969
• GET /v1/models and GET /models now report model input/output modalities in a new architecture object #29987
• /v1/embeddings: accept typed content (vision/audio/video) input #29556 and return HTTP 400 for invalid embedding requests #29060
• Allow RANK pooling batch splitting for causal LLM rerankers (Qwen3, Qwen3-VL) #28876
• Reject partial media truncation #24076
• Logging: self-contained colors and split child commands from logs in router mode #29895, allow preset to set log file #29334, fixed dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list #29938
• Fixed laya abort by limiting n_batch to n_ubatch #29903
• Remove the built-in UI's service worker when the UI is not served #29565

UI changes

• New model download pipeline #27959, Hugging Face Hub data layer #27947, model memory-fit estimation #27957 and model id grammar for sidecars, quants and capability parsing #27946
• Type-safe API types, fetch helpers and download-ready models store plumbing #29582
• Shared model display primitives #29644
• Fixed missing svg use and animation elements in preview and download #28962
• Use toLocaleString() formatting consistently across chat message statistics #27990

ggml changes

• ggml updated to v0.26.0: a new alloc_buffer_n/get_alloc_size_n buffer allocation API, sparse flash attention kernels on SYCL, Vulkan and Metal, and major lightning indexer improvements (halved score memory, tiling, MUSA support). The CPU backend gains BF16 ops and a tiled k-quant mul_mat, CUDA gains a model-driven W4A4 (NVFP4/MXFP4) mul_mat path plus MMVQ shared-expert fusion, and the Hexagon backend adds a sampler and more quant types. Model loading is faster with stricter GGUF size validation, Windows ARM64 MSVC builds are enabled, and WebGPU/OpenVINO/OpenCL/SYCL pick up numerous new kernels, ops and fixes.

Nightly build: b11429

More info

• Releases and versioning of ggml-org projects

Changelog since v0.5.0

d81235049 llama.cpp : bump version to 0.6.0 (#29997) 4d60b4d08 common, server : report model input/output modalities in GET /models (#29987) c06f84160 sync : ggml f05c8b278 ggml : bump version to 0.26.0 (ggml/1652) e117148a4 CUDA: make the alloc_deps check batch independent (#29986) 6c59c4007 vulkan: fix Flash Attention shmem write out of bounds (#29988) 3c9e747f7 vulkan: revert mul_mat_id tile selection PR #29182 (#29936) b809b886d cuda: use the vector lightning indexer kernel on MUSA (#29990) 994e8f222 ci : add "Require Docker" flag to make-release workflow (#29989) 9d853bb36 webui: Use toLocaleString() format consistently across chat message statistics (#27990) 8f9ae20c8 ci : disable failing test on virtual Metal device (#29993) 9871df591 server: support vision input for Clef (#29969) 8b2fbaf32 CUDA: Optimize accumulation in mmq for NVFP4 type (#29857) 2ed93db47 ci : disable unused qemu in docker build (#29984) 8e1642198 server: reject partial media truncation (#24076) 806eee984 vulkan: fix stale prealloc_y reuse across flash attention and soft_max (#29591) b3daa077a vulkan: sparse flash attention for quantized K/V (#29639) c173a53bd llama : fix unexpected graph reallocation in the k-pool models (#29958) 210791069 kv-cache: fix restoring mismatched KV cache rotation by saving exact rotation metadata (#28498) e5983d670 ci : winget urls must be separate strings (#29978) 4ca6b76f0 ci : fix docker workflow permissions (#29979) 9f12cd4a4 ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 (#28479) ebe18bee5 vulkan : Fix undeclared identifiers when -DGGML_VULKAN_RUN_TESTS=ON (#29912) 8216c8462 webgpu: add MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4 (#29483) 1b43d3116 cuda: tile the lightning indexer over keys and tokens for 4 heads (#29901) a3a1c4747 metal : few-row MMA mat-mul (#29869) 9d3aba6b5 CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size (#29633) d89651a7b CUDA: prefer whole-tile FlashAttention scheduling for efficient two-stage kernels (#29435) a7fb71fab log, server: self contained colors, split child commands from logs in router mode (#29895) 0bb496dbd llama: support both embd + raw tokens in batch (#29622) 2ca15f540 CUDA: refactor swizzling code (#29612) a7b94df2c ggml-cpu: support BF16/FP16/FP32 K tails in tinyBLAS on x86 (#29806) 0eb6d9a81 cuda : move neu_padded to where it is used (#29940) 2e7c58c54 ci : windows llvm build requires ninja multi-config (#29959) 7f2dd88b0 ci : add windows arm64 vulkan release (#29954) bf79dbbcd AGENTS.md : revamp (#29656) dbe4c3ed4 chat-peg-parser : clear current_tool when pending_tool_call is reset (#29942) 46847e615 ci : set default permissions (#29945) 2bc563573 cuda : move blocks_per_col to where it is used (#29939) dd266785c CUDA: fix MMQ memory fault if n_expert >> n_ubatch (#29941) 16c163d56 vulkan: fix rdna4 mat_vec tuning (#29934) 050439614 imatrix: calculate activation-based statistics for new format (GGUF) imatrices (#14891) 8330e9696 spec : fix n-gram drafts rejected at temp > 0 after truncation (#29924) 6716df694 common : prepare load_from_models_dir() for path conversion (#29674) bf9a0ccce server : fix dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list (#29938) 0faee5004 ci : pushing tag needs deploy key (#29937) f98b31c67 ci : improve release flow (#29913) 11fe02151 webgpu: add f16 support to fill/set_rows (#29897) 836d57176 mtmd : fix deprecated strdup warning on Windows (#29863) eec18f5d3 vendor : update cpp-httplib to 0.59.0 (#29886) 1537a0a8b server : fix laya abort by limiting n_batch to n_ubatch (#29903) edd6e2bbd common : add common_is_tty() helper and fix deprecated warnings on Windows (#29860) 9bf55f4a3 chat : honor json_schema in Ling 3.0 parser (#29813) a55e952b8 ci: fix flaky ADD_ADD f16 by using the fused ADD tolerance (#29904)