llama.cpp b10419

OpenVINO: Qwen3.5, memory optimization, and test-recurrent-state-rollback (#26952)

• OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support

• OpenVINO backend: disable TOPK_MOE op test

• OpenVINO Backend: Add op FILL support

• OpenVINO backend: enable set rows with multi dims

• fix the name missmatch in setrow + view

• OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID

• OpenVINO Backend: enable SQR & SQRT

• OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer

• OpenVINO backend: enable fallback for openVINO to CPU backend

• OpenVINO backend: fix accurace issue in gemma3n arch test

• fix mpt failed case

• OpenVINO backend: clean nodeinfo

• OpenVINO Backend: enable zero-size copy for view

• add concat ssm_conv in compute_dynamic_dim

enable qwen35

Fix after rebase

remove logging

• OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region

• OpenVINO backend: fix CPY op test failed issue

• OpenVINO backend: fix GATED_DELTA_NET op test failed issue

• handle in-place op, handle qwen35 dynamic clearing of cache in cgraph

• handle qwen35 dynamic clearing of cache correctly

• Enable qwen35 dense multi seq

• Fix qwen35 9b gqa

• Fix after rebase

• Disable SOLVE_TRI

• openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)

In stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads, head_size]) into the Multiply against the rank-4 cos/sin tables ([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the OpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage output (e.g. Phi-3-mini). Lift the data to rank-4 before the split/ Multiply so the operands are equal-rank, matching what the TYPE_NORMAL branch already does. CPU and stateless paths are unaffected.

Phi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful: before: PPL = 27120.43 after: PPL = 6.2263 (CPU reference: 6.2251)

• OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update

• OpenVINO Backenb: remove changes in llama.cpp

• Doc change (use x64 Native Tools Command Prompt for VS)

• Cleaner sentence

Co-authored-by: Copilot Autofix powered by AI

• OpenVINO Backend: cache key upgrade includes all src name

• OpenVINO Backend: enable llama arch test on ci

• OpenVINO Backend: move parameter node creating from decoder into translate

• OpenVINO Backend: create extra input ov node move from decoder to translate

• fix for op regression due to is_model_splitted

• openvino: fix CPY writeback for recurrent state rollback

Detect the rollback conv/gdn state writeback CPY nodes structurally instead of by tensor name, since the rollback path in build_conv_state does not call cb() and left the nodes unnamed. Add per-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the cached IR handles any kv head, sequence count and snapshot slot for both the conv state and the GDN state writeback.

Assisted-by: GitHub Copilot

• qwen35 moe

• optimize MoE expert aggregation with ReduceSum

• Skip GET_ROWS inaccurate test

• openvino: fallback dynamic MUL_MAT_ID shapes

• OpenVINO Backend: fix error in arch test model mpt

• fix error caused by cpy in arch test model kimi-linear

• OpenVINO Backend: fix error in arch test model minimax-m3

• openvino: fix GPU mul_mat_id op tests

• ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU

The OpenVINO weight Constants are zero-copy views into host buffers allocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU the plugin holds its own device copy after compile_model, so these host pages are dead weight for inference. For a 1B Q4_K_M model this leaves ~850 MB of host RSS resident that the GPU path never reads again.

Add an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s the registered host weight buffers once the model is compiled, dropping their resident pages while keeping the mappings valid (ggml still owns the lifetime; tensors still point in). Measured steady-state RSS drops from ~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged throughput and correct output.

The GPU backend uses a single dynamic-shape model for both prefill and decode, so a graph is compiled once and reused; the only event that forces a recompile is clear_caches() on backend teardown. The change therefore:

• releases on the first cache-hit (model compiled, plugin has its copy);
• pins the compiled-model cache across backend teardown so a later

context reuses it instead of recompiling against the dropped pages;

• fails loud (GGML_ABORT) on a cache-miss recompile or on a second model

load, both of which would otherwise read zeroed weights or silently reuse the wrong compiled graph.

Scope/limitations (all fail loud, never silently wrong): GPU only (the CPU plugin reads the host Constants at inference time), one model per process, and stable graph shapes. This reduces steady-state RSS, not the transient compile-time peak. All changes are confined to the OpenVINO backend.

• ggml-openvino: stream weight requantization to cut the compile-time RSS peak

requantize_to_buffers() dequantized the entire tensor to a temporary std::vector of n_elements before requantizing. For token_embd.weight (128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is the single largest contributor to the OpenVINO compile-time memory peak -- it also fires twice for token_embd (once at load, once at graph build, because token_embd is loaded via a CPU/mmap buffer and not cached as an OV weight extra).

Stream the dequant instead: process a fixed window of complete rows (CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk straight into the output buffers. The transient F32 footprint is now CHUNK_ROWS*ne0 floats regardless of tensor size.

quantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk writes its weights/scales/zp at the correct block. Streaming is applied to the Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0) path keeps the whole-array call because it packs two weights per byte with running zp ORs, and a fallback handles any future target whose block size does not divide a row.

Measured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB); 8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged ("capital of France is Paris"); throughput unchanged. Unlike GGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just steady-state, and needs no env flag. All changes confined to the OpenVINO backend.

• ggml-openvino: avoid redundant token_embd requantization at compile

token_embd.weight is referenced twice in the graph path: as the GET_ROWS embedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on every weight-node build, and is_model_splitted() built a full (naive) set of weight nodes just to test name membership - each requant is a ~1-2 GB F32 dequant of the 262M-element embedding.

Two changes:

• Add collect_weight_names(): a name-only collector for topology checks.

is_model_splitted() now uses it instead of create_weight_nodes(cgraph, true), so the splitted-check no longer triggers any weight extraction.

• Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime

cache keyed by tensor->data. These tensors have no OV buffer context to own a cached extra, so without this they were rebuilt on every (re)compile; prefill and decode graphs now share one build (verified: 2nd graph hits the cache instead of re-requantizing).

Peak RSS is unchanged (the streaming-requant commit already removed the F32 transient); this removes redundant compile-time work. Output verified unchanged ("capital of France is Paris"). Confined to the OpenVINO backend.

• ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM

The streaming requantization and the non-OpenVINO-buffer weight-node cache (plus the name-only is_model_splitted path that pairs with it) are now opt-in via GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers() fully materializes the F32 buffer and weights are rebuilt per compile exactly as before; when set, the streaming path and the cross-compile weight cache are used.

Default off keeps behavior identical to upstream unless explicitly enabled. Verified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output "capital of France is Paris" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS, added earlier, remains a separate opt-in for the steady-state release.)

• ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)

The plugin-level ov::cache_dir caches the compiled blob keyed by the OV model, but producing that model still runs the full frontend every time: weight requantization (incl. the large token_embd F32 transient) and the ggml->OV graph conversion. This adds an opt-in frontend cache keyed off a fingerprint computed directly from the ggml cgraph, so a hit imports a previously exported CompiledModel and skips requant + convert + compile entirely.

Key (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per node op/name), a sampled per-weight fingerprint (name/shape/type + bounded head+tail byte sample), and blob-affecting config (device, flash-attn, rope params, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar manifest stores every weight's fingerprint and is re-verified on load, so a sampled-hash collision cannot cause a wrong-model hit (verified: two different quantizations of the same model produce distinct cache entries).

Flow (dynamic single-model path only; split models defer to ov::cache_dir): on a verified hit, core.import_model() restores the CompiledModel and a lightweight decoder is built with a names-only weight map (membership is all the decoder needs for I/O mapping; weights live in the imported model). On a miss, compile as usual then export the blob (atomic temp+rename, manifest written first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/ CACHE_MODE are stripped from the config used for the cached compile and the import - a blob compiled with cache_dir set cannot be re-imported.

Measured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s (~2.4x faster compile phase). Output verified unchanged on cold and warm, standalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default off; confined to the OpenVINO backend.

• ggml-openvino: harden frontend model cache correctness

The frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.

Add a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.

DISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.

Also make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.

While touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.

• ggml-openvino: add memory optimization umbrella switch

Add GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.

Centralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.

Keep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.