LocalAI 4.10.0

LocalAI 4.10.0 Release!

LocalAI 4.10.0 is out!

Twenty-eight days and 280 pull requests. The work landed on three fronts: operating a fleet, feeding it models from private sources, and fixing the backends you depend on. A fleet operations dashboard replaces the flat node list with cluster-wide health, capacity, running models, and bulk lifecycle actions. A credentials file lets one credentials.yaml authenticate OCI registries, galleries and direct downloads without scattering tokens across environment variables. And local-ai benchmark measures end-to-end latency and throughput from the CLI.

The ds4 backend had four fixes that together make it reliable on CUDA: the kernels now build for the target architecture, abandoned inference is cancelled, generation boundaries are enforced, and prefilled reasoning is separated from content. Distributed mode gained seven stabilisation fixes across staging, liveness, virtual models and the control-plane database. Four CVEs were patched, enable_thinking=false is now honoured across every text backend, and Apple M5 no longer SIGSEGVs at startup.

Highlights:

• Fleet operations dashboard - a real-time cluster overview replacing the flat node list. Aggregate health bands, capacity gauges for VRAM/RAM/CPU/disk, a running-models tab showing replica placement across workers, bulk drain/resume/remove with bounded concurrency, and a slide-out node inspector. Workers now report CPU telemetry at registration and heartbeat.
• Credentials file for private sources - one credentials.yaml authenticates OCI registries, galleries, HTTP downloads and github: URIs. Supports basic auth, bearer tokens and custom headers, with secrets read at use time so rotated K8s secret mounts take effect without restart, and full masking in logs and error messages.
• local-ai benchmark CLI command - measures end-to-end latency and throughput against configured text models. Reports min/mean/max latency and completion tokens/sec across warmup and measured runs, with table and JSON output, redirect blocking, and API-key redaction.
• vllm-cpp LoRA fusion at load time - lora_adapters and lora_scales are now consumed by the vllm-cpp video engine, fusing LoRA deltas into DiT weights at engine load. Works for both LTX2.5 and MiniMax-H3 through one generic path.
• Environment variables per model - an env: map in model config injects variables into the backend process environment, so CUDA_VISIBLE_DEVICES or VLLM_CACHE_DIR can be pinned per model without wrapping the backend in a script.
• template.system_messages_after_first - agent frameworks that append system instructions mid-conversation no longer crash Qwen3.x models. Late system turns can be merged into the leading system message or forwarded as user turns.
• context_size in /v1/models/capabilities - clients can now read each model's effective context window to budget tokens and truncate prompts without probing the model config out-of-band.
• Four CVEs patched - ip-address 10.3.1 (CVE-2026-69192), containerd 1.7.32 and 1.7.33 (CVE-2026-46680, CVE-2026-53488), and react-router (GHSA-qwww-vcr4-c8h2).
• Apple M5 SIGSEGV fixed - go-m1cpu bumped to v0.2.2, fixing a segmentation fault on Apple M5 hardware at startup.

Plus a focus mode toggle in chat settings, face enrollment replay for the 1:N registry, a ROCm build of stablediffusion-ggml, in-place editing of distributed scheduling rules, OCI layer download resumption, and 140 new gallery entries.

---

This release in numbers

| | | |---|---| | Pull requests merged | 280 | | Commits | 382 | | Files changed | 633 (+46,550 / -3,316) | | Development window | 28 days (2026-08-20 to 2026-09-17) | | Human contributors | 29, of whom 16 first-time | | Gallery entries | 1,707 to 1,847 (+140) |

Where the work landed:

| Area | Change | |---|---| | core/ | +25,773 / -1,993 across 314 files | | gallery/ | +6,734 / -417 | | backend/ | +4,932 / -433 across 154 files | | pkg/ | +4,392 / -91 across 55 files | | docs/ | +2,526 / -101 across 48 files | | swagger/ | +322 |

---

TL;DR

| Area | Summary | |------|---------| | Fleet dashboard | The Nodes page is rewritten as a fleet operations dashboard. A ClusterOverview shows health bands (healthy/draining/attention) and capacity gauges for VRAM, RAM, CPU and disk derived from the single GET /api/nodes response. A NodeFleetTable offers sortable, groupable, paginated rows with checkbox selection across filters and pages, and bulk Drain/Resume/Remove with bounded concurrency (limit 8). A Running Models tab shows all loaded replicas on healthy workers grouped by model, with replica count, node count, in-flight requests and backend types. Slide-out inspectors for nodes and models provide detail and drill-down. Workers now report cpu_logical_cores, cpu_usage_percent and cpu_load_1 at registration and heartbeat. Lifecycle transitions are atomic: MarkDraining flips status and deletes model records in one transaction, ResumeNode is conditional (draining to healthy only), and HTTP endpoints return proper 404/409 instead of generic 500. | | Credentials file | --credentials-file / LOCALAI_CREDENTIALS_FILE loads a YAML list of rules, each matching a URL prefix and carrying one auth type: basic (username + password/password_env/password_file), bearer (bearer/bearer_env/bearer_file), or custom header (header.name + header.value/_env/_file). OCI registries consult the store first, then docker config. HTTP downloads, galleries and github: URIs authenticate through a RoundTripper that clones the request per redirect hop, so a cross-host redirect never carries the credential. Secrets are read at use time, not load time, so a rotated K8s secret mount takes effect without restart. Credential and Store implement String, GoString and LogValue so no fmt verb or slog call can print secret material. Unknown YAML keys are errors; match strings with user:token@, ? or # are rejected at load. | | CLI benchmark | local-ai benchmark MODEL [MODEL ...] sends non-streaming POST /v1/chat/completions requests at temperature:0 with configurable --runs (default 3), --warmup (default 1), --max-tokens (default 128) and --timeout (default 5m). Reports min/mean/max latency and end-to-end completion tokens/sec per model. Output is a table by default or --json for structured data with per-sample latency and nullable token counts. The HTTP client refuses redirects, the API key is redacted from error messages, and the full report is buffered and only written after all models succeed, so a failure leaves no partial output. | | vllm-cpp LoRA fusion | The lora_adapters, lora_scales, lora_adapter and lora_scale config fields are now consumed by the vllm-cpp video backend. buildLoraExtras converts them into indexed lora_path/lora_strength key-value pairs (first adapter unsuffixed, subsequent ones _2, _3, ...), packs them via the ABI v18 extras seam, and hands them to vllmVideoEngineLoad. vllm.cpp fuses the LoRA deltas into the DiT weights at load. One generic path serves both LTX2.5 and MiniMax-H3. Adapters are always loaded; per-request prompt-activated LoRA is out of scope. | | Per-model env vars | An env: map on the model config is appended to the child process environment after the base env, so model vars override inherited system vars. Delivered through the common process-spawning layer in pkg/model/process.go, so every gRPC backend inherits it with no per-backend wiring. The UI renders an editable key/value map. Model vars take effect only when a model spawns its own backend process, not when it reuses a worker-supervised one. | | Late system messages | template.system_messages_after_first controls what happens to system-role messages after the leading system block. "merge" folds them into the first system message with a \n\n separator; "user" rewrites them in place to role user; unset passes through. Fixes a Jinja exception (System message must be at the beginning) raised by Qwen3.x tokenizer templates when agent frameworks append system instructions mid-conversation. Runs after stripEmptySystemMessages and before tokenizer-template system_prompt injection. | | context_size exposed | /v1/models/capabilities now populates entry.ContextSize with backend.EffectiveContextSize(cfg) when the result is positive, letting clients budget tokens and truncate prompts without probing the model config out-of-band. | | CVE fixes | ip-address to 10.3.1 (CVE-2026-69192), containerd to 1.7.32 (CVE-2026-46680) then 1.7.33 (CVE-2026-53488), react-router to 7.18.2 / 8.3.0 (GHSA-qwww-vcr4-c8h2). | | Apple M5 | go-m1cpu bumped to v0.2.2, fixing a SIGSEGV on Apple M5 at startup. | | Models | 140 new gallery entries: Qwopus 3.6, OpenResearcher, LFM2.5 DSpark, Qwen3.8 small distills and Flash Next, UI-Mate 27B, Homura 30B, Ornith 1.5 35B and 9B, Granite 4.2, Tiel-Coder 35B, Thomson 1.0 Small, PhoneLLM, WeMM embedding, EXL3 vllm.cpp, Gemma 4 E4B, Orukeet for NeMo speech, and more. |

---

New Features & Major Enhancements

Fleet operations dashboard

The Nodes page was a flat list of per-node cards. It showed you what was registered but not what the cluster was doing, and every lifecycle action was one node at a time.

The rewrite is a fleet operations dashboard with two tabs and a shared summary poller.

ClusterOverview sits at the top. A health band segments nodes into healthy, draining and attention counts. Capacity gauges show VRAM, RAM, CPU and disk, each reporting how many workers do not report that metric so the number is honest rather than averaged over reporting nodes. An attention queue filters for pending approval, offline/unhealthy, low VRAM, low RAM and low disk, surfacing nodes that need action without double-counting the headline total.

NodeFleetTable is sortable, groupable (by label or type) and paginated at 50 rows per page, with checkbox selection that persists across filters and pages. Bulk Drain, Resume and Remove run with bounded concurrency (limit 8) so a fleet with thousands of registrations is usable. Columns: status pill, address, capacity (VRAM + RAM), CPU, workload and heartbeat. An Approve button appears on pending nodes.

Running models tab. A fleet-wide view of loaded replicas on healthy workers, grouped by model: model name, replica count, node count, in-flight requests, backend types and last used. Per-model Stop shuts down all placements through POST /backend/shutdown {model}, which now fires the remote unloader even when a local process exists, and deduplicates nodes before sending stop messages.

Inspectors. A slide-out NodeInspector shows health, labels, resource bars, CPU, loaded model count, backends (lazy-loaded) and in-flight, with Approve/Drain/Resume actions. A ModelInspector shows replica placement grouped by node, with clickable node names that drill into a NodeInspector.

Workers now report cpu_logical_cores, cpu_usage_percent and cpu_load_1 at registration and heartbeat. The dashboard shows CPU capacity and utilization, with idle cores derived as cpu_logical_cores * (1 - cpu_usage_percent / 100).

Lifecycle transitions are now atomic. MarkDraining flips status and deletes model records in one transaction rather than as separate operations that could interleave with SetNodeModel. ResumeNode is conditional: draining to healthy only, preventing overwriting of pending approval or concurrent state changes. HTTP endpoints return proper 404 (not found) and 409 (status conflict) instead of a generic 500.

PRs: #12046

Credentials file for private sources

Backends, models and gallery indexes can live behind private OCI registries, internal HTTP servers or private GitHub repos. Before this, each download path had its own ad-hoc auth mechanism, and tokens were scattered across environment variables.

One credentials.yaml file, loaded by --credentials-file / LOCALAI_CREDENTIALS_FILE, authenticates them all. It is a top-level YAML list of rules. Each rule has a match URL prefix and exactly one auth type: basic (username + password / password_env / password_file), bearer (bearer / bearer_env / bearer_file), or custom header (header.name + header.value / header.value_env / header.value_file). Exactly one of the plain, _env or _file form per secret; unknown keys are errors so a misspelled pasword_env fails loudly.

OCI registries (backend images, oci:// and ollama:// models, digest lookup, Range resume, cosign verify) consult the store first, then authn.DefaultKeychain (docker config). HTTP downloads, galleries and github: URIs authenticate through an http.RoundTripper that puts the credential on a clone of the request. net/http rebuilds each redirect hop from the original headers, so a redirect to another host never carries the credential. A caller-supplied credential (e.g. HF_TOKEN) takes precedence: the transport leaves an existing Authorization header alone.

Secrets are read at use time, not load time, so a rotated K8s secret mount takes effect without restart. Credential and Store implement String, GoString and LogValue, so no fmt verb or slog call, including when nested in unexported fields, can print secret material. Credentials go only to https, or to http when the rule sets allow_insecure. Match strings containing user:token@, ? (where signed URLs carry tokens) or # are rejected at load. YAML decode errors are redacted: quoted scalar values (often secrets) are stripped, keeping only line numbers and parser wording.

PRs: #12028

local-ai benchmark CLI command

A benchmark command for configured text models, answering "how fast is this model on this hardware" without leaving the terminal.

local-ai benchmark MODEL [MODEL ...] sends non-streaming POST /v1/chat/completions requests at temperature:0 with configurable --runs (default 3), --warmup (default 1), --max-tokens (default 128) and --timeout (default 5m). It reports min, mean and max latency and end-to-end completion tokens/sec per model.