LocalAI 4.8.0

LocalAI 4.8.0 Release!

LocalAI 4.8.0 is out!

Twenty-two days, 386 pull requests, and three new modalities. This release introduces vllm.cpp, a C++20 engine maintained by the LocalAI team, which began as a vLLM port and now carries its own featureset, shipping as the vllm-cpp backend in alpha development builds. Around it: 3D generation as a new modality, a multi-family audio.cpp engine, gallery entries that install the build your hardware can actually run, and a deep reliability pass on distributed mode driven by production incidents.

Highlights:

• vllm.cpp (alpha) - a C++20 engine maintained by the LocalAI team, which began as a vLLM port and keeps vLLM as its reference implementation: V1 serving architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Measured at 1.045x vLLM on Qwen3.6-27B NVFP4 at concurrency 1, with token-for-token identical output. Loads safetensors and GGUF, enforces structured output in-engine, and runs on CPU, CUDA, Metal and Vulkan. The Apple Silicon build ships the MLX GEMM provider, measured at 1.5x to 2.2x on an M4. Shipping as alpha development builds: try it, do not depend on it.
• 3D generation - a new modality end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend, and a UI page with a native GLB viewer and print remeshing.
• audio.cpp - one backend process serving six audio endpoints across many model families, picked from the GGUF's own metadata: speech, transcription, VAD, diarization, source separation and sound generation.
• One model, many builds - a gallery entry can declare variants:, and LocalAI installs the largest build that your host can actually run. No more hunting through the gallery for the right quantization.
• A much lighter web UI - gzip on the wire, immutable caching for hashed assets, and paginated trace endpoints: the React bundle is 3.48x smaller and the trace poll dropped from 21 MB to 7 KB.
• An Activity page - the stacked operations bar collapses to one line, and a new admin Activity page keeps the record of what installed, failed or was cancelled, instead of dropping it the moment it finished.
• Hugging Face artifact materialization - immutable snapshot resolution, authenticated downloads with real progress, and staged artifacts that remote workers can bind to.
• VRAM budgets - cap how much of a card LocalAI may use, per node, as a percentage (80%) or an absolute amount (12GB).
• Two new TTS engines - magpie-tts-cpp (NVIDIA Magpie Multilingual, 5 voices, 9+ languages) and moss-tts-cpp (48 kHz stereo with reference-audio voice cloning).
• Sub-2-bit models - a new bonsai backend serves the 1-bit and ternary Bonsai quantizations of Qwen3 and Qwen3.6-27B.
• Distributed mode hardening - the reaper no longer deletes rows for backends that are alive and busy, phantom replicas are cleaned up, and in_flight counters stop leaking.

Plus a Valkey vector store, systemd socket activation, persistent trace history, two security fixes, a documentation overhaul aimed squarely at onboarding, and a new localai.io.

---

This release in numbers

| | | |---|---| | Pull requests merged | 386 | | Commits | 392 | | Files changed | 1,204 (+152,918 / -42,012) | | Development window | 22 days (2026-07-14 to 2026-08-05) | | Contributors | 25, of whom 11 first-time | | New backends | 7 (vllm-cpp, audio-cpp, trellis2cpp, valkey-store, bonsai, magpie-tts-cpp, moss-tts-cpp) | | Gallery entries | 1,221 to 1,515 (+294) |

Where the work landed:

| Area | Change | |---|---| | core/ | +54,974 / -8,291 across 546 files | | gallery/ | +39,193 / -29,119 (variant ladders rewrite most of the index) | | backend/ | +25,668 / -1,305 across 256 files | | pkg/ | +9,678 / -385 | | .github/ | +7,526 / -61 | | docs/ | +4,285 / -2,461 (near-flat by design: the dedup pass removed as much as it added) | | website/ | +4,479 (new project site) | | scripts/ | +3,373 / -175 |

---

TL;DR

| Area | Summary | |------|---------| | vllm.cpp (alpha) | An Apache-2.0 C++20 engine maintained by the LocalAI team and developed in its own repository, which began as a vLLM port, shipping as alpha development builds of the vllm-cpp backend over its stable C ABI v5. It uses vLLM as its reference implementation and benchmark, and implements vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Safetensors + GGUF, in-engine structured output (JSON schema / regex / choice / GBNF). Chat and tool calling ride the llama.cpp autoparser path: full minja templates, tool_choice: auto as a lazy structural-tag constraint, 30 tool dialects and 7 reasoning parsers, streamed ChatDelta/ToolCallDelta. CPU amd64/arm64, CUDA 12/13 (Blackwell), L4T, Vulkan and Darwin Metal, the last with the MLX GEMM provider vendored in (1.5x to 2.2x on an M4). | | 3D generation | A new modality, wired end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend over TRELLIS.2, and a UI page with a native GLB viewer, IndexedDB history and previewable print remeshing. | | audio.cpp | New native C++ backend over audio.cpp, a multi-family ggml audio engine: one process serves /v1/audio/speech (supertonic, chatterbox, irodori-voicedesign), /v1/audio/transcriptions (citrinet, nemotron, forced-aligner), /v1/audio/vad, /v1/audio/diarize (sortformer), /audio/transform (htdemucs 4-stem separation, voice conversion, speech-to-speech) and /v1/sound-generation. Family comes from the GGUF's own audiocpp.model_spec.family key, so no per-model backend options. 13 gallery entries. CPU, CUDA 12/13, Vulkan, Metal. | | Activity page | The stacked operations bar becomes a permanent one-line strip (` now hides rather than cancels), with a new admin /app/activity page: in-progress detail with per-node breakdown, a "needs attention" lane with Cancel and Retry, and a bounded 50-entry record of what finished. | | **Valkey vector store** | New valkey-store backend adding Valkey Search as a vector store option. | | **New localai.io** | The site splits into a project site at the root and docs under /docs/, with 214 generated redirect stubs so every published URL keeps working. Adds an engines page driven by YAML, a blog, an ecosystem band and ADOPTERS.md. | | **Gallery variants** | An entry may declare variants: referencing other entries. Install-time selection drops builds the host cannot run (IsBackendCompatible) or cannot fit (VRAM, or cgroup-aware RAM on CPU hosts), then picks the largest that fits. Override with variant on POST /models/apply, local-ai models install --variant, the install_model MCP tool, or the UI split-button. GET /api/models?has_variants=true narrows the list. Older clients ignore the key and install the entry as before. | | **HF artifacts** | Immutable snapshot resolution, authenticated downloads with progress, gallery install and preload materialization, runtime binding to staged artifacts, and UI progress reporting. Python backends reuse the Go download path. | | **HTTP performance** | gzip middleware (--disable-http-compression, --http-compression-min-length), with streaming paths explicitly skipped. /assets/* served immutable, index.html no-cache. /api/traces and /api/backend-traces accept limit/offset/full and summarize by default, with GET /api/traces/{id} for the full record. React bundle 2,815,513 B to 807,918 B; backend-trace poll 21,131,097 B to 7,201 B. | | **VRAM budget** | LOCALAI_VRAM_BUDGET=80% or =12GB (also --vram-budget), on local-ai and local-ai worker. Standalone it is a hard per-process cap inherited by context-fit, GGUF warnings and the watchdog; distributed it is a placement ceiling the scheduler respects. Admin override via PUT/DELETE /api/nodes/:id/vram-budget and the set_node_vram_budget MCP tool. Unset means all detected VRAM. | | **magpie-tts-cpp** | New Go/purego backend over [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp), a ggml port of NVIDIA Magpie TTS Multilingual 357M with NanoCodec embedded. 5 voices, 9+ languages, 22.05 kHz mono, one self-contained GGUF. | | **moss-tts-cpp** | New Go/purego backend over [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) for MOSS-TTS-Local v1.5. 48 kHz stereo, optional reference-audio voice cloning, no Python at inference. | | **bonsai** | New backend on the [PrismML llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp), which is the only decoder for the Q1_0 and Q2_0 quant formats. Eight gallery entries across Bonsai 8B/27B and Ternary-Bonsai 8B/27B, from ~1.15 GB. | | **Distributed reliability** | A busy backend is no longer reaped: the worker is asked directly over a new models.running subject, and the port-probe fallback distinguishes DeadlineExceeded (busy) from Unavailable (gone), requiring three consecutive misses. Frontend model stubs are dropped when no healthy replica remains, in_flight leaks are closed, and model-load deadlines scale with checkpoint size. | | **Security** | Inline GRPO reward code in POST /api/fine-tuning/jobs is refused unless the operator sets LOCALAI_TRL_ALLOW_INLINE_REWARD=true`; the previous builtin allowlist was escapable to arbitrary code execution on an endpoint that is unauthenticated by default. Also picks up hono 4.12.25 for CVE-2026-54290. | | Models | MiniMax-M3, Gemma 4 llama.cpp MTP variants, Qwen3.5-4B DFlash, MOSS-TTS-Local v1.5, the APEX families as variant ladders, and the Bonsai families. Duplicate entries removed and linted against recurring. | | Docs | Onboarding overhaul: one model carried through install to first API call, a new "Build your first agent" walkthrough, a runtime-errors reference keyed on literal error strings, an agent actions catalog, and a new Operations section. |

---

New Features & Major Enhancements

Introducing vllm.cpp (alpha)

vllm.cpp against vLLM on Qwen3.6-27B, identical output at every concurrency.

vllm.cpp is Apache-2.0, maintained by the LocalAI team, and began as a C++20 port of vLLM. We want it community-first rather than a LocalAI-only engine, so it lives in its own repository with its own docs, benchmark record and issue tracker, and it is usable without LocalAI anywhere in the picture. It implements vLLM's V1 serving architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) on a portable tensor runtime with no Python, no PyTorch and no ggml at inference time, and uses vLLM itself as its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it.

It has since grown a featureset vLLM does not have, which is what the port was for. It loads GGUF as well as Hugging Face safetensors, runs on CPU, Apple Metal and Vulkan alongside NVIDIA CUDA, ships speculative decoding and KV offload, and enforces structured output in-engine (JSON schema, regex, choice, GBNF). Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because those are the engines it actually competes with on that hardware.

The project is expected to be renamed, with the new name still to be decided. It is drifting far enough from vLLM that calling it a port undersells it and calling it vllm.cpp will eventually mislead.

Numbers, from the project's own scoreboard

These come from vllm.cpp's BENCHMARKS.md, which reports ties as ties and losses as losses. Throughput is vllm.cpp over the reference, so above 1.0 is ahead.

| Reference | Workload | Result | |---|---|---| | vLLM | Qwen3.6-27B NVFP4, GB10 | 1.045x at concurrency 1, 1.007x to 1.017x at c2 to c32, output token-for-token identical | | vLLM | Qwen3.6-35B-A3B NVFP4, GB10 | 1.010x at c16 and 1.013x at c32; behind at c1 to c8 (0.817x at c1) | | vLLM | DeepSeek-V2-Lite MLA, GB10 | 0.86x to 0.95x throughput, TTFT ahead at c4 and c8 | | llama.cpp | Qwen3.5-2B GGUF, CPU aarch64 | prefill 1.18x, decode a tie, memory parity, byte-identical output | | MLX-LM | Qwen3-0.6B, Apple M4 | 97.6% of warm total, prefill ahead | | DwarfStar (ds4) | DeepSeek-V4-Flash IQ2_XXS, one DGX Spark | 18.69 vs 16.33 tok/s decode, 1.144x, same output | | vLLM | Laguna-XS-2.1 NVFP4, GB10 | 44.46 vs 43.10 tok/s, 1.03x, same output |

The upstream page is careful about its own noise band: on the 27B grid it calls c2 through c32 ties rather than wins, because the run-to-run spread is 0.5% and those margins land between 0.7% and 1.7%. The c1 result is the one it stands behind.

The DeepSeek-V4-Flash row is the one that shows how far the project has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 18.69 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible.

That figure moved twice in a week, and the second move came from one lever. The dense Q8_0 projection tower was being read from the GGUF mmap over unified memory, which the GB10 reads about 20% slower per-GEMV than device memory. Staging that ~6 GiB tower device-resident once at load, same bytes and same kernels, took decode from 16.23 to 18.69, generating the same tokens and using no more peak memory. The same change took Laguna-XS-2.1 from 87% of vLLM to 1.03x ahead of it.

Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 generates the same tokens as vLLM's MTP and runs about 4% faster at concurrency 1.