# Self-hosted model servers: how they place models, and where nobody has solved it *Researched 2026-10-08 (claude session e053c857, via a web-research agent). Dates/versions are what was seen that day. Some details came from search summaries or third-party mirrors where the repo could not be opened — marked where it matters. Frozen: a re-check is a new dated file.* Question (operator): "where has this problem reached a solution void?" — the problem being one OpenAI-compatible front door per box that decides, per call, what to load, what to evict and how to make a model fit, across engine types, and learns from what loads really cost. ## The projects ### LocalAI (mudler/LocalAI) - Go server, backends as separate processes; Docker or binary. ~v4.2/4.3 (May 2026, per mirrored changelogs https://gitea.varghacsongor.hu/GithubMirror/LocalAI); 3.10 (Jan 2026) added video + image. - **Broadest modalities:** chat, embeddings, ASR (whisper.cpp, faster-whisper), TTS, image + video (diffusers, LTX-2), realtime, diarization, face recognition; ~36 backends. - **Load/unload:** `--max-active-backends` (LRU count), idle/busy watchdogs, concurrency groups, pinned, `--vram-budget` (a ceiling). Busy models are not evicted: 30 retries at 1 s, then "proceed (which may cause out-of-memory errors)". Docs: it "cannot reliably estimate VRAM use for new models across backends"; the gallery shows a metadata estimate. https://localai.io/docs/advanced/vram-management/ · https://localai.io/docs/features/runtime-settings/ - **Auto-fit:** none documented beyond passthrough (e.g. `gpu_layers`). - **Cross-engine:** one LRU count + one VRAM ceiling across all backends; no per-model accounting. - **Fleet:** distributed mode (PostgreSQL + NATS); workers send MEASURED free VRAM each heartbeat; scheduler: node with the model → node with enough free VRAM → idle → least-loaded → LRU eviction. The model's need is not estimated (disk size + 5% check only). No history. https://localai.io/docs/features/distributed-mode/ - **Telemetry:** `/metrics`, OpenTelemetry, per-user usage; only free VRAM feeds placement. - **Auth:** open when unconfigured; legacy keys = full admin; user/OAuth system (required in distributed mode). https://localai.io/docs/features/authentication/ ### llama-swap (mostlygeek/llama-swap) — the closest design to ours - Go proxy, one YAML config; binary / Homebrew / WinGet / a "unified" Docker image with llama-server, stable-diffusion.cpp, whisper.cpp, audio.cpp. Very active (issues closed 2026-10-07). https://github.com/mostlygeek/llama-swap - **Modalities:** chat, completions, responses, Anthropic messages, embeddings/rerank, ASR, TTS, image generation/edit (video for sd.cpp requested, #976). Any engine = a configured command. - **Load/unload: purely static, no memory model.** Groups (`swap`, `exclusive`, `persistent`), a "matrix" DSL of allowed combinations (`(a|b) & c`) with per-model `evict_costs` (picks the cheapest allowed set), `ttl`/`globalTTL`. VRAM appears only as comments and CUDA_VISIBLE_DEVICES. https://raw.githubusercontent.com/mostlygeek/llama-swap/main/docs/config.example.yaml - **Auto-fit:** none (leans on the engine, e.g. llama.cpp `--fit`). - **Cross-engine:** yes, but by hand-written rules. - **Fleet:** `peers` = static forwarding to remote servers (`peer/model`). - **Telemetry:** Prometheus `/metrics` (GPU/system every 5 s); per-request token log to SQLite (`store.path`); request capture. Nothing feeds scheduling. - **Auth:** `apiKeys` list, empty = no auth. - **Users ask for what we built:** #1004 (open, Aug 2026) "memory-constrained scheduling with per-model RAM declarations" — and explicitly "Not asking for automatic detection (that's hard)". https://github.com/mostlygeek/llama-swap/issues/1004 · also #1127, #749, #970. ### Ollama - Go server, own engine (moving off llama.cpp); native or Docker; ~v0.31.x (Jul 2026). - **Modalities:** chat, vision, embeddings; experimental image generation since Jan 2026 (https://registry.ollama.ai/blog/image-generation). No ASR/TTS found. - **Load/unload:** `keep_alive` (5 min), `OLLAMA_MAX_LOADED_MODELS` (3 per GPU), queue + unload idle when a model does not fit (https://docs.ollama.com/faq). Sept 2025 "new model scheduling": the new engine "will now measure the exact amount of memory required" — a pre-load measurement for new-engine models; nothing indicates it is kept. https://ollama.com/blog/new-model-scheduling - **Auto-fit:** chooses GPU layers; default context scales with VRAM (4k / 32k / 256k) (https://docs.ollama.com/context-length). No automatic quant. - **Fleet:** multi-GPU on one host only. - **Telemetry:** logs, `ollama ps`; no native Prometheus found. **Auth:** none locally. - **Open problems:** #17099 vision memory estimate inflated (Jul 2026), #16617 CPU offload despite VRAM, #14632 two models on the same GPUs while another was free → CUDA OOM (https://github.com/ollama/ollama/issues/14632), #12094 crash instead of CPU fallback. Independent test of v0.12.1 found more CPU offload than claimed: https://glukhov.org/llm-performance/ollama/memory-allocation-in-ollama-new-version/ ### LM Studio (closed source) - Desktop app + `llmster` headless daemon. Chat, vision, embeddings. - JIT load on first request, 60-min idle TTL, auto-evict keeps one JIT model; "loading guardrails" refuse by estimate (`lms load --estimate-only`); the guardrail can shrink context. https://lmstudio.ai/docs/developer/core/ttl-and-auto-evict - Estimates can be far off: a headless load refused at ~22.9 GB estimated vs ~11.5 GB actual (e.g. https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2051). - Optional API token, off by default. ### GPUStack (gpustack/gpustack) — the fleet one - Python server + workers, UI; pip / Docker / Helm; v2.2.x (Jun–Jul 2026; v2.2.2 fixed missing authorization on cluster endpoints). - LLM, embeddings, rerank, image, audio (VoxBox); vLLM, SGLang, MindIE, llama-box/GGUF. - **Deploy-time scheduler**, not per-call swapping: filter workers → ESTIMATE memory (gguf-parser / backend formulas) → single GPU → multi-GPU → distributed → partial offload / CPU → binpack/spread. https://docs.gpustack.ai/latest/scheduler/ . No learning. - Heterogeneous fleet (NVIDIA, AMD, Ascend, Apple) is its strength. Prometheus + Grafana built in. Auth mandatory by default. - **Its estimate bugs are the clearest evidence of the gap:** #1601, #2101, #2412, #2443, #1899 (open), #4153, #5375 (open); #5880 (open, v2.4.0) asks only to SHOW actual usage next to allocated. https://github.com/gpustack/gpustack/issues/5880 ### Xinference (xorbitsai/inference) - Python supervisor + workers; v3.5.0 docs. LLM, embeddings, rerank, image, audio, video. - Explicit launch/terminate; "Idle First" GPU choice; v3.5 recommends engine/format/size/quant from offline metadata assuming 1 sequence × 2048 tokens × FP16 KV against 80 % of free memory — "does not guarantee sufficient memory"; not for image/video. https://inference.readthedocs.io/en/latest/user_guide/launch.html - Prometheus on by default; auth experimental and off by default. ### vLLM - One model per process; claims `gpu_memory_utilization` (0.9) of the card. "Sleep mode" offloads or drops weights for switching (https://docs.vllm.ai/en/latest/features/sleep_mode.html). No auto-fit (errors if KV does not fit). Rich Prometheus. `--api-key` off by default. - Swappers built on it: llmux (https://docs.rs/crate/llmux/0.7.6), Snowflake's "Semi-Persistence" (https://www.snowflake.com/en/blog/engineering/semi-persistence-gpu-model-swapping/). ### llama.cpp itself - **Router mode:** `--models-dir`/`--models-preset`, `--models-max` (default 4, a count + LRU, not VRAM), autoload, `--sleep-idle-seconds`; one child per model. - **`--fit` (ON by default):** the README's own words: "whether to adjust unset arguments to fit in device memory"; margin `--fit-target` 1024 MiB per device; `--fit-ctx` "minimum ctx size that can be set by --fit option" 4096 — so it shrinks context (only when `-c` is not given). The README does NOT list the other arguments it adjusts; the agent's claim that it also sets layers / tensor split / MoE experts is UNVERIFIED (hugpy-wrapper's own max-gpu path relies on it picking the layer split). Operator's experience (2026-10-08): poor results in practice. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md (checked 2026-10-08) One model vs free memory: no eviction planning, no quant choice, no history. - `--api-key` off by default; `--metrics` per model in router mode. ### Others KoboldCpp (single model; `--gpulayers -1` guesses layers, "experimental"); Paddler (llama.cpp load balancer by free slots, fleet UI, StatsD; model management only a future plan — https://github.com/intentee/paddler); LiteLLM proxy (routing/keys/budgets, never loads models); Text Generation WebUI (one model, manual). ## Where no project has a solution Documented absence or open requests (higher confidence): 1. **Measured per-model footprint feeding placement.** LocalAI says it cannot estimate across backends; GPUStack's estimate-bug trail (#1601 … #5880); llama-swap #1004 asks for hand-declared sizes and rules out automatic detection; Ollama measures pre-load for its new engine only and still ships estimate regressions. 2. **Learning from past loads.** Nobody documents keeping what a load really used (VRAM after load, CUDA context, growth during generation) and reusing it next time. 3. **Planning WHICH residents to evict so a model fits.** Everyone uses count + LRU, hand-written sets, or a fixed card fraction; LocalAI documents proceeding into a possible OOM. 4. **Choosing quant + context + layers + MoE offload together, with eviction.** llama.cpp `--fit` (documented: unset args incl. context, min 4096) and LM Studio's guardrail fit ONE model against free memory; Xinference recommends quant from a fixed assumption (not image/video). None combines it with eviction or measured history. 5. **One memory planner across engine types on one card.** llama-swap/LocalAI run llama.cpp, diffusers, whisper and TTS side by side under static rules or counts; vLLM takes a fixed fraction; GPUStack reserves by estimate. Not found (lower confidence — absence of evidence): 6. **Raw per-call telemetry feeding scheduling.** Everyone exports metrics; nobody reads them back. 7. **Fleet placement from each box's MEASURED per-model footprints**, with tuned per-model settings pushed back out (LocalAI uses measured free VRAM but no per-model need; GPUStack estimates). 8. **Auth open on the LAN, key required from outside, proxy-aware.** All use a global on/off switch (or always on). ## What this means for hugpy hugpy-wrapper 0.1.51 (released today) already does, by design: 1 (measured facts, sql/012 doctrine), 2 (facts persist in its store and are read back), 3 (evict-to-fit plan), 4 (sizing / variants pick quant, ctx, layers, MoE split), 5 (one planner for llama.cpp, transformers, vision, ASR, TTS, image, video), 6 (call_log rows are the inputs of the next decision), 8 (FITEVICT_REQUIRE_KEY=outside, trusted proxies). Gap 7 is hugpy-core's job (pull /v1/telemetry, push settings) — not built yet. So the void is exactly the combination the wrapper is built around; the nearest neighbours each cover one slice: llama-swap (the front door shape), LocalAI (modality breadth + a fleet), GPUStack (fleet placement by estimate), llama.cpp `--fit` (single-model fitting). Worth checking or borrowing (follow-ups, not decisions): - **llama.cpp `--fit` is on by default** in llama-server. Checked 2026-10-08: the wrapper already passes `--fit off` on its launches when the binary supports the flag (fitevict is the planner), and uses `--fit on` only for its dry-run projections and the pinned max-GPU mode (adapters/llama_cpp.py, ~lines 973 and 1158–1181 in 0.1.51). No action needed. - **A Prometheus `/metrics` endpoint**: every neighbour has one; adding it costs little and lets existing dashboards (Grafana) read a box without hugpy-core. - **Packaging channels** neighbours use: a Docker image with engines bundled (llama-swap "unified"), Homebrew + WinGet (llama-swap), Helm (GPUStack) — relevant to the multi-OS plan. - **llama-swap #1004** shows users want memory-aware swapping without hand tuning — a natural audience.