Self-hosted model servers: how they place models, and where nobody has solved it #

Researched 2026-10-08 (claude session e053c857, via a web-research agent). Dates/versions are what was seen that day. Some details came from search summaries or third-party mirrors where the repo could not be opened — marked where it matters. Frozen: a re-check is a new dated file.

Question (operator): "where has this problem reached a solution void?" — the problem being one OpenAI-compatible front door per box that decides, per call, what to load, what to evict and how to make a model fit, across engine types, and learns from what loads really cost.

The projects #

LocalAI (mudler/LocalAI) #

• Go server, backends as separate processes; Docker or binary. ~v4.2/4.3 (May 2026, per mirrored changelogs https://gitea.varghacsongor.hu/GithubMirror/LocalAI); 3.10 (Jan 2026) added video + image.
• Broadest modalities: chat, embeddings, ASR (whisper.cpp, faster-whisper), TTS, image + video (diffusers, LTX-2), realtime, diarization, face recognition; ~36 backends.
• Load/unload: --max-active-backends (LRU count), idle/busy watchdogs, concurrency groups, pinned, --vram-budget (a ceiling). Busy models are not evicted: 30 retries at 1 s, then "proceed (which may cause out-of-memory errors)". Docs: it "cannot reliably estimate VRAM use for new models across backends"; the gallery shows a metadata estimate. https://localai.io/docs/advanced/vram-management/ · https://localai.io/docs/features/runtime-settings/
• Auto-fit: none documented beyond passthrough (e.g. gpu_layers).
• Cross-engine: one LRU count + one VRAM ceiling across all backends; no per-model accounting.
• Fleet: distributed mode (PostgreSQL + NATS); workers send MEASURED free VRAM each heartbeat; scheduler: node with the model → node with enough free VRAM → idle → least-loaded → LRU eviction. The model's need is not estimated (disk size + 5% check only). No history. https://localai.io/docs/features/distributed-mode/
• Telemetry: /metrics, OpenTelemetry, per-user usage; only free VRAM feeds placement.
• Auth: open when unconfigured; legacy keys = full admin; user/OAuth system (required in distributed mode). https://localai.io/docs/features/authentication/

llama-swap (mostlygeek/llama-swap) — the closest design to ours #

• Go proxy, one YAML config; binary / Homebrew / WinGet / a "unified" Docker image with llama-server, stable-diffusion.cpp, whisper.cpp, audio.cpp. Very active (issues closed 2026-10-07). https://github.com/mostlygeek/llama-swap
• Modalities: chat, completions, responses, Anthropic messages, embeddings/rerank, ASR, TTS, image generation/edit (video for sd.cpp requested, #976). Any engine = a configured command.
• Load/unload: purely static, no memory model. Groups (swap, exclusive, persistent), a "matrix" DSL of allowed combinations ((a|b) & c) with per-model evict_costs (picks the cheapest allowed set), ttl/globalTTL. VRAM appears only as comments and CUDA_VISIBLE_DEVICES. https://raw.githubusercontent.com/mostlygeek/llama-swap/main/docs/config.example.yaml
• Auto-fit: none (leans on the engine, e.g. llama.cpp --fit).
• Cross-engine: yes, but by hand-written rules.
• Fleet: peers = static forwarding to remote servers (peer/model).
• Telemetry: Prometheus /metrics (GPU/system every 5 s); per-request token log to SQLite (store.path); request capture. Nothing feeds scheduling.
• Auth: apiKeys list, empty = no auth.
• Users ask for what we built: #1004 (open, Aug 2026) "memory-constrained scheduling with per-model RAM declarations" — and explicitly "Not asking for automatic detection (that's hard)". https://github.com/mostlygeek/llama-swap/issues/1004 · also #1127, #749, #970.

Ollama #

• Go server, own engine (moving off llama.cpp); native or Docker; ~v0.31.x (Jul 2026).
• Modalities: chat, vision, embeddings; experimental image generation since Jan 2026 (https://registry.ollama.ai/blog/image-generation). No ASR/TTS found.
• Load/unload: keep_alive (5 min), OLLAMA_MAX_LOADED_MODELS (3 per GPU), queue + unload idle when a model does not fit (https://docs.ollama.com/faq). Sept 2025 "new model scheduling": the new engine "will now measure the exact amount of memory required" — a pre-load measurement for new-engine models; nothing indicates it is kept. https://ollama.com/blog/new-model-scheduling
• Auto-fit: chooses GPU layers; default context scales with VRAM (4k / 32k / 256k) (https://docs.ollama.com/context-length). No automatic quant.
• Fleet: multi-GPU on one host only.
• Telemetry: logs, ollama ps; no native Prometheus found. Auth: none locally.
• Open problems: #17099 vision memory estimate inflated (Jul 2026), #16617 CPU offload despite VRAM, #14632 two models on the same GPUs while another was free → CUDA OOM (https://github.com/ollama/ollama/issues/14632), #12094 crash instead of CPU fallback. Independent test of v0.12.1 found more CPU offload than claimed: https://glukhov.org/llm-performance/ollama/memory-allocation-in-ollama-new-version/

LM Studio (closed source) #

• Desktop app + llmster headless daemon. Chat, vision, embeddings.
• JIT load on first request, 60-min idle TTL, auto-evict keeps one JIT model; "loading guardrails" refuse by estimate (lms load --estimate-only); the guardrail can shrink context. https://lmstudio.ai/docs/developer/core/ttl-and-auto-evict
• Estimates can be far off: a headless load refused at ~22.9 GB estimated vs ~11.5 GB actual (e.g. https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2051).
• Optional API token, off by default.

GPUStack (gpustack/gpustack) — the fleet one #

• Python server + workers, UI; pip / Docker / Helm; v2.2.x (Jun–Jul 2026; v2.2.2 fixed missing authorization on cluster endpoints).
• LLM, embeddings, rerank, image, audio (VoxBox); vLLM, SGLang, MindIE, llama-box/GGUF.
• Deploy-time scheduler, not per-call swapping: filter workers → ESTIMATE memory (gguf-parser / backend formulas) → single GPU → multi-GPU → distributed → partial offload / CPU → binpack/spread. https://docs.gpustack.ai/latest/scheduler/ . No learning.
• Heterogeneous fleet (NVIDIA, AMD, Ascend, Apple) is its strength. Prometheus + Grafana built in. Auth mandatory by default.
• Its estimate bugs are the clearest evidence of the gap: #1601, #2101, #2412, #2443, #1899 (open), #4153, #5375 (open); #5880 (open, v2.4.0) asks only to SHOW actual usage next to allocated. https://github.com/gpustack/gpustack/issues/5880

Xinference (xorbitsai/inference) #

• Python supervisor + workers; v3.5.0 docs. LLM, embeddings, rerank, image, audio, video.
• Explicit launch/terminate; "Idle First" GPU choice; v3.5 recommends engine/format/size/quant from offline metadata assuming 1 sequence × 2048 tokens × FP16 KV against 80 % of free memory — "does not guarantee sufficient memory"; not for image/video. https://inference.readthedocs.io/en/latest/user_guide/launch.html
• Prometheus on by default; auth experimental and off by default.

vLLM #

• One model per process; claims gpu_memory_utilization (0.9) of the card. "Sleep mode" offloads or drops weights for switching (https://docs.vllm.ai/en/latest/features/sleep_mode.html). No auto-fit (errors if KV does not fit). Rich Prometheus. --api-key off by default.
• Swappers built on it: llmux (https://docs.rs/crate/llmux/0.7.6), Snowflake's "Semi-Persistence" (https://www.snowflake.com/en/blog/engineering/semi-persistence-gpu-model-swapping/).

llama.cpp itself #

• Router mode: --models-dir/--models-preset, --models-max (default 4, a count + LRU, not VRAM), autoload, --sleep-idle-seconds; one child per model.
• --fit (ON by default): the README's own words: "whether to adjust unset arguments to fit in device memory"; margin --fit-target 1024 MiB per device; --fit-ctx "minimum ctx size that can be set by --fit option" 4096 — so it shrinks context (only when -c is not given). The README does NOT list the other arguments it adjusts; the agent's claim that it also sets layers / tensor split / MoE experts is UNVERIFIED (hugpy-wrapper's own max-gpu path relies on it picking the layer split). Operator's experience (2026-10-08): poor results in practice. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md (checked 2026-10-08) One model vs free memory: no eviction planning, no quant choice, no history.
• --api-key off by default; --metrics per model in router mode.

Others #

KoboldCpp (single model; --gpulayers -1 guesses layers, "experimental"); Paddler (llama.cpp load balancer by free slots, fleet UI, StatsD; model management only a future plan — https://github.com/intentee/paddler); LiteLLM proxy (routing/keys/budgets, never loads models); Text Generation WebUI (one model, manual).

Where no project has a solution #

Documented absence or open requests (higher confidence):

1. Measured per-model footprint feeding placement. LocalAI says it cannot estimate across backends; GPUStack's estimate-bug trail (#1601 … #5880); llama-swap #1004 asks for hand-declared sizes and rules out automatic detection; Ollama measures pre-load for its new engine only and still ships estimate regressions.
2. Learning from past loads. Nobody documents keeping what a load really used (VRAM after load, CUDA context, growth during generation) and reusing it next time.
3. Planning WHICH residents to evict so a model fits. Everyone uses count + LRU, hand-written sets, or a fixed card fraction; LocalAI documents proceeding into a possible OOM.
4. Choosing quant + context + layers + MoE offload together, with eviction. llama.cpp --fit (documented: unset args incl. context, min 4096) and LM Studio's guardrail fit ONE model against free memory; Xinference recommends quant from a fixed assumption (not image/video). None combines it with eviction or measured history.
5. One memory planner across engine types on one card. llama-swap/LocalAI run llama.cpp, diffusers, whisper and TTS side by side under static rules or counts; vLLM takes a fixed fraction; GPUStack reserves by estimate.

Not found (lower confidence — absence of evidence): 6. Raw per-call telemetry feeding scheduling. Everyone exports metrics; nobody reads them back. 7. Fleet placement from each box's MEASURED per-model footprints, with tuned per-model settings pushed back out (LocalAI uses measured free VRAM but no per-model need; GPUStack estimates). 8. Auth open on the LAN, key required from outside, proxy-aware. All use a global on/off switch (or always on).

What this means for hugpy #

hugpy-wrapper 0.1.51 (released today) already does, by design: 1 (measured facts, sql/012 doctrine), 2 (facts persist in its store and are read back), 3 (evict-to-fit plan), 4 (sizing / variants pick quant, ctx, layers, MoE split), 5 (one planner for llama.cpp, transformers, vision, ASR, TTS, image, video), 6 (call_log rows are the inputs of the next decision), 8 (FITEVICT_REQUIRE_KEY=outside, trusted proxies). Gap 7 is hugpy-core's job (pull /v1/telemetry, push settings) — not built yet.

So the void is exactly the combination the wrapper is built around; the nearest neighbours each cover one slice: llama-swap (the front door shape), LocalAI (modality breadth + a fleet), GPUStack (fleet placement by estimate), llama.cpp --fit (single-model fitting).

Worth checking or borrowing (follow-ups, not decisions):

• llama.cpp --fit is on by default in llama-server. Checked 2026-10-08: the wrapper already passes --fit off on its launches when the binary supports the flag (fitevict is the planner), and uses --fit on only for its dry-run projections and the pinned max-GPU mode (adapters/llama_cpp.py, ~lines 973 and 1158–1181 in 0.1.51). No action needed.
• A Prometheus /metrics endpoint: every neighbour has one; adding it costs little and lets existing dashboards (Grafana) read a box without hugpy-core.
• Packaging channels neighbours use: a Docker image with engines bundled (llama-swap "unified"), Homebrew + WinGet (llama-swap), Helm (GPUStack) — relevant to the multi-OS plan.
• llama-swap #1004 shows users want memory-aware swapping without hand tuning — a natural audience.