gpu-coordinator-proxy
Single-process aiohttp proxy that lets LMStudio and Ollama share the
same GPU without stepping on each other.
Both runtimes are happy to load a model into VRAM but neither knows the other
exists. On a single-GPU box this means a request to runtime A while runtime B
still holds VRAM either OOMs or silently falls back to CPU and times out. This
proxy fronts both runtimes on separate ports and serializes their model loads
through a shared lock.
What it does
Before forwarding any chat completion / embed / generate request:
- Acquires a single global VRAM lock.
- Unloads every model loaded in the other runtime.
- Optionally unloads non-target models in the same runtime.
- Sleeps
GPU_FREE_SETTLE_MS so the driver actually reclaims VRAM.
- Forwards the original request — JIT-loads the target as needed.
Listeners
| Port |
Fronts |
Coordinated paths |
| 1235 |
LMStudio |
POST /v1/chat/completions |
| 11435 |
Ollama |
POST /v1/chat/completions, POST /api/chat, POST /api/generate, POST /api/embed, POST /api/embeddings |
Everything else (model lists, /api/tags, /api/ps, /api/show, /healthz)
is a verbatim pass-through.
Run
docker build -t gpu-coordinator-proxy .
docker run --rm -p 1235:1235 -p 11435:11435 \
-e LMSTUDIO_UPSTREAM=http://host.docker.internal:1234 \
-e OLLAMA_UPSTREAM=http://host.docker.internal:11434 \
--add-host host.docker.internal:host-gateway \
gpu-coordinator-proxy
Then point your client at the proxy ports instead of the real ones. From an
OpenAI-compatible app:
OPENAI_BASE_URL=http://localhost:1235/v1 # LMStudio
OPENAI_BASE_URL=http://localhost:11435/v1 # Ollama
Configuration
All knobs are env vars; defaults are the safest sane choice.
| Variable |
Default |
Purpose |
LMSTUDIO_UPSTREAM |
http://127.0.0.1:1234 |
Real LMStudio server. |
OLLAMA_UPSTREAM |
http://127.0.0.1:11434 |
Real Ollama server. |
LMSTUDIO_PROXY_LISTEN_PORT |
1235 |
Proxy port for LMStudio. |
OLLAMA_PROXY_LISTEN_PORT |
11435 |
Proxy port for Ollama. |
GPU_AUTO_FREE_ENABLED |
1 |
Master switch. 0 = pure pass-through, no eviction. |
GPU_FREE_STRATEGY |
spare-target |
spare-target keeps the requested model warm; wipe-all evicts the target too. |
GPU_WIPE_ALL_ALLOWED |
0 |
Safety gate: requires explicit opt-in for wipe-all. Otherwise downgraded to spare-target with a warning. |
GPU_FREE_SETTLE_MS |
800 |
Pause after eviction before forwarding (driver reclaim time). |
REQUEST_TIMEOUT_SECONDS |
600 |
Upstream request timeout. |
GET /healthz returns the live values:
{
"ok": true,
"lmstudio_upstream": "http://host.docker.internal:1234",
"ollama_upstream": "http://host.docker.internal:11434",
"auto_free_enabled": true,
"free_strategy": "spare-target",
"wipe_all_allowed": false,
"free_settle_ms": 800
}
When to use which strategy
- spare-target (default) — your usual setup. The requested model stays in
VRAM across requests; only collateral models swap out. Cheapest.
- wipe-all — emergency mode. Reload every model from disk on every request
(+5–15 s/call). Pulls you out of stuck VRAM states (driver leaks, fragmented
allocations after long sessions). Gated by
GPU_WIPE_ALL_ALLOWED=1 so it
isn't accidentally enabled.
- disabled (
GPU_AUTO_FREE_ENABLED=0) — debugging. Skip eviction
entirely; useful when you're trying to reproduce a different bug and don't
want the proxy interfering.
Why eviction has to be cooperative
Neither runtime sees the other's VRAM use, but both share the same physical
GPU. A nvidia-smi snapshot during a swap shows VRAM stuck at the larger
of the two model sizes — they never coexist. Without coordination:
- LMStudio holds 13 GB. Ollama receives a chat request, tries to load its
own 13 GB model, sees the GPU is "free" from its perspective, fails to fit,
silently falls back to CPU. The request crawls and the client hits its
timeout.
- Reverse direction symmetrical.
The only fix is for one party to actually unload before the other loads.
Inside one runtime that's already implicit (LMStudio's JIT auto-evict, Ollama's
OLLAMA_MAX_LOADED_MODELS); across runtimes, somebody has to call out.
That's this proxy.
License
MIT.