HP Workstation Deployment (Part 2): Second Open WebUI Instance

Follow-up to llama.cpp + Open WebUI Setup. The workstation now runs two llama-server instances side by side, plus a second, fully isolated Open WebUI pointed at both — and later, two more models added on top.

Architecture

                                    ┌─────────┐
                                    │   You   │
                                    │(browser)│
                                    └────┬────┘
                       ┌──────────────────┴──────────────────┐
                       ▼                                       ▼
            ┌───────────────────────┐             ┌───────────────────────┐
            │  open-webui     :8080 │             │  open-webui-2   :8081 │
            └───────────┬────────────┘             └───────────┬────────────┘
                         │                                       │
     ┌───────────────────────────────────────────────────────────────────────────┐
     │  HP Workstation  (Quadro P4000, 8GB VRAM)                                  │
     │                                                                            │
     │   ┌─────────────┐  ┌──────────────────┐  ┌───────────────────┐  ┌───────────────────────┐
     │   │ phi-2  :8000│  │ tinyllama   :8001 │  │ gemma-3-4b-it:8002│  │ qwen2.5-3b-instruct:8003│
     │   └─────────────┘  └──────────────────┘  └───────────────────┘  └───────────────────────┘
     │                                                                            │
     └───────────────────────────────────────────────────────────────────────────┘

Both Open WebUI instances can reach all four llama-server backends (via OPENAI_API_BASE_URLS) — which model actually responds depends on which llama-inference*.service is running at the time, since VRAM only fits one or two of the four resident at once (see the VRAM footprint reference near the bottom of this page).

Current state (before adding more models)

ps -aux | grep llama

jay  814  /home/jay/llama.cpp/build/bin/llama-server -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --gpu-layers 15 --port 8001 --host 0.0.0.0
jay  815  /home/jay/llama.cpp/build/bin/llama-server -m phi-2.Q4_K_M.gguf                     --gpu-layers 30 --port 8000 --host 0.0.0.0

The existing open-webui container was left untouched (OLLAMA_BASE_URL=http://127.0.0.1:8000, host networking). Rather than reconfigure it, a second, isolated instance was added so the original setup stays stable.

Why a second container, not a reconfigured one

• --network=host on the original container means it can't be remapped to a different port without disrupting the running service.
• A dedicated container/volume keeps chat history, users, and settings separate between the two UIs.
• llama-server speaks the OpenAI-compatible API (/v1/chat/completions), not native Ollama — so the new instance is wired up with OPENAI_API_BASE_URLS instead of OLLAMA_BASE_URL.

Docker run (manual)

sudo docker run -d \
  --name open-webui-2 \
  -p 8081:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui-2:/app/backend/data \
  -e OPENAI_API_BASE_URLS="http://host.docker.internal:8000/v1;http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEYS="sk-not-needed;sk-not-needed" \
  -e ENABLE_OLLAMA_API=false \
  -e DEFAULT_LOCALE=en-US \
  ghcr.io/open-webui/open-webui:main

Key flags:

• No --network=host — isolates it from the existing instance and lets it bind cleanly to 8081.
• --add-host=host.docker.internal:host-gateway — since it's not on host networking, the container needs an explicit route back to the host's 127.0.0.1:8000 / :8001.
• OPENAI_API_BASE_URLS / OPENAI_API_KEYS — semicolon-separated lists, one entry per backend. llama-server doesn't check the API key, so any placeholder value works.
• -v open-webui-2:/app/backend/data — separate Docker volume from the original open-webui volume, so chat history/users don't mix.
• DEFAULT_LOCALE=en-US — Open WebUI otherwise auto-detects UI language from the browser's Accept-Language header; this pins it to English regardless of browser settings. (If it doesn't take effect for an already-created account, set it manually: Settings → General → Language → English.)

systemd service

sudo vi /etc/systemd/system/open-webui-2.service

[Unit]
Description=Open WebUI (secondary instance, port 8081)
After=network.target docker.service llama-inference.service llama-inference-2.service
Requires=docker.service

[Service]
Type=simple
User=jay
ExecStartPre=-/usr/bin/docker rm -f open-webui-2
ExecStart=/usr/bin/docker run \
  --name open-webui-2 \
  -p 8081:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui-2:/app/backend/data \
  -e OPENAI_API_BASE_URLS="http://host.docker.internal:8000/v1;http://host.docker.internal:8001/v1" \
  -e OPENAI_API_KEYS="sk-not-needed;sk-not-needed" \
  -e ENABLE_OLLAMA_API=false \
  -e DEFAULT_LOCALE=en-US \
  ghcr.io/open-webui/open-webui:main
ExecStop=/usr/bin/docker stop open-webui-2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

sudo docker rm -f open-webui-2
sudo systemctl daemon-reload
sudo systemctl enable open-webui-2.service
sudo systemctl start open-webui-2.service
sudo systemctl status open-webui-2.service

Verifying

1. Open http://<workstation-ip>:8081 — should load in English.
2. Log in / sign up.
3. Settings → Connections should show both OpenAI-compatible endpoints (:8000/v1, :8001/v1).
4. In a new chat, the model picker should list both phi-2 and TinyLlama 1.1B Chat.

Troubleshooting

• UI loads in the wrong language — DEFAULT_LOCALE only affects new signups on some Open WebUI versions; set the language manually under Settings → General for existing accounts.
• Models don't appear in the picker — confirm host.docker.internal resolves inside the container (docker exec -it open-webui-2 getent hosts host.docker.internal); if it doesn't, replace it with the workstation's LAN IP and restart the service.
• Port conflict on 8081 — check nothing else is bound there first: sudo ss -tlnp | grep 8081.

Mistakes made and lessons learned

1. Wrong flag: --c instead of -c

When bumping TinyLlama's context size, the systemd unit was first edited with --c 2048. llama-server doesn't recognize --c as a long option — only the short form -c (or the long form --ctx-size) is valid. A flag typo like this can silently fail to apply rather than erroring loudly.

Lesson: verify flags landed by checking the running process args in systemctl status (the CGroup: line), not just that the service says "active."

2. "Context size exceeded" wasn't a GPU/VRAM problem

Open WebUI threw a "Context size has been exceeded" error, and it was initially unclear whether this meant the GPU had run out of memory. nvidia-smi showed plenty of free VRAM, which ruled that out immediately.

Lesson: "context" in an LLM setup almost always means the model's token context window (the -c / --ctx-size flag), not GPU memory. Check nvidia-smi first to rule out VRAM, then check whether -c was ever explicitly set.

3. Setting -c larger than a model's trained context causes silent quality loss

phi-2 was restarted with -c 4096, but phi-2's native trained context is only 2048 tokens. llama-server logs a warning (GENERATION QUALITY WILL BE DEGRADED!), but the service still starts and looks healthy.

Lesson: a "successfully running" service isn't the same as a correctly configured one. Read the startup log lines, not just the Active: running status.

4. ChatML tokens (<|im_start|> / <|im_end|>) leaking into responses

phi-2 is a base/instruct model, not a chat-tuned model — it was never trained on ChatML formatting. llama-server was nonetheless applying a ChatML-style chat template, and because phi-2's tokenizer doesn't treat <|im_end|> as a real stop token, the model kept generating past its answer and hallucinated a fake next turn, printing the literal control tokens as visible text.

Lesson: the chat template used to format a model's prompt has to match how that model was actually trained. Fix by adding the missing stop sequence explicitly (--reverse-prompt "<|im_end|>" server-side, or a Stop Sequence in Open WebUI's per-model Advanced Params).

5. Picking models: chat-tuned beats base/completion models for a chat UI

Mistakes #3 and #4 trace back to the same root cause: phi-2 and TinyLlama are small/older models not designed for open-ended multi-turn chat. Moving to a properly instruct/chat-tuned current-generation model (e.g. Gemma 3 4B Instruct) avoids both problems by construction.

Lesson: when picking a model for a chat UI, prioritize "instruct/chat-tuned" variants over base/completion models of the same size.

6. Guessing model repo IDs (and trusting a fetched page) leads to broken/fake downloads

hf download google/gemma-3-4b-it-GGUF ... failed with "Repository not found" — Google doesn't publish a bare -GGUF repo under that name. Separately, a fetched page for a guessed URL (google/gemma-4-E4B-it) came back describing a plausible-sounding model — but the description was nearly identical to the real, different model Gemma 3n E4B, strongly suggesting a wrong URL or a hallucinated summary. There is no known "Gemma 4" family.

Lesson: never guess Hugging Face repo IDs and run hf download directly against the guess. Resolve the exact repo ID first, and treat any single fetched page describing a model as unverified until confirmed in a browser.

7. hf search doesn't exist — the real command is hf models list

Guessed at hf search <query> based on convention from other CLIs; the actual hf CLI has no search subcommand. The correct tool is hf models list (alias hf models ls).

Lesson: check <command> --help before running an invented command — CLI tools don't always follow the naming conventions you'd expect from a different tool.

How to search Hugging Face from the CLI

hf models list (alias hf models ls) is the real search command — there is no hf search.

# Search by keyword
hf models list --search "gemma-3-4b-it gguf" --sort downloads --limit 10

# Search restricted to a specific publisher
hf models list --search "gemma-3-4b-it" --author google --sort downloads

# Search restricted to models runnable by llama.cpp specifically
hf models list --search "gemma-3-4b" --apps llama.cpp --sort downloads --limit 10

# Search by parameter count range instead of a name
hf models list --search "instruct gguf" --num-parameters min:2B,max:5B --sort downloads

# Qwen models specifically
hf models list --search "qwen2.5 instruct gguf" --sort downloads --limit 10

# List the actual files/quant sizes inside a specific repo before downloading
hf models list unsloth/gemma-3-4b-it-GGUF
hf models list Qwen/Qwen2.5-7B-Instruct-GGUF

Always run the last form (hf models list <repo_id>) before hf download, to confirm the repo exists and see the exact filename/quant available.

Checking file sizes before downloading

hf download <repo_id> <filename> only downloads the one file you name, not the whole repo — but a GGUF repo typically contains every quantization level of the same model (often 20-30+ files), so it's important to list files first.

hf models list unsloth/gemma-3-4b-it-GGUF -h

      7.8 GB  gemma-3-4b-it-BF16.gguf
      2.4 GB  gemma-3-4b-it-IQ4_NL.gguf
      2.3 GB  gemma-3-4b-it-IQ4_XS.gguf
      1.7 GB  gemma-3-4b-it-Q2_K.gguf
      1.7 GB  gemma-3-4b-it-Q2_K_L.gguf
      2.1 GB  gemma-3-4b-it-Q3_K_M.gguf
      1.9 GB  gemma-3-4b-it-Q3_K_S.gguf
      2.4 GB  gemma-3-4b-it-Q4_0.gguf
      2.6 GB  gemma-3-4b-it-Q4_1.gguf
      2.5 GB  gemma-3-4b-it-Q4_K_M.gguf
      2.4 GB  gemma-3-4b-it-Q4_K_S.gguf
      2.8 GB  gemma-3-4b-it-Q5_K_M.gguf
      2.8 GB  gemma-3-4b-it-Q5_K_S.gguf
      3.2 GB  gemma-3-4b-it-Q6_K.gguf
      4.1 GB  gemma-3-4b-it-Q8_0.gguf
      1.2 GB  gemma-3-4b-it-UD-IQ1_M.gguf
      1.2 GB  gemma-3-4b-it-UD-IQ1_S.gguf
      1.6 GB  gemma-3-4b-it-UD-IQ2_M.gguf
      1.3 GB  gemma-3-4b-it-UD-IQ2_XXS.gguf
      1.7 GB  gemma-3-4b-it-UD-IQ3_XXS.gguf
      1.8 GB  gemma-3-4b-it-UD-Q2_K_XL.gguf
      2.2 GB  gemma-3-4b-it-UD-Q3_K_XL.gguf
      2.5 GB  gemma-3-4b-it-UD-Q4_K_XL.gguf
      2.8 GB  gemma-3-4b-it-UD-Q5_K_XL.gguf
      3.6 GB  gemma-3-4b-it-UD-Q6_K_XL.gguf
      5.2 GB  gemma-3-4b-it-UD-Q8_K_XL.gguf
    851.3 MB  mmproj-BF16.gguf
    851.3 MB  mmproj-F16.gguf
      1.7 GB  mmproj-F32.gguf

hf models list Qwen/Qwen2.5-3B-Instruct-GGUF -h

      4.0 GB  qwen2.5-3b-instruct-fp16-00001-of-00002.gguf
      2.8 GB  qwen2.5-3b-instruct-fp16-00002-of-00002.gguf
      1.4 GB  qwen2.5-3b-instruct-q2_k.gguf
      1.7 GB  qwen2.5-3b-instruct-q3_k_m.gguf
      2.0 GB  qwen2.5-3b-instruct-q4_0.gguf
      2.1 GB  qwen2.5-3b-instruct-q4_k_m.gguf
      2.4 GB  qwen2.5-3b-instruct-q5_0.gguf
      2.4 GB  qwen2.5-3b-instruct-q5_k_m.gguf
      2.8 GB  qwen2.5-3b-instruct-q6_k.gguf
      3.6 GB  qwen2.5-3b-instruct-q8_0.gguf

Key takeaway: the total size hf models list -h shows is for the entire repo, not a single download. hf download <repo> <one-filename> only fetches that one row. Don't run hf download <repo> without naming a file, or it pulls everything listed (in Gemma's case, ~68GB across every quant + BF16 + mmproj files).

What the quant suffixes mean, briefly:

• The number after Q is roughly bits-per-weight (Q2 ≈ 2-bit, Q8 ≈ 8-bit) — lower number = smaller file and faster, but more quality loss.
• _K_M / _K_S = "K-quant", a mixed-precision scheme (M = medium, S = small), generally better quality-per-byte than plain Q4_0/Q4_1.
• IQ* = "importance quantization", newer and even smaller for the same rough quality, at the cost of slightly slower inference.
• UD-* (Unsloth Dynamic) = per-layer adaptive quantization, tends to preserve quality better than a same-numbered standard quant.
• BF16/F16/F32/fp16 = unquantized or near-unquantized full-precision weights — much larger, not what you want for a VRAM-constrained box.
• mmproj-*.gguf = the separate vision projector file, only needed for image input (Gemma 3 is multimodal); not needed for text-only chat.

Downloading both models

cd ~/llama.cpp/models

hf download unsloth/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir ~/llama.cpp/models

hf download Qwen/Qwen2.5-3B-Instruct-GGUF qwen2.5-3b-instruct-q4_k_m.gguf --local-dir ~/llama.cpp/models

Output:

gemma-3-4b-it-Q4_K_M.gguf: downloading bytes: 100% 2.49GB, 95.0MB/s
gemma-3-4b-it-Q4_K_M.gguf: reconstructing file: 100% 2.49GB / 2.49GB, 168MB/s
✓ Downloaded
  path: /home/jay/llama.cpp/models/gemma-3-4b-it-Q4_K_M.gguf

qwen2.5-3b-instruct-q4_k_m.gguf: downloading bytes: 100% 2.09GB, 89.8MB/s
qwen2.5-3b-instruct-q4_k_m.gguf: reconstructing file: 100% 2.10GB / 2.10GB, 146MB/s
✓ Downloaded
  path: /home/jay/llama.cpp/models/qwen2.5-3b-instruct-q4_k_m.gguf

Correction to an earlier assumption: both downloads succeeded without hf auth login — unsloth/gemma-3-4b-it-GGUF turned out not to be gated (community re-uploads of Gemma weights are often ungated even though Google's own google/gemma-3-4b-it repo is), and Qwen was never gated. The "Warning: unauthenticated requests" line is just a rate-limit notice, not an auth failure.

Running both as new systemd services (ports 8002, 8003)

sudo vi /etc/systemd/system/llama-inference-3.service

[Unit]
Description=Llama Inference Server (Gemma 3 4B Instruct)
After=network.target

[Service]
Type=simple
User=jay
ExecStart=/home/jay/llama.cpp/build/bin/llama-server \
  -m /home/jay/llama.cpp/models/gemma-3-4b-it-Q4_K_M.gguf \
  --alias gemma-3-4b-it \
  --gpu-layers 99 \
  -c 4096 \
  --port 8002 \
  --host 0.0.0.0
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo vi /etc/systemd/system/llama-inference-4.service

[Unit]
Description=Llama Inference Server (Qwen2.5 3B Instruct)
After=network.target

[Service]
Type=simple
User=jay
ExecStart=/home/jay/llama.cpp/build/bin/llama-server \
  -m /home/jay/llama.cpp/models/qwen2.5-3b-instruct-q4_k_m.gguf \
  --alias qwen2.5-3b-instruct \
  --gpu-layers 99 \
  -c 4096 \
  --port 8003 \
  --host 0.0.0.0
Restart=on-failure

[Install]
WantedBy=multi-user.target

Notes on the flags:

• --alias — sets a short, friendly model name reported via /v1/models. Without it, llama-server reports the full filesystem path (e.g. /home/jay/llama.cpp/models/phi-2.Q4_K_M.gguf) as the model ID, which is what Open WebUI then shows in the picker — ugly, but purely cosmetic, and fixed entirely server-side with this one flag.
• --gpu-layers 99 requests full GPU offload (more than either model actually has, which is fine — llama.cpp just offloads all of it). Watch nvidia-smi after starting each one; if VRAM runs out, lower this to partially offload instead of failing outright.
• -c 4096 is well within both models' native trained context (Gemma 3 4B: 128K, Qwen2.5 3B: 32K) — no quality-degradation warning like the phi-2 mistake.
• No --chat-template override — both are properly instruct-tuned models with correct template metadata baked into the GGUF, so the ChatML-leak bug from phi-2 shouldn't recur here.

Starting on demand instead of always-on: VRAM is tight — the workstation has an 8GB Quadro P4000, phi-2 + TinyLlama already use ~2.9GB, and Gemma 3 4B alone pushed usage to 6711MiB/8116MiB once loaded (its real footprint with KV cache is closer to 3.7GB, not just its 2.5GB file size). Qwen2.5 3B doesn't fit alongside everything else at once — don't enable all four services at boot. Load and start only what you're actively testing:

sudo systemctl daemon-reload

# Start whichever one you want to try — not both new ones at once
sudo systemctl start llama-inference-3.service   # Gemma 3 4B
sudo systemctl status llama-inference-3.service
nvidia-smi

# Free VRAM before starting another, e.g. stop TinyLlama (the weakest model) to make room
sudo systemctl stop llama-inference-2.service
sudo systemctl start llama-inference-4.service   # Qwen2.5 3B

If VRAM checks out fine with more than two running at once, sudo systemctl enable llama-inference-3.service (and/or -4) later to make it start at boot like the first pair.

Wiring new models into Open WebUI

Two ways to add the new :8002 / :8003 endpoints to open-webui-2:

Option A: quick, via the Admin UI (no restart needed)

1. Open http://<workstation-ip>:8081, log in as admin.
2. Profile icon (top-right) → Admin Panel → Settings → Connections.
3. Under OpenAI API, click + Add Connection and add one entry per new model: URL http://host.docker.internal:8002/v1 and http://host.docker.internal:8003/v1, Key sk-not-needed for both.
4. Save, then start a new chat — the model picker should list all connected models (only the currently-running llama-server will actually respond).

This is stored in the open-webui-2 Docker volume, so it survives container restarts — just not a full volume wipe.

Option B: persistent, baked into the systemd unit

Update the two -e lines in open-webui-2.service to list all four ports, semicolon-separated, then recreate the container. This is more failure-prone to hand-edit (see the mistake below) — Option A is the safer default.

Mistake: editing the systemd unit broke the container with "invalid reference format"

Editing ExecStart to add the two extra endpoints resulted in:

docker: invalid reference format
Run 'docker run --help' for more information
open-webui-2.service: Main process exited, code=exited, status=125/n/a

"invalid reference format" means the image name argument to docker run got mangled — almost certainly a missing or stray backslash line-continuation left over from a manual edit, so systemd fed docker run a broken argument list where the image reference merged with a neighboring flag.

Lesson: when a multi-line ExecStart with backslash continuations breaks, don't patch individual lines — replace the whole block from a known-good copy. Every continued line must end with \ and nothing after it (no trailing spaces), which is easy to break with in-place `vi` edits. Fix was to paste the complete file back in cleanly rather than hunting for the exact broken character.

VRAM footprint reference

Loading a 2.5GB weight file added ~3.7GB of actual VRAM usage once KV cache for a 4096-token context was included — a useful reminder for planning capacity that the GGUF file size on disk is not the same as the runtime VRAM footprint.