2026-07-27

Adding a Hermes Agent Alongside Open WebUI

We already run several llama-server instances on the jay-z820 workstation (Quadro P4000, 8GB VRAM) fronted by Open WebUI, plus an mcpo bridge that exposes MCP tools (like get_current_time) as an OpenAPI Tool Server. This adds NousResearch's Hermes, a model trained specifically for structured function/tool calling, so students can watch a local model actually decide to call a tool instead of just chatting.

Every command below is logged in the order it was actually run, including the wrong turns.

1. Recon: is there room for another model?

nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv
systemctl list-units --type=service --state=running | grep -iE 'llama|webui|mcpo'
ss -tlnp | grep -E ':(8000|8001|8002|8081|8050)'
df -h /home
ls /etc/systemd/system/ | grep -iE 'llama|webui|mcpo'

Result: the P4000 had only 1.35GB VRAM free out of 8GB — Gemma 3 4B, Qwen2.5 3B, and TinyLlama were already loaded. A phi-2 unit exists but was inactive.

Gotcha: don't assume "there's a GPU, so use it" — check nvidia-smi free memory first. An 8B model at Q4 (~4.9GB) would not have fit next to the existing three models.

free -h
nproc
sudo ss -tlnp | grep -E ':(8000|8080|8081|8082)'
cat /etc/systemd/system/open-webui.service
cat /etc/systemd/system/open-webui-2.service

46GB RAM (31GB free), 16 CPU cores — plenty of headroom to run a small model on CPU instead of fighting for the last sliver of VRAM. Two Open WebUI containers exist: open-webui (port 8000/8080, --network=host) and open-webui-2 (port 8081, reachable via host.docker.internal). Ports in use: 8000/8080 open-webui, 8001 TinyLlama, 8002 Gemma 3, 8003 Qwen2.5, 8050 mcpo, 8081 open-webui-2, 8101 phi-2 (disabled). 8004 free — used for Hermes.

2. Picking a model that actually fits

8B Hermes (the usual recommendation) was out given the VRAM situation. Checked Hugging Face for smaller official quantized releases rather than guessing a repo name:

curl -s "https://huggingface.co/api/models?search=Hermes-3-Llama-3.2-3B&limit=20"
curl -s "https://huggingface.co/api/models/NousResearch/Hermes-3-Llama-3.2-3B-GGUF"

Picked NousResearch/Hermes-3-Llama-3.2-3B-GGUF (the official quant repo), quant Q4_K_M (~2GB).

Decision:even at 2GB, weights + KV cache still wouldn't cleanly fit into 1.35GB free VRAM without a live OOM risk mid-class. Ran it CPU-only (--gpu-layers 0) at first instead of fighting the P4000 for scraps.

3. First systemd unit (CPU-only)

python3 -c "
from huggingface_hub import hf_hub_download
p = hf_hub_download(repo_id='NousResearch/Hermes-3-Llama-3.2-3B-GGUF', filename='Hermes-3-Llama-3.2-3B.Q4_K_M.gguf', local_dir='.')
print('DOWNLOADED:', p)
"

Gotcha: hf and huggingface-cli weren't on PATH (Python 3.10.12, huggingface_hub installed as a library only) and python3 -m huggingface_hubisn't runnable as a module. Used the Python API (hf_hub_download) directly instead.

[Unit]
Description=Llama Inference Server (Hermes-3-Llama-3.2-3B, agentic tool-calling)
After=network.target

[Service]
Type=simple
User=jay
WorkingDirectory=/home/jay/llama.cpp/build
ExecStart=/home/jay/llama.cpp/build/bin/llama-server \
  -m /home/jay/llama.cpp/models/Hermes-3-Llama-3.2-3B.Q4_K_M.gguf \
  --gpu-layers 0 \
  -c 8192 \
  --port 8004 \
  --host 0.0.0.0
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now llama-inference-hermes.service
curl -s http://localhost:8004/v1/models

Gotcha: sudo over a non-interactive SSH session refuses a TTY password prompt. Fixed by writing the password to a throwaway file (/tmp/.sp, deleted right after) and using sudo -S ... < /tmp/.sp for each privileged command.

4. First tool-calling test — and it silently failed

curl -s http://localhost:8004/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "hermes",
  "messages": [{"role": "user", "content": "What time is it right now?"}],
  "tools": [{"type": "function", "function": {"name": "get_current_time", "description": "Get the current time in a given timezone", "parameters": {"type": "object", "properties": {"timezone": {"type": "string"}}, "required": ["timezone"]}}}]
}'

It just made up a time("11:14 AM on August 3, 2023") instead of calling the tool — even with "tool_choice": "required", it narrated fake reasoning instead of emitting a real tool call.

Root cause: curl -s http://localhost:8004/props showed "chat_format": "Content-only". The GGUF's baked-in chat template (and even the base NousResearch/Hermes-3-Llama-3.2-3B repo's tokenizer_config.json) is bare ChatML — no {% if tools %}Jinja logic at all. llama-server can't render tool definitions into the prompt without a template that knows about them, so it silently degrades to plain chat. The tools param was a no-op the whole time, hence the confident hallucination.

Fix: Hermes models ship a second, separate chat-template variant named tool_use (visible in NousResearch/Hermes-2-Pro-Llama-3-8B's tokenizer_config.json, as a list of {name, template} objects) that adds the <tools> system-prompt block and <tool_call>/<tool_response> XML tags. Pulled it out and passed it explicitly via --chat-template-file.

5. VRAM freed up mid-session — switched to GPU offload

Partway through, two other llama-server instances were stopped by hand, freeing VRAM to 5GB. Updated the unit to add --chat-template-file and switch --gpu-layers 0 → 99 (full offload). Hermes ended up using ~2.8GB VRAM fully offloaded, leaving ~2.3GB free — not enough headroom for the 8B Hermes variant, confirming the earlier decision to go with the 3B.

6. Tool-calling worked — but only at temperature 0

With the tool_use template in place, the default sampling settings (temp 0.8) still produced a garbled attempt:

{"content": "<SCRATCHPAD>\n{\"name\": \"get_current_time\", \"arguments\": {\"timezone\": \"Denver\"}}\n</tool_response>"}

Right idea, wrong tags (<SCRATCHPAD> instead of <tool_call>, mismatched closing tag) — a small 3B model wandering off the exact format under sampling noise. Setting "temperature": 0 produced a clean, correctly parsed tool_calls response every time. Baked --temp 0into the systemd unit itself so Open WebUI's requests (which won't set it explicitly) still get reliable tool calls by default.

Lesson for the class: tool-calling reliability on small local models is sensitive to both the chat template andsampling temperature — a good live example of why "it's not calling my tool" is usually a prompt/template problem, not a fundamentally broken model.

7. Registering Hermes in Open WebUI — wrong instance first

docker ps -a showed open-webui-2 (the :8081 instance used in earlier docs) had actually exited 7 days ago. The container currently live is the original open-webui service, on port 8080 (--network=host) — a separate Docker volume with its own independent config/database from open-webui-2.

Gotcha: these are two totally separate Open WebUI installs (separate SQLite DBs inside separate named volumes) even though both run the same image. A connection or tool server registered in one does not carry over to the other. Used :8080since it's the one actually running.

Logged in via the API to get an admin token, read the existing OpenAI-compatible connections (GET /openai/config) so the new one could be appended without clobbering the Gemma/Qwen/TinyLlama connections already there, then added Hermes as a 4th connection pointing at http://127.0.0.1:8004/v1.

8. The MCP tool server wasn't registered on this instance either

Per the earlier MCP & Tools doc, MCP tools (via the mcpo-timebridge) aren't attached to individual models — they're registered once, globally, as an External Tool Server, then toggled on per-chat via the wrench icon.

curl -s "http://localhost:8080/api/v1/configs/tool_servers" -H "Authorization: Bearer $TOKEN"
# {"TOOL_SERVER_CONNECTIONS": []}

Empty — same reason as the connections above, this fresh :8080 instance never had mcpo-time registered. Registered it (localhost:8050 works directly here, no host.docker.internal needed, since open-webui runs with --network=host).

9. Custom model for a clean display name

Same pattern as the existing qwen-model custom model — wraps the raw connection model with a friendly name, description, and suggested prompts via POST /api/v1/models/create. Confirmed Hermes Agent (tool-calling demo) now shows up in the model list.

10. What's left — do this live in the browser

Everything above was verified at the API layer (direct llama-server calls, direct mcpocalls, Open WebUI's admin/config API). The chat UI flow itself needs a real click-through before class:

Open http://192.168.1.92:8080, log in, start a new chat, pick Hermes Agent (tool-calling demo).
Click the wrench/tools icon, enable the mcpo tool server if it's not already on.
Ask "What time is it right now in Tokyo?" and confirm a tool call happens (not a hallucinated answer).
If it doesn't fire: log out/in (or hard-refresh) so Open WebUI re-fetches the external tool server's openapi.json.

Summary for the class

Why Hermes, not another local model:it's trained specifically to emit structured <tool_call>output when given tool definitions — a good hook for teaching "what makes a model agentic" vs. just chatty.
Why the 3B, not 8B:VRAM was the hard constraint — 8B Q4 (~4.9GB) doesn't fit safely alongside the other models running.
Two real bugs, both good teaching moments: the default GGUF chat template has no tool-calling logic at all, and tool-call formatting was unreliable above temperature 0 on this small model.
Infra gotcha, not a Hermes thing: open-webui and open-webui-2 are two independent installs with separate databases — confirm which instance is actually running (docker ps) before assuming prior setup carried over.