2026-08-05

5 Simple Tricks for Running Hermes Agent Against a Local LLM

Hermes Agent(Nous Research's CLI coding/agent assistant) doesn't need an API key from a hosted provider — it talks to any OpenAI-compatible endpoint, which means it works against a local llama-server instance the same way it would against a cloud model. This doc distills five practical habits for pointing it at a local LLM effectively, pulled from what actually happened running it against real local models on the jay-z820 workstation in two earlier live sessions: the original Hermes-3B setup and the DeepSeek V4 Flash demo run. Rather than re-running the same commands again, this doc is the "here's what we learned, here's how to actually use it well" writeup — every claim below traces back to one of those two sessions.

1. Point Hermes at your local server as a "custom" provider

hermes configaccepts any base URL, not just OpenAI's or Anthropic's — a local llama-server exposing an OpenAI-compatible /v1 route is a first-class citizen. This is the one-time setup step everything else in this doc depends on:

$ hermes config
◆ Model
  Model:        {'default': 'unsloth/DeepSeek-V4-Flash-GGUF:UD-IQ3_XXS',
                  'provider': 'custom',
                  'base_url': 'http://192.168.1.91:11434/v1',
                  'api_key': '***'}

The API key can be anything non-empty — most local servers don't check it, they just require the field to be present because the OpenAI client library won't send a request without one.

2. Use one-shot mode for anything scripted or demoed live

hermes -z "prompt" --cliruns a single prompt to completion and exits, instead of dropping into the interactive TUI. For local models — which are often much slower than a hosted API — this matters more than it sounds: it's the difference between a clean, scriptable command you can time and pipe, and a session you have to babysit.

$ hermes -z "In one sentence, who are you and what model are you running on?" --cli

Set speed expectations before you demo this live. A 284B-parameter model (DeepSeek V4 Flash) running IQ3-quantized on a single GPU took ~2 minutesto answer a one-sentence prompt. Local inference on a big quantized model is a "kick it off and talk while it thinks" demo, not a snappy live-typing one.

3. Use --yolo to skip confirmation prompts — but only when you trust the sandbox

By default Hermes Agent pauses for approval before running tool calls (file writes, code execution, web requests). --yolo auto-approves them, which is what makes one-shot demos like web search or file read/write actually run end-to-end without a human in the loop:

$ hermes -z "Create a file at /tmp/hermes-demo.txt containing some text, then read it back and show me the contents." --cli --yolo

Only use this against infrastructure you control. It auto-approves every tool call the model requests, including ones it hallucinates a need for. Fine for a local box you own; not something to wire into anything touching production data or a shared environment.

4. Check the chat template before trusting any tool-calling result

This is the single highest-leverage trick in this list. Both prior sessions independently hit the same root causefor MCP tools, skills, memory, and subagent delegation all silently failing: the local GGUF's baked-in chat template had no Jinja logic for rendering tools into the prompt at all. One curl command tells you in advance whether tool calling has any chance of working on a given local server:

$ curl http://192.168.1.91:11434/props | grep chat_format
"chat_format": "Content-only"

"Content-only" means tool calling is a no-op— the model is never even shown the tool definitions, so it will confidently say "I don't have that tool" (an honest answer) or, worse, narrate using a tool and claim success on an action that never happened (a hallucinated one — this is exactly what happened when Hermes Agent was asked to save a memory that was never written to disk). The fix is starting llama-server with an explicit --chat-template-file that actually renders tool definitions — the same move that fixed tool calling for the Hermes-3B setup.

5. Register MCP servers with absolute paths, not bare command names

Once the chat template is fixed and tool calling actually works, adding an MCP server is one command:

$ hermes mcp add mcp-time --command uvx --args mcp-server-time

Gotcha:Hermes Agent's MCP subprocess launcher doesn't inherit the shell's PATH. A bare command name that works fine typed at the terminal fails with FileNotFoundError: [Errno 2] No such file or directory when Hermes tries to launch it. Always use the absolute path:

$ hermes mcp add mcp-time --command /home/jay/.local/bin/uvx --args --with 'mcp<1.10' mcp-server-time
✓ Connected! Found 2 tool(s) from 'mcp-time':
  get_current_time     Get current time in a specific timezones
  convert_time          Convert time between timezones

Two more free tips that fall out of the same command: pin dependency versions with --with 'mcp<1.10' if a reference MCP server predates the latest SDK, and put any flag meant for hermes mcp add itself (like --connect-timeout) before --command/--args — --args swallows every token after it, so anything placed later gets passed to the MCP server binary instead of to Hermes.

Bonus: hermes doctor, before you touch the LLM at all

A good first (and last-resort) move when something isn't working: a fast, free health check that verifies config files and dependencies without spending a single token or waiting on inference.

$ hermes doctor
◆ Configuration Files
  ✓ ~/.hermes/.env file exists
  ✓ API key or custom endpoint configured
  ✓ ~/.hermes/config.yaml exists
  ✓ Config version up to date (v33)
◆ Required Packages
  ✓ OpenAI SDK
  ✓ Rich (terminal UI)
  ✓ Croniter (cron expressions) (optional)

Summary for the class

Tricks 1–3 get you a working, scriptable local setup fast: point at any OpenAI-compatible endpoint, use one-shot mode, and use --yolo deliberately.
Trick 4 is the one that actually matters most for local models specifically — check chat_formatbefore you demo or rely on anything tool-related, because a "Content-only" template will make the model either honestly refuse or confidently hallucinate success, and only one of those is obvious at a glance.
Trick 5saves the ten minutes of confusion that comes from a PATH bug that only shows up inside Hermes's subprocess launcher, never at the terminal.