Hermes, my local AI assistant

What it is

I wanted an assistant for the jobs I give an AI most often: research, guidance, a bit of coding, the odd set of slides. I also wanted it to run fully on local hardware, independent of any frontier provider.

Hermes is that assistant. It chats through a desktop app, reads files, writes files and fetches live web pages. The model runs locally, and the only thing that leaves the machine is the web request itself.

I can also reach it on the go. Through Telegram, I message it from my phone and give it tasks while I’m away from my desk.

How it’s built

The hardware is a Dell Pro Max with GB10: an Arm machine with 128 GB of memory shared between CPU and GPU.

The model is Qwen3.8-27B in NVFP4, a 4-bit format that takes about 21 GiB. It’s served by vLLM in Docker, bound to localhost only, with a 262,144-token context window.

Speed comes from speculative decoding, where a small model drafts ahead and the main model checks the draft. I tested 4 setups and chose DFlash with a draft length of 7.

Setup	Single stream
No speculation	10.5 tokens/s
DFlash, draft 7	26.3 tokens/s

Through Hermes itself I see 33.7 tokens/s with thinking off and 37.6 with it on. At 8 streams the total reaches 105 to 138 tokens/s. A 106K-token prompt takes 99 seconds cold and 4 seconds once cached.

Web access has 2 parts. SearXNG handles search. A native Arm build of Chrome, driven by agent-browser, handles pages that need a real browser. I looked at Firecrawl and dropped it to keep the setup simple.

What went wrong

The first model, Qwen3.6-35B-A3B, wasn’t up to it. It made malformed tool calls, fell into loops and missed specs during deep research. A vLLM parser bug and my own config limits shared the blame. Qwen3.8-27B replaced it and hasn’t looped yet.

The rest was smaller, and each fix was short:

• The browser toolset was switched off in the config.
• Chrome for Testing has no Arm build, so I used native Chrome.
• Compose stripped the quotes from the speculative-decoding JSON.
• I’d set the context to 65,536 instead of 262,144.
• A crashed container had restarted 2,690 times and skewed my benchmarks.
• A bare custom provider quietly sends requests to OpenRouter. A named provider doesn’t.

One limit remains. A vehicle listings site blocked Hermes 3 ways: a bot check stopped the browser, search can’t run JavaScript, and curl got a 403. I handed that research to Claude instead.

What I’d do differently

1. Start on the right model. I lost time on Qwen3.6 before moving to Qwen3.8.
2. Check context length first. A wrong setting looks like a model problem.
3. Send one-off scraping to Claude. A local model is for steady work, not for fighting bot checks.