Jonathon's AI Wiki

Hermes on Apple Silicon — Local Model, Backend & Quant Guide (Mac/MLX)

13 min read

Source: raw/reddit-1uc7rw5.md — r/hermesagent “Mac + MLX Megathread — Hermes Agent on Apple Silicon” (OP Jonathan_Rivera, 142 score, last updated 2026-06-21; community-aggregated from 20+ threads + GitHub issues + benchmarks). 2026-07-14 addition: raw/Local_AI_models_explained_-_How_to_run_a_fleet_of_Mac_Studios_and_GPUs_at_home.md (How I AI episode, Claire Vo × Alex Finn — a cross-machine-class hardware framework beyond Apple Silicon; added as the section below). 2026-07-24 addition: raw/reddit-1v1tr2b.md — r/hermesagent “Qwen3.6 Community Variants 27B (Dense) & 35B-A3B (MoE) Definitive Guide for Limited Local Hardware” (same OP Jonathan_Rivera, 43 score, last updated 2026-07-20; a mostly-CUDA companion megathread — see the Mac-relevant extract below).

The most-asked Hermes-on-Mac question is “what model do I download, and why is tool calling broken?” This is the consolidated community answer for running Hermes Agent locally on Apple Silicon — model-by-RAM picks, the backend landscape, and the pitfalls that waste an afternoon. Caveat: this is a community megathread, not Nous’s official docs, and the local-model/quant/backend stack changes weekly — treat specific model names, tok/s figures, and bug numbers as time-stamped (June 2026) and verify against linked sources before relying on them. ^[the recommendations are extracted from the thread; the “best pick” framing is community consensus, not a benchmark this wiki ran]

Key Takeaways

  • 7B–9B is the floor for agent/tool use; 2B–4B models are chatbots, not agents. One tester: gemma4:e2b on an M4 16GB “can’t even handle one request”; Qwen3.5-9B “sort of worked.” Tool calling for 6+ chained calls is the real bar.
  • The current Mac default (32GB+) is Qwen3.6-35B-A3B (MoE, ~3B active/token, ~20GB at 4-bit MLX, runs like a 3B for speed). 16GB floor = Qwen3.5-9B Q4_K_M with a quantized KV cache and 64K (not 128K) context.
  • Tool calling breaking is the #1 failure mode — and it’s usually the backend, not the model. Test /v1/chat/completions with a tool-calling prompt before blaming a model; if one backend fails, try another.
  • MTP (multi-token prediction / speculative decode) is a net loss on Apple Metal — the opposite of what model cards market. Do not enable it on Mac.
  • Memory bandwidth beats chip generation. An M3 Max (~400 GB/s) generates tokens faster than an M4 Pro (~273 GB/s). Shopping for Hermes: Max > Pro > base, even a generation older.
  • The agent context tax is real and punishing on local Macs (context = RAM). Hermes’ orchestrator can spend ~15K tokens just to reply “hi.” The community consensus is a hybrid stack: fast local model for routine work + cheap cloud fallback for hard tasks.

Which model to download (by RAM)

Mac RAMDownloadBackendWhy
8GBQwen3.5-4B Q4_K_M / Gemma 4 E2B Q4Ollama / llama.cppSimple chat only — not heavy agent work
16GBQwen3.5-9B Q4_K_M or MLX 4-bitllama.cpp (compat) / OllamaPractical floor; preserve RAM for context, quantize KV
24GBQwen3.6-27B Q4_K_M or Qwen3.6-35B-A3B 4-bit MLXllama.cpp (dense) / MLX-LM (MoE)Dense = stronger/predictable coding; MoE = faster decode
32–48GBQwen3.6-35B-A3B 4-bit MLX (OptiQ if available)MLX-LM / oMLXThe Mac sweet spot — ~3B active, ~20GB file
48–64GBQwen3.6-27B Q6_K/Q8, or 35B-A3B 8-bitllama.cpp (dense quality) / MLX (MoE speed)Q6/Q8 is the serious-agent quant
64GB+Gemma 4 26B-A4B Q4 (MoE alt), Qwen3.6-27B Q8MLX / llama.cppMore RAM → better quants + longer context, not automatically a better model

Start with stock models. Uncensored variants (Heretic, HauhauCS) are advanced options — “the boring model that follows schema for 6+ tool calls beats the spicy one that talks itself into a ditch.”

Backend landscape

  • llama.cpp (GGUF) — max compatibility, full KV-cache quantization control (critical on 16GB), vision/mmproj, Jinja templates; fastest time-to-first-token in Hermes’ own testing. Predictable behavior often wins for tool loops despite slower raw generation.
  • MLX-LM / oMLX — 20–30% faster generation, gap widens on MoE (35B-A3B 4-bit: ~61 tok/s vs ~17 for dense 27B 4-bit on M1 Max 64GB). Known June-2026 bugs: Qwen3.5/3.6 non-Coder tool-parser mismatch (mlx-lm #1293), MTP-variant multi-turn failures (#1292).
  • Ollama — easiest setup; MLX backend (v0.19+) ~2× decode. Known KV-cache memory leak on M4 Max (#16698) that swap-deaths token gen — set OLLAMA_KV_CACHE_TYPE=q8_0 + OLLAMA_FLASH_ATTENTION=1.
  • LM Studio — GUI comfort; tool-call parser bugs on Qwen/Gemma in some versions (upgrade first).
  • Rapid-MLX (new, June 2026) — claims 2–4× faster than Ollama, 0.08s cached TTFT, 17 tool parsers / 100% tool-calling, drop-in OpenAI replacement; currently the strongest Mac backend for tool-calling reliability (3,000+ stars, actively maintained). Newer/less battle-tested than llama.cpp/Ollama.

Critical pitfalls (read before you waste 4 hours)

  1. MTP = slower on Mac. llama.cpp #23752: 25.3 → 19.3 tok/s; Qwen3.6-35B self-MTP collapses to 1.93 tok/s. Never enable --spec-type draft-mtp on Metal.
  2. Tool calling breaks across backends — test before trusting; rotate backends before blaming the model.
  3. 16GB is the floor, not the sweet spot. Usable RAM after macOS is ~10–12GB; a strong 9B at decent quant beats a crippled 27B in swap.
  4. Bandwidth > chip generation (see above).
  5. Agent context tax — quantize KV, keep context conservative (64K), consider a small fast orchestrator that delegates heavy work to cloud sub-agents.
  6. KV cache is the hidden memory killer — always --cache-type-k q8_0 --cache-type-v q4_0. One M3 Pro 18GB user went from timeouts to working via 4-bit KV quant + Hermes 0.8.0 lazy skill loading + 40K context (first message ~14K tokens, then ~600).
  7. Qwen overthinks — on RAM-limited Macs, turn thinking off to escape timeout territory.

Sampling (Qwen3.6, Hermes agent work)

WorkloadThinkingTempTop-PTop-KPresence
Coding / tool loopsON0.60.95200.0
Research / chatON0.8–1.00.95200.0
Summarization (latency)OFF0.70.8201.5

The Qwen3.6 variant megathread — what it changes for Mac (2026-07-24)

[Reddit signal — r/hermesagent 2026-07-20] Source: raw/reddit-1v1tr2b.md (43 score, 11 comments, same OP as the Mac/MLX megathread this article is built on; a combined refresh of that author’s May 24 and July 8 posts, mirrored at github.com/AtlasOmnia/hermesagent-megathreads). The thread’s own disclaimer applies to everything below: no benchmark in it is independent unless flagged; all are publisher-reported. The single exception it names is an 85 GPU-hour community shootout by nathandreamfast.

Most of this thread does not apply to Macs. Its center of gravity is the NVFP4 + Blackwell frontier — a 4-bit float format that will not load on Ampere or Ada, let alone Metal — and its headline MTP speedups (~1.5–2×, byte-for-byte verified at greedy decoding) are CUDA-side numbers. The Mac guidance in this article is unchanged: MTP remains a net loss on Apple Metal (see pitfall 1). Do not import the thread’s MTP or NVFP4 enthusiasm onto a Mac. What follows is the subset that does transfer.

The base-model choice, now with numbers

The 24GB row of the table above says “dense = stronger/predictable coding; MoE = faster decode.” The thread quantifies that with Qwen’s official (publisher-reported) figures:

Qwen3.6-27B (dense)Qwen3.6-35B-A3B (MoE)
Active params27B (all)~3B of 35B (256 experts, 8 routed + 1 shared)
SWE-bench Verified77.2%73.4%
Terminal-Bench 2.059.3%51.5%
MMLU-Pro86.2%85.2%
GPQA Diamond87.8%86.0%
AIME 202694.1%92.7%
LiveCodeBench v683.9%80.4%
Context262K native (1M w/ YaRN)262K native (1M w/ YaRN)
LicenseApache 2.0Apache 2.0

The thread’s summary: 27B dense wins coding by ~4 points of SWE-bench; 35B MoE is ~3.5× faster on the same hardware; math is near-identical. That is consistent with this article’s existing Mac recommendation — the Mac sweet spot is 35B-A3B because Apple Silicon is bandwidth-bound and the MoE’s ~3B active params are what make it usable, not because it is the better model. If your Hermes work is coding-heavy and you can tolerate the slower decode, the dense 27B is the stronger pick on quality.

Variant picks — what the one independent test found

This article’s standing advice is “start with stock models.” The thread doesn’t overturn that, but it does supply capability-preservation data for users who go off-stock:

  • Best-preserved abliterated variants: llmfan46 heretic (KL divergence 0.0021 vs base, refusals 6/100 down from 92/100, MMLU 86.65% → 85.67% — a 0.98% drop) and huihui-ai abliterated, which the 85 GPU-hour shootout rated tied-best with benchmark drops under 1%.
  • Publisher “better than base” claims did not survive testing. On AEON Ultimate Uncensored, the shootout’s verdict is quoted directly: “Claims NOT supported by benchmark data.”
  • Coder-branded fine-tunes can regress the thing they’re named for. Qwopus3.6-27B-Coder-MTP reports 67.0% SWE-bench Verified — about 10 points below the 77.2% base 27B. The thread’s read is that reasoning traces help structured agent workflows while hurting raw SWE-bench.
  • Reasoning-distilled variants (Opus 4.6/4.7 traces) emit <think> blocks even on trivial questions — the thread reports 5–30k thinking tokens per reply and a known looping risk, with the mitigation being a 4096-token reasoning budget. On a RAM-limited Mac this is the same trap as pitfall 7 (“Qwen overthinks”), so the existing advice — thinking off for latency-sensitive local work — holds and now has a second source.
  • A new agent-specific category exists: unsloth/Qwen-AgentWorld-35B-A3B, tuned for agentic tool-calling and claiming to address the looping and tool-leakage failures that plague heretic/distilled variants on long chains. No independent benchmarks published, so treat it as a candidate to test rather than a default. It is the most directly Hermes-relevant variant in the thread. ^[inferred — the thread lists it generally; the “most Hermes-relevant” judgment is this wiki’s]
  • Gemma 4 31B gets a one-line community nod for long-horizon agent loops over Qwen, with no data attached.

One number in the thread is internally inconsistent

The thread’s “what not to pick” section says NVFP4 on the 35B is “27% BELOW base 35B” on SWE-bench (citing 50.2% against the 73.4% Qwen-official figure), while its own RedHatAI table shows BF16 54.8 → NVFP4 50.2, a 91.6% recovery (~8.4% relative loss). Those two framings compare against different baselines and cannot both be the headline. Since NVFP4 is Blackwell-only and irrelevant on Metal, this article takes no position — but it is a reminder to read the underlying table rather than the summary line. ^[inferred — the inconsistency is this wiki’s observation, not flagged in the source]

Root cause of broken tool calling on Qwen3.6: backend vs. chat template

Existing claim: (from hermes-agent/hermes-apple-silicon-local-models.md, sourced to raw/reddit-1uc7rw5.md) — “Tool calling breaking is the #1 failure mode — and it’s usually the backend, not the model. Test /v1/chat/completions with a tool-calling prompt before blaming a model; if one backend fails, try another.” New source says: (from raw/reddit-1v1tr2b.md) — “The ‘template bug’ is the single biggest factor in Qwen 3.6 tool-calling quality. Froggeric’s patched chat templates fix most issues.” Quoting u/i_am_me0_0 (+47, r/LocalLLM): “The worse tool calling is because of bugs in the standard chat template. On huggingface froggeric has some patched templates. I use those and since that I have had almost no issues with tool calling for qwen 3.6 35b a3b q4km.” The thread further claims the widely-cited 95% agent success rate requires 27B Q8 plus narrow tool schemas plus the froggeric templates, and that stock settings yield roughly 75%. Why this matters: the two diagnoses prescribe different fixes — rotate backends, versus patch the chat template (huggingface.co/froggeric/Qwen-Fixed-Chat-Templates). Both are community claims from the same subreddit, neither independently verified, and both agree the model weights are not the problem. They may also be compatible — a backend swap can incidentally change which chat template ships — but no source establishes that. Status: unresolved

Beyond Apple Silicon — choosing local-inference hardware (2026-07-14)

A second community source — the How I AI episode (Claire Vo × Alex Finn, raw/Local_AI_models_explained_-_How_to_run_a_fleet_of_Mac_Studios_and_GPUs_at_home.md) — widens the picture past Mac/MLX to a cross-machine-class decision framework. The governing tradeoff is unified memory vs. VRAM vs. memory bandwidth. Creator claims (not benchmarked here)

ClassExampleMemorySpeedBest for
Big-unified-memory MacMac Studio 512GB (~$10k; creator runs three)Huge unified memory → runs frontier-size modelsLow bandwidth = slow — GLM 5.2 (≈Opus-4.8-level) runs on one Mac Studio, but a prompt can take ~5 min to respondFrontier intelligence locally when latency doesn’t matter
AI workstationNVIDIA DGX Spark (~$4,000–4,600)~128GB unified memory + CUDADecent memory and speed — the sweet spotMid-size models (e.g. Qwen-36-class) quickly; plug-and-play, headless (no monitor)
Discrete GPURTX 5090 (~$4,000)Only 32GB VRAMLightning-fast / cloud-like, very high bandwidthSpeed-critical work on smaller models
Legacy / smallMac mini, old laptopSmall—Embeddings + small point-solution models (e.g. agent memory)

Two patterns worth stealing:

  • The “always-on” cost reframe. The point of local isn’t ROI vs. an API bill — it’s unlimited 24/7 tokens on hardware you own, so an agent can run around the clock unmetered. Judge the purchase on that, not on per-token math.
  • Agent-as-IT-guy fleet provisioning over Tailscale. Put every box (Mac Studio, DGX Spark, 5090, laptop) on one Tailscale private network, then tell Hermes (or OpenClaw) to inspect each machine’s hardware, pick an appropriate model, and load it — no manual per-box setup. The same mesh lets you reach a box’s localhost from your phone. Suggested division of labor: a cheap local model as an always-on “BDR” for high-volume grunt work, with Claude Code as the “closer” (via /loop) for the hard tail.

Try It

  1. Match RAM → model from the table; stay stock first. On 16GB start with Qwen3.5-9B Q4_K_M.
  2. Run a local OpenAI-compatible server (llama.cpp llama-server … --jinja --cache-type-k q8_0 --cache-type-v q4_0, or Ollama/MLX-LM/Rapid-MLX).
  3. Point Hermes at it: hermes config set model.provider custom:local-mac; model.base_url http://127.0.0.1:8080/v1; model.api_key local-no-key; model.default <id>.
  4. Verify tool calling immediately — curl /v1/models, then a tool-calling prompt. If it loops/hangs, switch backend before switching model — and check the chat template too: a second community megathread blames the stock Qwen3.6 template for most tool-calling failures and points at froggeric’s patched templates. See the unresolved contradiction above; both are cheap to test, so try the template swap and the backend swap before concluding the model is at fault.
  5. Adopt the hybrid pattern — local for routine, a cheap cloud fallback (e.g. a 1/Mtok) for hard tasks; one reported M4 Max setup runs 95% local at ~$1/week.

Open Questions

  • How durable are these picks? Models/quants/backends turn over weekly — the article is a June-2026 snapshot; re-verify against the linked GitHub issues and HuggingFace pages before quoting figures.
  • Official Nous guidance — Nous’s Run Local LLMs on Mac doc should be the tiebreaker where it disagrees with community consensus.
  • Backend or chat template — which actually breaks Qwen3.6 tool calling? Unresolved (see the contradiction callout). A cheap resolution exists: hold the backend fixed, swap only the chat template, and measure tool-call success across a fixed set of prompts. Nobody in either source appears to have run that isolation test.
  • Do the froggeric-template numbers hold on Apple Silicon? The ~95%-with-fixes vs ~75%-stock figures come from a CUDA-centric thread; no Mac/MLX-specific measurement is cited, and MLX’s known Qwen3.5/3.6 tool-parser mismatch (mlx-lm #1293) is a separate failure path that a template patch would not address.
  • Is unsloth/Qwen-AgentWorld-35B-A3B actually better for Hermes tool loops? It is the first agent-tuned variant in the family and claims to fix looping and tool leakage, but has no independent benchmarks and no reported Mac/MLX quant testing.

Graph View

Backlinks