Jonathon's AI Wiki

Hermes tldraw-offline Skill — Driving a Desktop Whiteboard by Local HTTP API

6 min read

Source: ai-research/hermes-tldraw-offline-skill-2026-07-24.md — the official Hermes Agent docs page for the skill (hermes-agent.nousresearch.com/docs/user-guide/skills/optional/creative/creative-tldraw-offline, fetched 2026-07-24), which publishes the complete SKILL.md verbatim. Announcement date and framing cross-checked against ai-research/nous-portal-model-promos-2026-07-24.md (x_search over @NousResearch, permalink x.com/NousResearch/status/2080319476243861569).

tldraw-offline is a first-party optional Hermes skill (v1.0.0, MIT, authored “Teknium + Hermes Agent”) announced 2026-07-23 — three days after the v0.19.0 Quicksilver release, and shipped as a skill rather than a version bump. It lets the agent read, edit, and script a canvas in the tldraw offline desktop app. The interesting part is not the diagramming: it is how the agent drives a GUI application. Instead of computer-use pixel-clicking, the skill talks to a local HTTP API the app already exposes, using plain curl from the agent’s terminal.

Key Takeaways

  • Install: hermes skills install official/creative/tldraw-offline. Path optional-skills/creative/tldraw-offline, version 1.0.0, MIT, platforms linux/macOS/Windows. Tags: tldraw, canvas, whiteboard, document-script, diagramming.
  • No computer-use, no file surgery. The SKILL.md explicitly states the agent does not GUI-click and does not hand-edit the .tldraw file. Both of the obvious approaches are ruled out in favor of the app’s own local HTTP API — the same path the tldraw homepage demo uses to have Codex edit a canvas live.
  • Local API on port 7236, Bearer-token auth. The token is read from server.json, whose location is platform-specific (~/.config/tldraw/ on Linux, ~/Library/Application Support/tldraw/ on macOS, %APPDATA%\tldraw\ on Windows). The skill instructs the agent to re-read port and token on every shell call, because each terminal invocation is a fresh session.
  • Two distinct workflows. POST /api/doc/{DOC}/exec runs transient code against a live editor object for one-off layout and shape generation. Writing script/main.js embeds JavaScript inside the .tldraw file so it runs on load — giving the document durable, reactive behavior that survives reload and travels with the file.
  • Document scripts are the novel primitive. A saved .tldraw file can open with interactive buttons and preserved state, because the behavior ships in the file rather than in the agent session that made it. The artifact stays live after the agent is gone.
  • The gotchas are re-entrancy gotchas. Scripts rerun on every load, so shape creation must be idempotent (createShapeIfMissing with stable IDs). store.listen fires one tick after a commit, not synchronously. Listeners must detach on signal abort or they duplicate. And editor.run(fn, { history: 'ignore' }) keeps script writes out of the user’s undo stack.
  • Keep the app open. The skill drives a running instance; it is not a headless .tldraw file generator.

Why the mechanism matters more than the feature

This wiki already documents the other way an agent controls a desktop app: Hermes Computer Use, which grants screen-recording and accessibility permissions and drives the GUI by looking at pixels and clicking. That path is general — it works on any app — but it is slow, brittle against layout changes, permission-heavy, and (per that article) prone to overloading the context window with screenshots.

tldraw-offline takes the opposite trade: give up generality, get determinism. Because tldraw offline happens to expose a local control API, the agent can issue typed calls with real return values (return editor.getCurrentPageShapes().length) instead of inferring success from a screenshot. No permission grants, no vision tokens, no click coordinates.^[the computer-use-vs-local-API comparison is this wiki’s framing; the SKILL.md states only that the agent does not use computer-use, without contrasting the two approaches]

The generalizable lesson for anyone writing skills: before reaching for computer-use, check whether the target app already speaks HTTP on localhost. Many Electron and local-first desktop apps do. That check is the difference between a reliable integration and a flaky one. This is the desktop-side echo of the argument in The Agent-Readable Web — agents work far better against a declared interface than against a surface built for human eyes.

Implementation

Tool/Service: tldraw-offline Hermes optional skill + the tldraw offline desktop app (offline.tldraw.com)

Setup:

  1. Install and launch the tldraw offline desktop app; keep it open while the agent works.
  2. hermes skills install official/creative/tldraw-offline
  3. The agent locates port + Bearer token in server.json under the platform config directory, then drives the canvas over curl.

Cost: The skill itself is MIT and free; cost is ordinary agent token usage. Because control is API calls rather than screenshots, it avoids the per-step image-token cost that computer-use incurs.^[inferred — the docs do not discuss cost]

Integration notes: Key endpoints are GET /api/search (find the focused document), POST /api/doc/{DOC}/exec (transient code with live editor + helpers), POST /api/doc/{DOC}/script-workspace (retrieve/edit the script file path), and GET /api/doc/{DOC}/script-status (verify the script applied). A minimal shape-creation call:

-H "Authorization: Bearer $TOKEN" \
 -d '{"code":"const {createShapeId,toRichText}=await import(\"tldraw\"); editor.createShape({id:createShapeId(),type:\"geo\",x:0,y:0,props:{geo:\"rectangle\",w:200,h:100,color:\"blue\",fill:\"solid\",richText:toRichText(\"hello\")}}); return editor.getCurrentPageShapes().length"}'

Try It

  1. Install the tldraw offline desktop app and leave it running with a canvas open.
  2. hermes skills install official/creative/tldraw-offline, then ask the agent to describe what is currently on your canvas — this exercises GET /api/search + a read exec and confirms the token path resolves on your platform.
  3. Ask for a generated diagram (an architecture sketch, a flow) via one-off exec calls. Check that shapes land where you expect before trying anything stateful.
  4. Then ask for a document script — e.g. a canvas with a button that toggles a layer. Save, close, and reopen the .tldraw file to confirm the behavior persisted without the agent running.
  5. If shapes duplicate on reopen, that is the idempotency gotcha: have the agent switch to createShapeIfMissing with stable IDs.
  6. Pair with skill bundles if diagramming is part of a recurring workflow — bundle it behind one slash command so the agent reliably loads it instead of probabilistically choosing it.

Open Questions

  • Adoption and maturity are unmeasured here. The skill is v1.0.0 and days old at ingest. No usage data, issue history, or failure reports have been reviewed — only the official docs page and the announcement.
  • Security posture is undocumented in the skill page. A localhost HTTP API that executes arbitrary JavaScript in the app context is a meaningful capability; the docs page covers authentication (Bearer token) but does not discuss what happens if that token leaks to another local process, nor whether the skill interacts with the approval gates described in the security model.
  • Whether the pattern generalizes officially. The SKILL.md notes the tldraw homepage demo uses this same local-API path with Codex, implying the API is a tldraw feature rather than a Hermes one. Whether Nous intends to ship more “drive a local app by its own API” skills is not stated.

Graph View

Backlinks