OpenAI Symphony + Linear — Managing Autonomous Coding Agents from Your Phone

Source: raw/How_I_run_autonomous_coding_agents_from_my_phone_with_OpenAI_Symphony_+_Linear.md — How I AI podcast (host Claire Vo), episode with Alessio Finelli, founder of Kernel Labs and co-host of the Latent Space podcast (youtube.com/watch?v=KtmaWUVdnx4, fetched 2026-07-06).

Alessio Finelli walks through moving from being an “agent prompter” to an “agent manager”: running multiple autonomous coding agents on a cloud VPS, using OpenAI Symphony (an OpenAI open-sourced framework) to turn Linear tickets into agent work end-to-end, and managing the whole loop — including reviewing and requesting rework on pull requests — from his phone. He also demonstrates a second, non-software use case: autonomous Pokémon-card-trading research and pricing for his own card shop, illustrating the same pattern applied to a small business rather than a codebase.

Key Takeaways

• The shift is from prompting to managing. Finelli’s framing: early “autonomous coding” demos were fun but models weren’t good enough for genuinely long-running tasks. What changed his workflow wasn’t a smarter prompt — it was moving the whole loop off his laptop and onto a cloud VPS with multiple channels to reach the agents (text, Linear, direct shell), so intervention doesn’t require being at his desk.
• Zoo — his own agent+VPS setup. A cloud box (32GB RAM, 4 cores in his example) with coding agents pre-logged in, usable as his own always-on “sourcing” server; open-source models can run there too if desired.
• OpenAI Symphony, in plain terms: an open-sourced framework that watches a Linear board and, when an issue is moved to “To Do,” spins up a Codex agent against it. The agent creates a workpad — a plan, acceptance criteria, and validations — works the task, and opens a PR once ready, moving the Linear ticket to Human Review. Reviewer comments on the PR move it to Rework, where the agent generates a rework checklist addressing each comment line-by-line and re-submits; once merged, the ticket moves to Done.
• Linear becomes the state machine, not just a tracker. The whole point, in Finelli’s words: “you don’t really have to worry about the framework of how that task gets broken down, how it gets implemented, even how your comments get reviewed” — Linear’s states (Backlog → To Do → In Progress → Human Review → Rework → Done) carry the full lifecycle, reachable and steerable from the phone, desktop, or web without touching the underlying orchestration.
• Setting it up is mostly one markdown file. Finelli: the core of adapting Symphony to a new codebase is editing a workflow.md that describes how work should be done in that repo, then building a UI on top (Symphony ships with no visual UI and no built-in token-usage ledger by default — Finelli built that himself).
• Token cost is the practical unit of “how big is this task,” used ahead of time, not just after. Symphony logs token usage per task; most of Finelli’s tasks run 15–60M tokens, but one — migrating a system originally built to run locally onto a proper deployment (storage rewrite, request-handler changes) — ran 221 million tokens. The stated discipline: if a task’s actual token spend is wildly out of line with your prior expectation for a task of that shape, that’s a signal your tooling (not the model) needs improvement — better checks, better descriptions, better scoping.
• Glimpse — a Playwright extension Kernel Labs built so coding agents can take screenshots, run visual diffs between screenshots, and record video, specifically so long-running agent tasks can keep going and prove their own work without coming back for human review at every step.
• AGENTS.md / skill-file hygiene lessons. Models tend to add rather than remove instructions — telling an agent “you don’t need to always use the work-tree manager” produces an added caveat line, not a deleted rule, and the file grows more confusing over time. Finelli’s practice: periodically “red-diff” (review and prune) markdown/skill files rather than only ever appending to them. He’s also skeptical of highly-prescriptive per-skill files (citing Codex’s create skill feature) — an under-descriptive skill file can make a model more rigid and worse-off than giving it a general operating principle and letting it work.
• Practical business case study: Pokémon card trading (his shop, Merlin Games). Codex is given browser access and a defined skill to (a) extract PSA certificate numbers from card images to look up grade-specific pricing via an API, batching 5 cards at a time to avoid rate limits/blocks, and (b) search eBay for underpriced cards matching a “power buyer” watchlist, reconciling different grading companies’ equivalent grades (e.g., PSA 10 ≈ BGS/CGC 10). A second, in-progress use: real-time card pricing at trade shows, where manually searching each card on eBay/TCGPlayer while a seller waits is slow enough that “you’re actually losing a lot of money” — Finelli’s framing: for this class of task, raw AI response speed matters less than using autonomous, long-running research to save a real person’s clock time.
• Personal-life extension: context offloading. Finelli used a (since-discontinued) unlimited-token open-model service to have an agent read his Gmail every 5 minutes and flag only what needed a response, removing the low-grade anxiety of “should I check my inbox” — framed as the same “offload the checking, not just the doing” pattern as the business use cases above.
• Lightning-round advice for when an agent goes off the rails: switch to a different model/provider before re-prompting endlessly; restart the conversation; break the task into smaller pieces (“if you’re not getting enough failures, you’re probably not being ambitious enough” is offered as the opposite-direction caution); and — his own habit — switch from typing to dictating a longer, more detailed prompt once frustration sets in, since typing tends to under-specify compared to talking through the same request.

Try It

1. If you’re already using Codex or a similar coding agent day-to-day, look at OpenAI Symphony’s reference implementation and adapt its workflow.md to your own repo rather than building agent orchestration from scratch.
2. Put Linear (or your existing tracker) in the loop as the state machine before building a custom dashboard — Finelli’s point is that the ticket states already are the orchestration, and a tracker gives you free mobile/web access to it.
3. Start logging token cost per task even informally — it’s the cheapest leading indicator that a task is more (or less) complex than you scoped it for.
4. Schedule a recurring pass to prune your own AGENTS.md/CLAUDE.md/skill files rather than only appending — the failure mode Finelli describes (models add caveats instead of removing rules) applies to any long-lived instruction file, not just Symphony’s.
5. For a browser-dependent, repetitive research task (price-checking, data extraction), consider explicitly batching agent actions (his 5-per-batch eBay search rule) to avoid rate-limit/anti-bot blocks.

Related

• Nous Research Hermes Agent — a sibling self-hosted, persistent, cron/heartbeat-driven agent architecture; comparable “manage from anywhere” surface to Zoo + Symphony.
• Paperclip — Multi-Agent Company Orchestration Platform — another ticket/tracker-driven multi-agent orchestration pattern, closest architectural sibling to Symphony’s Linear-as-state-machine design.
• Ryan Carson’s Clawd Chief — same “agents are cron jobs and markdown files” thesis, different stack (OpenClaw instead of Symphony/Codex).
• Autobrowse — same “the agent needs to prove its own work without a human checking every step” problem Glimpse solves, different mechanism (strategy-graduation vs. screenshot/visual-diff).
• Maintain the Harness, Don’t Pile On Tools — directly relevant to the AGENTS.md-hygiene lesson here (files should shrink and get maintained, not just grow).
• The High-Trust Paperwork Agent Skeleton — same “agent prepares, human/gate approves” discipline as Symphony’s Human Review → Rework loop.

Open Questions

• Whether OpenAI Symphony has an official public repo/license beyond the “reference implementation” Finelli mentions adapting — not confirmed in the source.
• Cost of running Zoo (the VPS) itself, separate from model/token costs — not stated.
• Whether Glimpse is open-sourced or a Kernel Labs-internal tool only — not stated in the source.
• How Symphony’s workpad/acceptance-criteria generation differs mechanically from a standard Codex or Claude Code planning step — the source describes the workflow but not the underlying prompt/architecture.