Pipeline Diagram — How This Wiki Actually Works (2026-05-09)

Source: This vault’s CLAUDE.md schema (every operation, layer, and script described in the diagram traces directly to a section there).

A single-image architecture diagram of the karpathy LLM wiki pipeline as it stands on 2026-05-09. Captures the end-to-end flow from inputs (with the /inbox-refresh skill at the top of the fan-in) through the immutable staging layer, the compile operation with its three-way triage: fork, the wiki layer, the publish pipeline, and the live Cloudflare + Notion outputs. The maintenance loop sits on top as a recurring band; QMD sits underneath as the retrieval substrate; bin/ scripts run the operational glue.

How to read this

• Six numbered columns, left-to-right: ① Inputs → ② Staging → ③ Compile/Ingest → ④ Wiki layer → ⑤ Publish → ⑥ Live.
• Maintenance loop runs across the top as a recurring band — Lint, Refresh (Tier-1 / Tier-2), Watch, Cross-link, Connection — and writes back into the wiki layer between operations. Friday 23:00 UTC and Sunday 16:00 UTC are the cloud-routine cadences.
• QMD substrate runs across the bottom: BM25 + vector + LLM rerank, ~2GB of GGUF models all-local. The 7-stage pipeline (query → expansion → parallel BM25/vec → RRF → reranker → position-aware blend → ranked results) is broken out so you can see where each stage lives. The substrate is queryable from every layer above it — by humans, by Claude during ingest, by lint checks, by cross-link sweeps.
• bin/ scripts band is the operational layer: post-ingest, lint-stale-sources, build-audit-db, refresh, yt-transcript, sync-notion, output-review. Idempotent; invoked manually or by the maintenance loop.
• Triage routing is the core decision in compile: every staged source carries a triage: field with one of three values (ingest / refresh:topic/article / skip:<reason>). Each routes differently — full new article, prefixed sentence in an existing article, or manifest-only trace. The skip path is load-bearing: it records what was considered and rejected, preserving the reasoning trail.
• Two-layer publish gate (the rose dashed box in column ⑤): quartz.config.ts ignorePatterns blocks topics wholesale, and per-article publish: true frontmatter is required by the ExplicitPublish plugin. Both must pass for an article to ship.
• Concrete numbers sit at the top: 297 published articles, 17 topics, 411 indexed docs / 8,497 vectors in QMD, 60-120s deploy time on the Cloudflare Worker.

Why this exists

This vault is meta — it documents itself. Most of the patterns visible here (the triage: field on raw frontmatter, the post-ingest hook chaining QMD refresh + audit DB rebuild + stale-sources lint, the priority-tiered lint auto-fix loop, the carry-forward refresh) emerged from concrete operational needs and got named in this CLAUDE.md after they paid off. The diagram exists so future-me, future-Claude, and any reader who lands at this URL can see the whole system at once instead of reconstructing it from 17 sections of a 600-line schema file.

It also doubles as a synthadoc-style architecture artifact: a single-glance reference for “what does a Karpathy LLM-wiki pattern actually look like in production?” Anyone copying this pattern can use the diagram as a checklist of layers, scripts, and feedback loops — and skip the ones their use case doesn’t need.

Related

• Synthadoc — LLM-Powered Wiki Engine — the architecturally-similar Python+Obsidian implementation that contributed several patterns visible here (status frontmatter, query decomposition, lint stale-sources, audit DB)
• Karpathy Wiki Additions from Synthadoc (2026-05-05) — the pattern-by-pattern ledger of what got adopted, what got skipped, and why
• Stride Starter Vault — minimal-template counterpart at the other end of the spectrum
• QMD — Local Hybrid-Search MCP — the retrieval substrate; everything in the bottom band of the diagram is QMD
• Karpathy Techniques for Claude Code — the wiki-vs-semantic-RAG tradeoff thesis the architecture answers
• Wiki Community Enhancements — broader survey of Karpathy-pattern community implementations

Try It

1. Read the diagram top-to-bottom, left-to-right. Title bar → maintenance band → six columns → QMD substrate → bin/ scripts → legend. The flow narrative at the very bottom of the legend block is the one-paragraph version.
2. Pull the source SVG. pipeline-diagram-2026-05-09.svg is a single self-contained file (~48KB) — embed it in slides, blog posts, READMEs without external dependencies.
3. Pull the @2x PNG (3,600 × 2,280) from output/diagrams/karpathy-pipeline-2026-05-09@2x.png for places that don’t render SVG (some Slack previews, some PDF readers).
4. Date-anchor. This snapshot is 2026-05-09. As the schema evolves the diagram will too — newer versions get fresh pipeline-diagram-YYYY-MM-DD.svg files; old ones stay for the historical record (immutable-by-naming, the same discipline raw/ and ai-research/ use).