Source: raw/Culling_the_Herd_-_The_Secret_Sauce_and_Struggles_of_Running_AI_at_Home.md
Creator: Intelligent Machines (TWiT), episode 896, Leo Laporte with Paris Martineau and Jeff Jarvis; guest Ash Hart · URL: https://www.youtube.com/watch?v=9NlDUky6bQU · Duration: not stated (TensorFold interview is the first segment; the rest of the episode is news) · Platform: YouTube · Recorded: 2026-11-11
TensorFold is an inference engine for running open-weight models on your own hardware, written by one developer, Ash Hart (a UK CTO who handles cyber security for eight countries in his day job), mostly with AI coding help. It runs on CUDA, Apple Metal and AMD ROCm. Hart says it gives “perfect serial decode”: the same output the model would give without it, faster, rather than skipping computation. All speed figures below are Hart’s and Laporte’s own reports on a podcast, not independent benchmarks. The project site is tensorfold.dev, with code on GitHub.
Key Takeaways
- The idea: parallel lanes for one user. Data-centre engines are tuned for many users at once; a home user has one stream and idle silicon. TensorFold splits a prompt (his example: 512 tokens) into 16 lanes computed “in a diagonal line so they all know about each other”, then reads them at once. Someone online called it “token parallelism”.
- Claimed speedups are usually 2x, sometimes 4x or more, and depend on the model’s architecture. Models such as DeepSeek 4.1 with a large lookup table can offload that table to NVMe for further gains.
- Hart’s numbers on Qwen 27B (dense):
- M5 Max: serial decode about 22 tokens/s before; “yesterday I did 130”.
- M5 Max, 128 lanes, 4-bit MLX: 622 tokens/s.
- RTX 5090 on native Windows: about 200 tokens/s, then “upwards of 1,600” after lane and diagonal batching.
- ^[ambiguous: the 622 and 1,600 figures follow talk of multiple lanes and concurrent connections, so they may be total throughput across lanes rather than one stream; the transcript does not say]
- Laporte’s number: GLM 5.3 Flash in a 3-bit EXL3 quant on two DGX Sparks went from about 20-30 tokens/s to roughly double.
- Hart thinks the engine reaches about 30% of the throughput the hardware allows.
- The engine matters as much as the hardware and the model. Laporte’s summary: hardware, model and engine (vLLM, DSpark, TensorFold) each move speed a lot, and engines are improving fastest.
Other local-AI details from the episode
- Mac plus Spark, split by job (“MCDMA”, Mac-CUDA direct memory access). Hart made RDMA work between a Mac and an Nvidia machine, both holding the same model: the Spark does prefill (it has stronger GPUs) and the Mac Studio does decode (it has faster memory). Over ordinary Cat 5 this works for disaggregated runs; low-latency use needs RDMA over Thunderbolt 5. He says Apple allows one RDMA cable and he has enabled up to six, about 480 Gbit/s aggregate.
- Hardware mentioned: DGX Spark memory bandwidth 273 GB/s (Laporte found his pair disappointing for two months until better recipes arrived); M5 Ultra Mac Studio, 512 GB, 1.2 TB/s bandwidth, first available 2026-11-17, with the 512 GB option about 4,000 against the Spark’s $5,000 (Laporte; Hart calls AMD’s platform “Venice”, ^[ambiguous: the chat also calls it Ryzen Halo]).
- Speed versus smarts. Hart uses Qwen 3.8 Next Flash, which he rates “about 80 to 85% as good as” GLM 5.3 but three to four times quicker, for most work. Laporte: tokens per second mean little without the intelligence.
- Where TensorFold is going: DHH’s Linux distribution Omarchy (captioned “Omari”) will ship with it built in; Hart is writing his own Metal kernels (“MLX_TF”) because he thinks Metal is underperforming in software. Mia AI Labs publishes install “recipes” built on TensorFold that you point your coding agent at.
- Hart’s split: cloud for planning and frontier work; “90% of your stuff can be done at home.”
- Living weights (a toy): Hart has a separate experiment that updates a model’s weights in place so it remembers facts you tell it, including the contents of a 100-page document. He calls it a toy.
Try It
- Before buying hardware, try a faster engine on what you have. Laporte’s Sparks sat unused until a better engine and recipe arrived. Install TensorFold (tensorfold.dev / GitHub) or a Mia AI Labs recipe on your current machine and measure.
- Measure your own workload, not the headline. Record single-stream tokens/s and time to first token with one agent, then with the number of agents you actually run. The quoted 622 and 1,600 figures may be multi-lane totals.
- Check output quality, not only speed. Hart’s claim is identical output to serial decoding; run the same prompts with and without the engine and compare.
- Pick the model by quality per second. A model 80-85% as good at three to four times the speed (Hart’s Qwen Next Flash versus GLM 5.3) may suit agent loops better. See Qwen 3.8 27B.
Implementation
Tool/Service: TensorFold, a local inference engine (originally Python, now Zig, for direct Metal and CUDA access). Setup: from tensorfold.dev or GitHub; Laporte recommends Mia AI Labs’ one-click install recipes, driven by your coding agent. Cost: not stated. Hart funds hardware through a Ko-fi crowdfund (he says it raised about £24,000 in about 21 hours for an M5 Ultra Mac Studio); he is in Nvidia’s Inception programme. Integration notes: runs CUDA (DGX Spark, RTX 5090), Metal (Apple Silicon) and ROCm; mixed Mac-plus-Spark setups need the same model on both machines.
Open Questions
- No independent benchmark of TensorFold is in the vault. Whether the 622 and 1,600 tokens/s figures are single-stream or aggregate is unclear.
- “Perfect serial decode” (no quality loss) is the author’s claim; no comparison data was shown.
- Licence, pricing and repository URL were not given beyond tensorfold.dev.
- Hardware prices are hosts’ quotes on the show and may be out of date by the time you read this.
Related
- Qwen 3.8 27B — The Local-Model Inflection — the model most of these numbers are on, and the cost counter-case
- Mac Mini Local AI Economics — whether owning hardware pays off
- Hermes on Apple Silicon — Local Models — running an agent on local models
- Ollama + Claude Code — pointing Claude Code at a local model
- DeepSeek V4.1 Flash — an open-weight model in the same local tier
- Frontier Model Landscape, November 2026 — where open-weight models sit this month