Jonathon's AI Wiki

The Compute Economics of the AI Buildout (Epoch AI, 2026)

16 min read

Source: raw/newsletter-epoch-ai-add82097a6.md (“Is a compute crunch coming?”, Luke Emberson & Jaime Sevilla) · raw/newsletter-epoch-ai-e05a84c055.md (“Frontier labs don’t use most AI compute (yet)”, Josh You) · raw/newsletter-epoch-ai-a6ad3f7054.md (“Diversion and resale: estimating compute smuggling to China”, Isabel Juniewicz) · raw/newsletter-epoch-ai-6fb94702d7.md (“What we learned from 1,604 Chinese AI job postings”, Cheryl Wu, JS Denain, Anson Ho) · Epoch Brief data-insights: raw/newsletter-epoch-ai-61525dda28.md (June 1), raw/newsletter-epoch-ai-66b49f0f8d.md (May 22), raw/newsletter-epoch-ai-45df197d73.md (May 15), raw/newsletter-epoch-ai-ac7e42b77d.md (June 12), raw/newsletter-epoch-ai-b3167bb772.md (June 26), raw/newsletter-epoch-ai-b9855d3cc7.md (May 8) Publisher: Epoch AI. The three lead pieces are Gradient Updates — Epoch’s explicitly opinionated/informal series representing the named authors’ views, not an Epoch institutional position. The smuggling piece excerpts a fuller Epoch report; the bracketed Data Insights are Epoch’s standard (non-opinion) data work.

A consolidation of Epoch AI’s mid-2026 quantitative work on the cost and physical limits of the AI buildout, distilled for the one question a practitioner actually cares about: what does this imply for the price and availability of frontier intelligence? The through-line across all of it is a squeeze — token demand appears to be outrunning inference supply, the two labs driving demand (Anthropic and OpenAI) are on track to absorb the world’s compute headroom within a few years, and the physical inputs (memory, data centers, chips) are the binding constraints. This is capability-and-cost forecasting, not marketing advice — but it grounds decisions about model choice and when to expect frontier access to get more expensive.

Key Takeaways

  • A compute crunch is plausibly near — and it hits long-context agentic workloads first. Emberson & Sevilla model that today’s Blackwell GPUs could serve 500 million–20 billion output tokens/second, with global inference capacity more than tripling each year (~3–4×). But token demand — estimated at 200 million–4 billion tokens/second at current prices — appears to be growing ~10× per year, plausibly outpacing supply already. ^[The supply and demand ranges are wide and the authors flag them as highly uncertain.]
  • The practical consequence: frontier long-context access gets pricier; everyday users get pushed to smaller/cheaper models. Epoch’s own read — “the price of access to frontier capabilities [rises] for those willing to pay, while everyday users shift to cheaper, smaller models.” Because inference/training efficiency keeps improving fast, this need not mean worse quality: tomorrow’s smaller, cheaper models quickly match today’s frontier. This is the cost lever behind the wiki’s cheap-workhorse-vs-daily-driver split.
  • Frontier labs don’t yet use most AI compute — but Anthropic and OpenAI are on track to. Josh You estimates the five most compute-rich developers (OpenAI, Anthropic, xAI, Google DeepMind, Meta) together held under half of world compute at end-2025; OpenAI alone was ~10–15%. Yet OpenAI and Anthropic are growing compute ~4×/year vs the industry’s ~3× — enough to consume the headroom and reach ~80% of world compute within ~5 years, at which point scaling stalls unless total capex accelerates dramatically.
  • The buildout is already ~1T annualized in 2026 (~1% of gross world product, ~3% of US GDP); hyperscaler capex has quadrupled since GPT-4 and is on track to overtake hyperscalers’ operating cash flows by end of 2026, forcing external financing. GPU-hour rental prices rose ~30% this year — a supply-tightness signal you can already feel in API pricing.
  • Memory, not logic, is the bottleneck. High-bandwidth memory (HBM) grew from 52% to 63% of AI-chip component cost (Q1 2024→Q4 2025), spend rising ~32B — faster than any other component. Servers account for 60% of the total cost of owning a 1 GW data center (8.5B/year), dwarfing energy.
  • China’s compute is heavily smuggled, and Chinese labs lean on domestic chips only for the easy jobs. Epoch estimates 660,000 H100-equivalents smuggled into China through 2025 (90% CI 290k–1.6M) — roughly a quarter to a third of China’s total AI compute. Separately, Chinese job postings suggest domestic chips are used frequently for inference, rarely for pretraining large models (GLM-Image, at 16B params, was trained entirely on domestic chips — but that is 10–100× smaller than their frontier models).
  • Open-weight models trail the closed frontier by ~4 months in Epoch’s capability index (an 8-point gap, ≈ the GPT-5→GPT-5.5 difference) — the durable lag to price into any open-weight routing decision.

1. Is a compute crunch coming? (the demand/supply squeeze)

Gradient Update by Luke Emberson & Jaime Sevilla, 2026-05-26. The authors model the supply side precisely (how many tokens the world’s chips could serve) and compare it to imperfect demand proxies.

  • Supply. Serving a Kimi K2.6-class open model on all the world’s Nvidia GB200 + GB300 chips (1.9M and 1.5M GPUs at Q4 2025, ~40% of aggregate FLOP/s), and accounting for real-world inefficiencies (calibrated against SemiAnalysis’s InferenceX data → ~65% compute / 30% bandwidth efficiency), yields ~400,000 tok/s per GB200 NVL72 rack and a global total of 500 million–20 billion output tokens/second — depending heavily on request context length. That’s 150,000–7 million tokens per month for every person on earth. Capacity is more than tripling each year.
  • Demand. Proxies (Google’s ~1.2B tok/s across its platforms ≈ 130M output tok/s; Exponential View’s ~40 quadrillion tokens/quarter ≈ 5B tok/s industry-wide) suggest current demand of 200M–4B tokens/second, growing ~10×/year — a faster slope than supply’s ~3–4×.
  • The crunch and who feels it. If these slopes hold, demand overtakes supply soon (if not already), “particularly for the long-context workloads that drive agentic AI.” Long context is the culprit because attention compute grows quadratically — the batch size at which inference becomes compute-bound shrinks as context grows (e.g. from ~870 concurrent users per rack at 8K context down to ~130 at 25K). ^[inferred: the “agentic = long context = hits the compute wall first” causal chain is the article’s argument, restated here as the practical takeaway.]
  • Why quality need not regress. Inference and training efficiency are improving fast enough (Epoch’s own price-trend data) that “the smaller, cheaper models of tomorrow will quickly match today’s frontier.” The crunch reprices frontier access without necessarily degrading what everyday users get.

Practical read: budget for frontier long-context/agentic work to get relatively more expensive over time, and lean on the “this year’s mid-tier ≈ last year’s flagship” migration for routine work. This is the supply-side mechanism underneath Cost & Intelligence Levers.

2. Frontier labs don’t use most AI compute — yet (the consolidation clock)

Gradient Update by Josh You, 2026-05-21. The author flags these lab-compute estimates as more tentative than Epoch’s standard data work.

  • Today’s split. Cumulative AI compute sold ≈ 20M H100-equivalents (H100e) by end-2025; ~16M operational after a one-quarter install lag. OpenAI ≈ 1.7M H100e / 1.9 GW (up from 0.6 GW in 2024, 0.2 GW in 2023). Anthropic ≈ 1.4 GW (~70% of OpenAI), much of it ~500k H100e of Trainium2 at Amazon’s Project Rainier. OpenAI + Anthropic + xAI together < 4M H100e ≈ 20–30% of world compute; Google + Meta own ~1/3 of world compute but allocate perhaps half of that to their frontier labs (~15% of world total).
  • The consolidation clock. OpenAI and Anthropic grew compute ~4×/year in 2025 vs the industry’s ~3×. A naive extrapolation (top-two at 20% share today, growing 33% faster than the world) has them doubling their share in 2.5 years and reaching ~80% within five. Anthropic’s revenue run-rate went 30B annualized in Q1 2026 (an acceleration on last year’s ~10× growth), funding aggressive compute deals — including renting xAI’s entire ~300k-H100e Colossus 1 cluster (up to $15B/year).
  • The wall isn’t capability — it’s capex. Sustaining 4×/year once the headroom is gone would require more than doubling capex annually from a ~$1T-in-2027 base — feasible “only if AI starts to dramatically accelerate economic growth.” Otherwise frontier compute growth converges down to overall compute growth. Epoch is explicit this is not a hard 2029 “wall”: flat capex still grows the stock, chips keep improving. But the compute-scaling driver of progress is “not sustainable unless the world fundamentally changes soon.”

Practical read: the two vendors most likely to tighten supply and raise prices are also the two you most likely depend on. This is the supply-consolidation case for keeping a portable harness and an open-weight fallback (Lever 4).

3. The physical inputs: capex, memory, data centers

Epoch’s (non-opinion) Data Insights quantify the binding constraints:

ConstraintFindingSource
Hyperscaler capexQuadrupled since GPT-4; ~1T in 2027; Q1 2026 came in at 155.1B projection)June 1 brief
Capex vs cash flowOn track to overtake hyperscalers’ operating cash flows by end-2026 → external financing (Alphabet’s $80B raise, Amazon bond offerings)June 26 brief (also MirrorCode’s source)
Memory (HBM)Rose from 52% → 63% of AI-chip component cost (Q1’24→Q4’25); ~32B spend; the dominant cost and primary supply bottleneckMay 22 brief
Data-center servers60% of total cost of a 1 GW data center — 8.5B/year, dwarfing energyMay 15 brief (via May 22 ICYMI)
Single-site scaleRecord for compute at one data center has doubled every 7 months since Colossus 1 (Aug 2024)June 12 brief
Macro footprintAI infra ≈ 1.5% of US GDP (data-center construction alone ~0.8% in Q1 2026), up from a ~0.7% 2015–22 average; now the leading driver of US private-investment growthJune 12 brief
Compute ownershipFive hyperscalers own >2/3 of global AI computeMay 8 brief
Unit economicsAnthropic and OpenAI earn ~5.5M revenue per employee — higher than any public tech company on the Forbes Global 2000May 8 brief

Practical read: HBM is the choke point, so supply relief depends on memory fabs, not just logic nodes — a slower lever. The revenue-per-employee figures are a useful sanity check when sizing what these labs can afford to spend on compute.

4. China: smuggled chips + a domestic-silicon workaround

Two Epoch pieces bear on Chinese compute — relevant for anyone forecasting the open-weight frontier (GLM-5.2, Kimi K3).

  • Smuggling (Isabel Juniewicz, full report). Cumulative allegations of diverted/missing chips total ~300,000 H100e by end-2025 — about a quarter of what China acquired legally. A Monte-Carlo estimate puts true smuggling at 660,000 H100e (90% CI 290k–1.6M) ≈ 3% of the global compute stockpile (comparable to what xAI held), with the upper bound implying most of China’s compute was smuggled. Context: China imported ~2.5B Supermicro diversion scheme. Cloud compute outside China used by Chinese customers is a separate, generally-permitted channel not counted here.
  • Domestic chips, from 1,604 job postings (Cheryl Wu, JS Denain, Anson Ho). Scraping DeepSeek, MiniMax, Moonshot, Z.ai, ByteDance, and Alibaba postings: Chinese labs still depend on Nvidia/CUDA/TensorRT-LLM for inference, but are actively hiring for Ascend/Cambricon “heterogeneous computing.” Best guess: domestic chips are used frequently for inference, rarely for pretraining large models — the exception being post-training or small models (Z.ai’s GLM-Image, 16B params, was trained entirely on domestic chips, but that’s 10–100× smaller than their frontier models). Other findings worth keeping: Chinese labs have distinct “personalities” (Z.ai is B2B-heavy like Anthropic — 73.7% of 2025 revenue from running models on customers’ infra; MiniMax/Moonshot are consumer/international — 73% of MiniMax revenue is international vs Z.ai’s 9.8%), require far less experience (US labs avg 5.5 years vs 1.6 in China), and cluster in Beijing/Hangzhou/Shanghai (93% of located postings) rather than one hub the way US labs cluster in SF (85%).
  • The open-closed gap. Epoch’s index puts the best open-weight models ~4 months behind the closed frontier since Jan 2026 — an 8-point capability gap ≈ GPT-5→GPT-5.5.

Practical read: export controls are leakier than headline policy implies, but China’s frontier training is still Nvidia-bound; the domestic-chip story is real for inference and small models, not yet for frontier pretraining. Price the ~4-month open-weight lag into any cost-driven routing to GLM/Kimi-class models.

Epoch Brief, 2026-07-31 — five new publications (2026-08-05 addition)

Source: raw/newsletter-epoch-ai-49dab45301.md — The Epoch Brief, 2026-07-31, Elliot Stewart. Pointers only — none of the five underlying publications has been fetched into this vault. Recorded here so they are findable, and filed as research-agenda candidates.

  • Parallelization constraints could delay a technological singularity (epoch.ai/publications/parallelization-constraints-could-delay-a-technological-singularity) — the parallelizability of AI R&D, framed as “a critical and overlooked parameter for determining when, or even if, a technological singularity will happen.” The most consequential of the five for this article, since it constrains the compute-to-capability translation this piece models: compute you cannot parallelize into research throughput does not buy proportional progress.
  • AI energy demands guide (epoch.ai/publications/ai-energy) — including climate and local-community impacts. Directly adjacent to this article’s compute-economics scope.
  • FrontierMath: Open Problems (epoch.ai/frontiermath/open-problems) — an expanded benchmark built from unsolved research-mathematics problems. Relevant to the benchmarks article: a benchmark with no known answer key is a structurally different evaluation object.
  • Data Insight — reliability of AI-text detectors (epoch.ai/data-insights/ai-detectors-false-negatives), on false negatives.
  • Data Insight — evidence of AI uplift in OpenAI’s Codex repo (epoch.ai/data-insights/codex-engineer-effort) — measured engineer-effort uplift from a real repository. This is the rare measured productivity datum in a literature dominated by vendor claims and self-report, and worth prioritising for a full ingest.

Try It

  • Treat “frontier gets pricier, mid-tier catches up” as a planning assumption, not a hope. The compute crunch predicts exactly the pattern the wiki’s cost work already recommends: route routine, center-of-distribution work to cheaper/smaller models and reserve frontier spend for genuinely novel-capability tasks. See Cost & Intelligence Levers.
  • Watch long-context pricing specifically. Agentic and long-context workloads are the first to hit the compute wall. If your workflow depends on 100K+ token contexts, expect that to be where price pressure and rate limits land first — design for context economy.
  • Keep an open-weight fallback, and price the lag. The supply-consolidation thesis (Anthropic + OpenAI absorbing headroom) is the structural case for a portable harness. Budget open-weight routing at a ~4-month capability lag, not parity.
  • Use HBM as your supply-relief indicator. Because memory is the binding constraint, watch HBM capacity announcements (not logic-node news) as the leading signal for whether the crunch eases.
  • Sanity-check “AI bubble” claims against the capex-vs-cash-flow line. Epoch’s data (capex overtaking operating cash flow, forcing external financing in 2026) is the concrete number to cite when the buildout’s sustainability comes up.

The measurement problem underneath: AI’s GDP contribution is understated (Epoch, 2026-08-24)

A companion Epoch report argues that the standard explanation for a puzzle in this space is incomplete — and the correction matters for anyone reasoning about the buildout from macro data.

The puzzle. US investment in computing equipment has roughly tripled since 2023, to about $400 billion per year, while the measured impact on GDP growth stayed modest. The conventional explanation is that much of the spend goes on imported technology goods, which are subtracted from GDP.

Epoch’s claim: that explanation is incomplete. GDP statistics have a blind spot around the value American firms — Nvidia above all — create by designing AI chips that are manufactured and sold abroad. The mechanism is specific: because no physical goods leave the US, no goods export is recorded; and because no foreign buyer pays explicitly for the IP, no IP export is recorded either. Epoch says it reviewed every category where the value-add could plausibly land — goods exports, IP exports, service exports, merchanting — found it in none, and confirmed the analysis with the Bureau of Economic Analysis.

The size. US GDP growth over the past year is understated by about 0.3 percentage points. If Nvidia’s growth continues at its current pace, Epoch projects the gap could widen to almost two percentage points of growth per year by 2028.

Why now, given the blind spot is old. It applies to any “factoryless” manufacturer and has existed for years; historically the value that slipped through was small. Nvidia’s growth changed the magnitude, not the mechanism.

The fix, and why it is slow. International guidelines updated in 2025 already call for recording factoryless manufacturers’ overseas sales as goods exports. Epoch outlines an alternative — recording their markups as IP exports — which departs from international standards but sidesteps political resistance to counting overseas production as US manufacturing. Either change “could take years to implement.”

The practical caution for this wiki: any argument about whether AI investment is producing returns that leans on measured GDP contribution is leaning on a number with a known, quantified, and growing understatement in it. That cuts against bubble arguments built on “the macro data does not show it” — and it does not, on its own, establish that the returns are there.

Open Questions

  • The full Epoch report was not fetched — this section is built from the newsletter summary announcing it. The methodology behind the 0.3pp and ~2pp figures is not captured here.

  • How much of the understated value is Nvidia specifically? Epoch says the blind spot applies to other factoryless manufacturers, but every figure quoted is Nvidia-anchored.

  • How wide are the true supply/demand slopes? Epoch’s own ranges (supply 500M–20B tok/s; demand 200M–4B tok/s) are order-of-magnitude wide and depend on unknown average model size behind current demand. The crunch timing is directional, not precise.

  • Will Anthropic/OpenAI actually sustain ~4×/year? The consolidation clock assumes continued 4× compute growth; a capex slowdown after 2026 (not guaranteed) would push the “headroom exhausted” date out.

  • Does the ~$1T-capex-can’t-double-again ceiling bind, or does AI-driven growth relieve it? The entire “scaling stalls” conclusion hinges on whether AI accelerates economic growth enough to fund the next doubling — Epoch leaves this open.

  • How much undetected smuggling? The 660k median is bracketed by heavy uncertainty on detection rate (median 24.5%) and whether allegedly-diverted chips actually reached China.

  • Not folded here — Epoch’s AGI-transition economics tail. Four adjacent Epoch Gradient Updates (superstar-researcher pay, capital control after AGI, the “missing half” of futurism, an O*NET for AI R&D) were read during this ingest but judged too abstract for this wiki’s applied bar. Flagged for the main session in case a dedicated “AGI-transition economics” synthesis is ever wanted.

Graph View

Backlinks