How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope — Summary & Key Points

"Batch size needs to be bigger than approximately 300 times sparsity."

The memory wall limits context

The analysis reveals that memory bandwidth, not compute, limits context length. The cost per token for KV cache grows linearly with context length, creating a 'Goldilocks zone' around 200k tokens where memory and compute are balanced. Beyond this, costs skyrocket, explaining why frontier models have stalled at 200k context despite theoretical potential.

Rack limits MoE scaling

Mixture of Experts requires an all-to-all communication pattern that demands full connectivity within a rack. Moving across racks introduces an 8x slower scale-out bottleneck, meaning the optimal deployment is a single rack of 72 GPUs. This physical constraint explains why labs like Google (with TPU pods) might have an advantage in scale-up domains.

Pricing reveals memory tiers

API pricing for cache hits and input/output tokens acts as a leak of hardware architecture. The 10x price difference for cache hits suggests a tiered memory system where HBM is used for short-term storage (5 minutes) and slower tiers like flash or spinning disk are used for longer retention. Additionally, a heuristic suggests labs over-train models by a factor of ~100 because training and inference costs should theoretically be equalized.