The recent demonstration that a 744‑billion‑parameter mixture‑of‑experts model can be run on a single MacBook Pro M5 Max by streaming its 434 GB of expert weights from NVMe storage rather than keeping them in RAM exposes a persistent structural dynamic: when a resource becomes a bottleneck, system designers repeatedly replace it with a cheaper but slower alternative, exploiting bandwidth‑limited I/O to mask the shortage, while the coupling between compute and storage remains a hidden constraint that reappears across centuries and domains.
The GLM‑5.3 configuration described in the public repository shows a model composed of 4‑bit quantised experts that together occupy 434 GB on disk. A single SSD delivers roughly 2 tokens s⁻¹; by striping the same data across four drives the throughput rises to 3.5 tokens s⁻¹, and an internal build with a yet‑unpublished patch reaches 4.2 tokens s⁻¹. The hardware platform provides 128 GB of RAM, far less than the model’s full footprint, so the system continuously streams weight slices into memory as needed. The method is built on the open‑source “argodrive” project, itself a fork of a lightweight disk‑based key‑value store. The announcement notes a convergence of tone between US and Chinese providers, and raises the question of whether the claimed 100 k accelerators are entirely locally manufactured.
At the core of this achievement lies a trade‑off: memory, the fastest addressable storage, is scarce and expensive; NVMe flash, while orders of magnitude slower in latency, offers far larger capacity at a lower cost per gigabyte. By arranging the data on multiple drives, the designers increase aggregate bandwidth, allowing the compute cores to keep pace with the streaming schedule. The performance numbers—2 tokens s⁻¹ per drive, 3.5 tokens s⁻¹ with four drives, 4.2 tokens s⁻¹ with a patch—are not the result of algorithmic breakthroughs but of a structural substitution that accepts higher latency in exchange for sufficient throughput.
This pattern recurs whenever a system confronts a resource ceiling. In the late 1960s, mainframe operating systems introduced virtual memory and paging to overcome the limited physical RAM of computers such as the IBM System/360. By moving infrequently used pages to magnetic disks, the OS could present applications with an illusion of abundant memory, despite the fact that disk access latencies were three orders of magnitude higher than RAM accesses. The performance penalty manifested as “thrashing” when the working set exceeded the available RAM, yet the strategy persisted because the cost of adding more memory outpaced the cost of adding disk capacity.
A similar substitution took place in the 1990s with the rise of web content delivery networks (CDNs). Edge servers, equipped with modest local storage, cached copies of popular objects that originally resided on distant origin servers. The bottleneck shifted from origin bandwidth to edge storage, and the CDN architecture relied on the fact that storage costs fell faster than network bandwidth costs. By distributing copies across many edge nodes, the system achieved higher aggregate throughput, trading the latency of a remote fetch for the slower, but plentiful, local disk reads.
The pattern extends beyond computing. In the early 19th century, the British railway network faced a shortage of locomotives capable of hauling long freight trains across the expanding system. Companies responded by constructing “wagon‑shops” that assembled trains from smaller, interchangeable wagons on‑the‑fly, using telegraph lines to coordinate the arrival of each component. The telegraph, a slower communication medium compared to visual signalling, became the substitute for direct line‑of‑sight coordination, allowing the railway to scale without the prohibitive expense of building a larger fleet of locomotives.
Even biological systems exhibit an analogous dynamic. When nutrient availability limits cell growth, some bacteria induce a “stringent response” that down‑regulates ribosomal synthesis (the fast, high‑cost translational machinery) and up‑regulates transport proteins that scavenge scarce resources from the environment. The transport proteins operate at a slower rate but are cheaper to maintain, enabling survival under prolonged scarcity.
Each of these cases shares three structural elements. First, a primary resource (fast memory, high‑throughput bandwidth, locomotives, ribosomes) becomes a limiting factor due to cost, physical constraints, or demand growth. Second, designers or organisms introduce a secondary resource (disk I/O, cached storage, telegraph signalling, transport proteins) that is abundant and inexpensive but slower or higher‑latency. Third, system performance hinges on the coupling between the fast and slow subsystems; when the coupling is insufficient—when the I/O channel cannot deliver data quickly enough—the overall throughput stalls, leading to observable degradation.
The GLM streaming effort illustrates how the coupling constraint is quantified in modern hardware. An M5 Max chip can issue roughly 30 tera‑operations s⁻¹, yet the streaming pipeline supplies only a few megabytes per second per drive. The disparity forces the inference engine to interleave compute with I/O waits, effectively idling the cores during each fetch. By striping across four drives, the aggregate bandwidth approaches the compute ceiling, but the system remains vulnerable: a single drive failure reduces throughput by more than a quarter, and any increase in model size or reduction in per‑token compute would again expose the I/O bottleneck.
Historical attempts to mitigate the same coupling have taken different forms. The introduction of hierarchical paging in the 1970s added a second‑level page table stored on disk, reducing the frequency of page faults by keeping a larger working set in fast memory. The deployment of “write‑back caches” in storage controllers in the 1980s introduced a fast DRAM buffer that absorbed small writes before flushing them to slower disks, effectively smoothing the I/O demand curve. In the 2000s, solid‑state drives (SSDs) replaced magnetic disks, narrowing the latency gap and allowing larger in‑memory datasets to be kept on relatively cheap flash, thereby shifting the substitution point further downstream. Each innovation re‑balanced the trade‑off but never eliminated the fundamental incentive to replace a scarce fast resource with a more plentiful slower one.
The incentive driving these substitutions is primarily economic: the marginal cost of adding a gigabyte of flash storage is orders of magnitude lower than the marginal cost of adding a gigabyte of DRAM, and the marginal cost of installing an additional telegraph line is lower than purchasing new locomotives. The system designer’s objective function—maximising throughput per unit cost—naturally leads to a solution that pushes the bottleneck onto the cheapest lever. This objective function is invariant across eras; whether the lever is a disk, a telegraph, or a transporter protein, the calculus remains the same.
Consequences of this substitution become apparent when the coupling is stressed. In the virtual memory era, aggressive multitasking on under‑provisioned machines produced thrashing, where the OS spent more time swapping pages than executing user code, effectively collapsing system performance. In CDNs, a sudden flash‑crowd on a popular video can saturate edge storage I/O, causing latency spikes despite abundant network capacity. In railway operations, a telegraph line failure can halt the assembly of wagons, stalling freight movement even though locomotives remain available. In microbial populations, over‑expression of transport proteins can drain cellular energy, reducing overall growth rates.
The GLM streaming setup is already encountering similar stress. The reported 4.2 tokens s⁻¹ internal build relies on a “not‑yet‑published patch” that presumably optimises prefetch patterns or reduces per‑token data transfer. Without such optimisation, the raw bandwidth of four NVMe drives would cap the system at 3.5 tokens s⁻¹, a figure insufficient for many real‑time applications. Any increase in model complexity—additional experts, higher precision, or larger context windows—will push the required bandwidth beyond the four‑drive limit, re‑introducing a performance ceiling.
Across domains, attempts to break the I/O substitution cycle have involved redesigning the fundamental architecture rather than merely adding more of the secondary resource. The shift from monolithic mainframes to distributed memory clusters in the 1990s replaced a single memory bottleneck with a network of machines, each with its own local memory, thereby moving the coupling point to the interconnect. In modern deep‑learning infrastructure, model parallelism distributes different layers across GPUs, while tensor‑parallelism splits the weight matrix itself across devices, reducing per‑device memory requirements without relying on disk streaming. However, these approaches introduce new coupling constraints—network latency, synchronization overhead—that become the next limiting factor.
The persistence of the I/O substitution cycle suggests that any solution must address the incentive structure directly. Cost reductions in fast memory (e.g., emerging non‑volatile RAM technologies) can shift the economic balance, making it cheaper to keep larger working sets in place. Similarly, advances in interconnect bandwidth (e.g., silicon‑photonic links) can raise the ceiling of the secondary resource, allowing streaming to keep pace with compute. Yet even as those technologies mature, the underlying calculus—minimise cost per unit of performance—will continue to drive designers toward the cheapest lever that satisfies the immediate throughput requirement.
The GLM streaming effort, the virtual memory systems of the 1960s, the CDN edge caches of the 1990s, the railway telegraph coordination of the 1800s, and the bacterial stringent response all illustrate the same structural dynamic: a scarcity of a high‑performance resource triggers the adoption of a more abundant but slower alternative, with system performance ultimately constrained by the coupling between the two. The specific numbers—744 billion parameters, 434 GB on disk, 2 tokens s⁻¹ per drive, 3.5 tokens s⁻¹ across four drives, 4.2 tokens s⁻¹ with a patch—are instances of a broader pattern that recurs whenever cost, physics, or biology limit the primary resource. Understanding this pattern clarifies why performance plateaus appear repeatedly, why incremental fixes (adding more drives, widening caches, installing more telegraph lines) postpone but do not resolve the underlying limitation, and why breakthroughs that alter the cost hierarchy of resources have outsized systemic impact.
The final implication is that any future architecture that hopes to scale without encountering a fresh coupling bottleneck must either eliminate the scarcity of the fast resource—through radical cost reductions or novel physical substrates—or redesign the workload to avoid reliance on that resource altogether. Until such a shift occurs, the I/O substitution cycle will reappear in new guises, each time preserving the same incentive‑driven structure that allowed a 744‑billion‑parameter model to run on a laptop by streaming from NVMe.