In the laya‑coreml library a single Core ML predict call that mixes two different question types takes about 850 ms, whereas a call that keeps the question type uniform finishes in 31‑42 ms. This twenty‑fold slowdown appears whenever the processing pipeline assumes that all items in a batch share the same computational shape and therefore must synchronize on the most demanding shape before any result can be returned. The mechanism is not a quirk of a particular model or framework; it is the inevitable consequence of a design that batches heterogeneous work and forces the whole batch to wait for the slowest element.
When a batch is assembled, the underlying executor inspects the incoming items to determine the resources it will need. It looks for the largest tensor shape, the greatest number of layers, or the most complex operation graph among the items. Once that maximum is identified, the executor allocates buffers, schedules kernels, and launches the computation for that worst‑case profile. Simpler items that could have been processed with fewer resources are nevertheless carried along in the same launch; they sit idle while the hardware finishes the extra work required by the complex items. The total latency of the batch is therefore the sum of the fixed overhead plus the time needed to execute the hardest case, independent of how easy the other cases are. If the easiest case would have taken t₁ and the hardest t₂, the batch latency approaches t₂ plus a constant overhead, giving a slowdown factor roughly equal to t₂⁄t₁ when t₂≫t₁. In the laya‑coreml example t₂ is about 850 ms and t₁ is about 35 ms, yielding the observed twenty‑fold increase.
This same pattern shows up whenever a system groups together items that differ in their processing cost and then treats the group as a single unit for scheduling or resource allocation. Consider a manufacturing line that assembles products in batches. If the line is set up to install a particular component that requires a specialized fixture, every product in the batch must wait while the fixture is engaged, even those products that do not need that component. Historically, the Venetian Arsenal in the fifteenth century built galleys and round‑hull ships in the same slipways. When a galley required a different rigging arrangement than a round‑hull, the slipway had to be retooled for the more complex rigging before any hull could leave the yard, delaying the simpler hulls despite their lower construction time. The throughput of the yard dropped not because the workers were slower but because the batch forced a uniform setup on heterogeneous tasks.
In the nineteenth‑century United States the boom of patent medicines offers another illustration. Manufacturers often blended an active alkaloid with a variety of inert fillers to increase volume and mask taste. Regulators of the era, lacking modern assay techniques, had to test each batch for safety by observing the slowest reacting component — usually the filler that dissolved most gradually. Because the test could not finish until the inert material had fully dispersed, batches containing a high proportion of filler took longer to clear, even though the active ingredient was present at the same concentration as in a purer batch. The net effect was that the time to market for a medicine depended on the proportion of the hardest‑to‑process excipient, not on the potency of the drug itself.
A comparable situation appears in modern computing when a thread pool receives a mix of I/O‑bound and CPU‑bound tasks and assigns them to a shared work queue without distinguishing their nature. The pool’s dispatcher typically pulls the next task from the head of the queue and assigns it to the first idle thread. If a long‑running CPU task sits at the front, all subsequently queued I/O tasks — which would complete in milliseconds if they could run immediately — are forced to wait for the CPU task to finish. This head‑of‑line blocking inflates the average response time far beyond the service time of the short tasks, exactly as the batch latency inflates when the hardest case dominates the schedule.
Biological systems exhibit the same constraint. A neuron integrates excitatory and inhibitory postsynaptic potentials across its dendrites. To reach firing threshold, the neuron must wait for the slowest depolarizing input to sum with the others; fast excitatory inputs that arrive early cannot trigger an action potential if a delayed inhibitory input is still canceling the charge. The neuron's response time is therefore limited by the slowest synaptic kinetics in the mixture, not by the average speed of its inputs. Experiments that vary the proportion of fast and slow synapses show a linear increase in latency with the fraction of slow synapses, mirroring the twenty‑fold slowdown observed when hard and easy question types are mixed in laya‑coreml.
In law, a court docket that bundles civil and criminal cases must allocate procedural steps according to the longest required process. Criminal cases often demand more extensive discovery, longer pre‑trial motions, and stricter evidentiary hearings than many civil matters. When a judge schedules a mixed docket, the clerk reserves time slots based on the criminal case’s needs; civil cases that could be resolved in a single hearing sit idle until the criminal case’s schedule clears. Historical records from the English Court of Chancery in the eighteenth century show that the average disposition time for simple debt claims rose sharply during periods when the court heard a high volume of complex equity suits, precisely because the docket forced a uniform timetable on heterogeneous litigation.
Financial clearinghouses provide a parallel example. When a clearinghouse nets trades for settlement, it must hold collateral until the latest settlement date among all netted positions. If a member’s portfolio includes both T+1 equities and T+2 fixed‑income instruments, the clearinghouse cannot release collateral tied to the equities until the fixed‑income leg settles, even though the equities are ready earlier. This results in higher capital costs for members whose portfolios contain a mix of short‑ and long‑dated securities, a phenomenon documented in the European Central Bank’s reports on collateral efficiency after the introduction of T+2 settlement for bonds in the 2010s.
Across these examples the causal chain is identical: a system aggregates items, determines a uniform processing requirement based on the most demanding item, allocates resources for that requirement, and then executes the batch as a whole. Items with lower intrinsic cost are forced to incur the idle time associated with the higher‑cost items, and the total latency scales with the ratio of hardest to easiest case rather than with the average case. The mechanism does not depend on any particular technology, era, or domain; it follows from the logical necessity of synchronizing heterogeneous work when the synchronization point is set by the maximum.
The laya‑coreml incident therefore is not an isolated bug in a machine‑learning wrapper; it is a concrete measurement of a general principle that whenever a batch processing interface ignores internal variance and imposes a uniform schedule, the observable performance will be bounded by the slowest constituent. The principle predicts slowdowns whenever the variance in processing time is large and the batch size is non‑trivial, and it predicts that reducing variance — by sorting items into homogeneous batches, by splitting the workload, or by exposing the scheduler to per‑item cost estimates — will restore latency close to the fast case. The persistence of the pattern across manufacturing, regulation, computing, biology, law, and finance shows that the underlying dynamic is a property of any system that couples tasks through a shared, max‑determined resource allocation step. Recognizing this coupling makes it possible to anticipate the slowdown without measuring every possible combination, and to design interfaces that either avoid batching heterogeneous work or provide a way to bypass the joint wait when it is not needed.