Breaking, on purpose

Wednesday, 11 p.m. Sam sits in the empty office with a mug of cold tea and deliberately does to staging what Sunday will do to production.

The nightly fake marathon has a crowd dial, and tonight Sam turns it. A full field of synthetic runners. Half a million synthetic spectators, refreshing like people who love somebody out there. Then more than half a million. Simulating the big day’s traffic and watching what the machines do with it is load testingConcept · lights on your mapload testingSimulating peak traffic, or ten times it, while watching the system’s vital numbers. Capacity rather than correctness: how the terrible day is survived in advance., and the freeze week version has one rule: find the limit before Sunday does.

Its quieter cousin runs all year: performance testingConcept · lights on your mapperformance testingMeasuring response times and resource use under normal conditions. Speed rather than correctness, where load testing is survival rather than speed., measuring ordinary speed on ordinary days. Tonight is not about ordinary. Tonight the dial goes up until something breaks, on the theory, correct every single year, that something will.

What gave

The dial climbs, and staging holds, and holds. Then at 1:14 a.m., at one point four times projected race load, the projection service starts answering slowly, and everything that waits on it queues behind its slowness. The whole system’s speed collapses to the speed of its slowest necessary part, and that part has a name every performance conversation starts with: the bottleneckConcept · lights on your mapbottleneckThe one component that caps the whole system’s pace, the part everything else waits on. Adding capacity anywhere except the bottleneck changes nothing, which is why “where is the bottleneck?” opens every performance investigation.. The insight that pays for the whole night is the second sentence of its definition. Adding capacity anywhere except the bottleneck changes nothing. You can double every other machine Traversal rents and Sunday would still arrive at 1:14 a.m. and break in exactly this spot.

So the fix is narrow: the projection service gets more headroom and a cache it should always have had, riding the same fire lane Wednesday morning’s patch rode — tests and a way back. Staging gets broken twice more to prove the limit moved, and by 3 a.m. the fake race survives twice the projected peak. Sam terminates the synthetic crowd, writes four sentences in the capacity doc, and goes home past the bakery as it opens.

Thursday’s design review will want those four sentences. It is a meeting about a different question: not whether the system can take the load, but what taking it costs.