Data at scale

Act II named three words and promised race week would pay for them. Race week pays now. The half-million-spectator problem cannot be discussed without them.

Sunday’s arithmetic is brutally lopsided: a quarter-million writes, and reads beyond counting, since every one of half a million spectators refreshes all morning. The read side survives by copies. replicationConcept · lights on your mapreplicationKeeping live copies of the database on multiple machines. One safety story, the standby twin, and one scaling story: read replicas absorb the read-heavy traffic while the primary spends itself on writes. The catch is that copies take a moment to catch up. keeps live copies of the data on several machines, and the scaling trick is to let the copies answer the readers. The primary database spends its strength on the sacred writes. A small crowd of read replicas serves the leaderboard, each copy moments behind the truth and close enough for anyone.

When even one machine cannot hold the data itself, shardingConcept · lights on your mapshardingSplitting the data itself across multiple databases, customers A to M here, N to Z there, when it outgrows any single machine. Each shard is small and fast again, no machine sees everything, and it is famously hard to undo. splits it: this half of the alphabet here, that half there. Each shard is small and quick again, and no single machine can answer a question about everything, which is a real price. Traversal does not shard, does not need to at twenty-five thousand runners, and Sam says so in the review with visible relief, since sharding is famously the move you cannot easily take back.

The moment of disagreement, named at last

Copies lag. Read a just-written truth from a replica and you can receive the recent past, and this whole story has been rehearsing that fact since 6:04 on lottery day. Its name is eventual consistencyConcept · lights on your mapeventual consistencyWhen truth is copied across machines, the copies briefly lag: read from a replica and you may see the recent past. Eventual means the copies converge in moments. A chosen trade, fine for a leaderboard, wrong for a bank balance., and the fluency move is knowing it is a chosen trade with a dial. A leaderboard thirty seconds stale is fine. An entry count during the lottery cannot be, which is why the draw read the primary. Molly’s map on Sunday will be the trade made visible, and by then you will know exactly which dial somebody set.

And the family resemblance you may have noticed is real: cachingConcept · lights on your mapcachingThe whole three-layer story, browser, application, CDN, seen as one idea: every cache is a copy trading freshness for speed. Replication’s sibling, making the same bargain at every layer. is this same bargain, freshness for speed, made at every layer you have met: Molly’s browser, the projection service, the replicas. One family, one trade, and Sunday runs on it.

The review adjourns. Friday is for the machines themselves, and the buildings nobody at Traversal has ever stood inside.