The Story Path/Act III of IV

Race Week

Race weekThe freeze, then four hours on Sunday morning.26 scenes · ~66 min

The freeze, the load tests, and then four hours on race morning when the system meets the day it was built for.

Previously

3 claims the story has made so far, with a piece of each left out. Reach for it before you read on; if you have not met them yet, the answer tells you where they happened.

  1. Maple Grove is to Harborview what the blank is to the production environment.

  2. The draw ran as one all-or-nothing unit. The database's word for that promise is a blank.

  3. The cheer feature reached production days before anyone saw it, waiting behind a blank.

The scenes

  1. The freezeMonday of race week. Nothing new reaches production until the race is run.2 min
  2. The advisoryWednesday, 8:12 a.m. The outside world declines to respect the freeze.3 min
  3. Breaking, on purposeWednesday, 11 p.m. Sam turns the crowd dial past race day.2 min
  4. What race day costsThursday, 10 a.m. The design review begins.2 min
  5. The numbers that matterThe review, hour two. Sam bans the word fast.3 min
  6. Growing biggerTwenty times bigger by Sunday, and two ways to get there.2 min
  7. What breaksThe hunt for the things there is only one of.3 min
  8. Data at scaleThe debt from Act II, paid at last.3 min
  9. Other people's computersFriday. The building, located at last.2 min
  10. Renting by the sliceWhat a server is when nobody owns it.2 min
  11. The two-day serverThe certificate generator works two days a year.2 min
  12. Renting, and sellingTraversal sells SaaS and buys IaaS, from both sides.2 min
  13. Boxes that shipThe box that killed the oldest joke in software.3 min
  14. Config, secrets, and codeSaturday. The most important change of the week builds no feature.3 min
  15. Reading the whiteboardSaturday afternoon. The wall, read in full.2 min
  16. Monolith and microservicesThe thick-markered box, and the argument both sides won.2 min
  17. Now, or laterThe queue, finally named.3 min
  18. Events, and who is listeningOne crossing, three consequences, zero introductions.3 min
  19. The front doorEverything between the internet and the servers.3 min
  20. Caches, the CDN, and stateWhich boxes are allowed to multiply.2 min
  21. The wall at dawnSunday, 5:00 a.m. The ops room, lit like a greenhouse.3 min
  22. The four signals8:00:00. The gun, and the curves leave the floor.3 min
  23. The pager at 9:41The phone moves toward the edge of the table, buzzing.2 min
  24. The bridge10:52. The call, made out loud.4 min
  25. The trace12:34:19. Case stops running; the questions start.2 min
  26. After the fireMonday. Blame systems, not people.3 min

What lights on your story map here

0/122concepts lit

one cell per scene · width is its conceptsreading a scene lights its cell

107 fluency — say them in a meeting7 recognition — nod when you hear them8 named technologies

Show all 122, scene by scene
productionmonitoringobservabilityrollbackhotfixbackward compatibilitybreaking changesecurity testingload testingperformance testingbottleneckscalabilityreliabilityavailabilitycostperformancelatencythroughputmaintainabilitysecurityhorizontal scalingvertical scalingload balancingsingle point of failureredundancyfailoverregionavailability zonereplicationshardingeventual consistencycachingdata centercloud computingon-premisescomputestoragenetworkingvirtual machineinstanceEC2managed serviceself-managed serviceserverlessLambdaobject storageIaaSPaaSSaaSS3containerDockercontainer imageregistryartifactKubernetesclusternodepodECSenvironment variableconfigurationsecretsecrets managerinfrastructure as codeSecrets Managersystem architecturearchitecture diagramservicedata flowmonolithmicroservicesdistributed systemsynchronous processingasynchronous processingqueueworkerSQSeventproducerconsumerpublish/subscribetopicSNSload balancerAPI gatewayproxyreverse proxyfirewallapplication cacheCDNstatefulstatelesslogmetricdashboardalerthealth checkSplunkerror ratesaturationuptimedowntimeautoscalingSLOSLAincidenton-call engineerescalationrunbookincident commanderservice owneroutagegraceful degradationresilienceback pressuredebuggingtracebugroot causeroot-cause analysispostmortem

Act III — the boundary

The half-second at the matThe machinery under load

480 msMolly's phone shows the split.

  1. 0 msThe chip wakes inside the mat's field.
  2. 2 msIt sends one number: 11208.
  3. 9 msThe mat's reader asks Traversal to record the crossing.
  4. 31 msTraversal writes the split down, and it stays written.
  5. 34 msThe write joins a queue, so nothing waits on what comes next.
  6. 38 msTraversal answers the reader: recorded.
  7. 120 msA worker takes the split off the queue.
  8. 190 msThe leaderboard's cached copy is refreshed.
  9. 480 msMolly's phone shows the split.

9 steps so far. The rest of the half-second arrives in later acts.

At 34 ms the write joins a queue; not until 120 ms does a worker take it off. Why is Traversal's answer to the mat's reader already out at 38 ms, long before the queued work is done?