Ten years of races

Finn’s spike ended on a folder of charts. This scene is where the charts came from.

The registrations database is built for now: current entries, current truth, fast answers. Ten years of finished races is a different appetite, and it lives in different furniture. The curated shelves, history reorganised for answering questions, are a data warehouseConcept · lights on your mapdata warehouseA system for storing and analysing large amounts of historical data, reorganised into shapes that make reporting easy. Curated shelves, where the lake is a permissive pool.. The permissive pool, splits and weather and GPS traces and a decade of photos kept raw, is a data lakeConcept · lights on your mapdata lakeA store holding large amounts of data in raw or flexible form, structured and unstructured together, permissive about what enters and how it is shaped.. Traversal has both, and the projected-finish model Finn is scoping wants to drink from each.

The plumbing between worlds

History does not walk to the warehouse. Every night an automated sequence carries the day’s truth across, and any such sequence is a data pipelineConcept · lights on your mapdata pipelineAn automated sequence of steps moving data from one place to another: reading sources, standardising, combining, producing the tables dashboards read. The plumbing between operational systems and analytics.. The classic recipe is ETLConcept · lights on your mapETLExtract, Transform, Load: pull data from sources, clean and reshape it, then write it into the destination. Transforms before loading, which is exactly what distinguishes it from ELT., clean first, then shelve; the modern reordering is ELTConcept · lights on your mapELTExtract, Load, Transform: land the raw data first and reshape it inside the destination. Neither order is superior; the difference is where the transforming happens., shelve raw, clean in place. Traversal’s nightly run is the second kind, and nobody watches it happen, which is the whole point.

Nightly is a choice, and you met its name last scene: the pipeline runs as a batch. Its opposite also has a home here. The timing events on race day will pour in continuously and be processed as they arrive, and that is streamingConcept · lights on your mapstreamingProcessing data continuously as events arrive, with low delay. Fresher than batch, and it pays for the freshness in infrastructure and failure-handling complexity.. History by the night, the race by the instant. Same data, two tempos.

One more word, and three saved for race week

Around any serious data conversation a few more words circulate, and one earns its definition now. partitioningRecognition — just know it existspartitioningDividing a large dataset into pieces, by date or region or key, so work can skip what it does not need. divides a big dataset into pieces so work can skip what it does not need, and the nightly run reads one race at a time for exactly this reason. Three others, replication, sharding, and eventual consistency, belong to race week, when Traversal’s copies multiply under load and briefly disagree. You have already lived the disagreement twice, at 6:04 and in Molly’s cached map. Act III gives the living a proper name.

The data exists. Finn’s maybe becomes a prototype by week ten, and Act IV finally opens the box. Now: the machines all of this runs on.