Undo buttons: rollbacks, hotfixes, and compatibility

Every gate in this module will someday wave a bad change through because gates filter risk; they do not abolish it.

So the final discipline of shipping is knowing your undo buttons, and their limits.

What is a rollback?

The big red button is the rollbackConcept · lights on your maprollbackPutting yesterday’s version back after today’s misbehaves. Minutes of work when every build is kept and numbered, with a limit worth memorizing: it restores the code, not whatever the bad version already wrote into the database (Module 7).: redeploy the previous artifact.

Because every version is a preserved, numbered artifact (Module 11’s time machine, industrialized by the pipeline), returning to yesterday’s known-good build is a pipeline command, not an archaeology project:

But hear the veteran caveat because it reprises Module 7: code rolls back; data often doesn’t.

If the bad version ran a schema migration, or spent its forty minutes writing wrong rows into the database, restoring old code does not restore old truth. The bad writes are still there.

This is why migrations get rehearsed like surgery. And it is why “roll-back-ability” is itself a design requirement, considered before a change ships, not after it breaks.

What is a hotfix?

Sometimes backward isn’t an option. The old version has a security hole, or the bug corrupted the path back. Then comes the hotfixConcept · lights on your maphotfixAn emergency repair that jumps the queue: tiny change, urgent priority, every test still run, shipped the moment it passes. The pipeline with the lights flashing, not a detour around it., the smallest possible fix, rushed through the pipeline ahead of all normal work.

The operative word is through. A hotfix rides the same pipeline with every gate still on, sirens but the same road. And one that skips the tests is how one incident becomes two.

Engineers call this choice roll back versus fix forward: return to the old version, or rush the repair ahead. It is the first decision of every production fire, and Module 13 shows you the room where it gets made.

What is backward compatibility?

Underneath both buttons sits a quieter discipline: backward compatibilityConcept · lights on your mapbackward compatibilityA new version’s ability to get along with everything older than itself: yesterday’s data, last year’s app on a customer’s phone, the version before it still running on the next server over. Without it, no canary, no rollback, no rolling deploy can be safe.: new versions continuing to work with what already exists.

The reason is that old and new versions always briefly share the world. During a rolling deploy, some servers run the new code while others still run the old, and some customers stay on last month’s mobile app for a year (Module 5’s mobile reality).

New code must therefore tolerate:

• Old data — rows written by every version before it
• Old requests — clients still speaking last month’s API
• Old companions — instances of the previous version running beside it

That tolerance is what makes gradual rollouts, canaries, and rollbacks possible at all. Both versions must be able to hold hands.

What is a breaking change?

Break that tolerance and you have shipped a breaking changeConcept · lights on your mapbreaking changeThe opposite of backward compatible: a change that forces everything depending on the old behavior to break or be updated, such as renaming a field in Module 6’s API contract. Legitimate at times, always announced loudly, and the reason major version numbers (Module 3) exist., a change that old clients, old data, or old integrations cannot survive. Renamed API fields, dropped endpoints, and incompatible schemas are the usual culprits.

You have met this idea twice before, in Module 3’s major-version bump and Module 6’s broken API contract. This is the same event, now seen from the shipping side.

Not every breaking change is an accident. Sometimes the old behavior genuinely has to go, and the team breaks it on purpose. What separates a planned break from a disaster is warning. A planned one ships wearing a label, in the form of a major version number, an announcement, and a migration window, so everyone still relying on the old way has time to move.

The mental model to remember

A rollback redeploys the previous known-good artifact, fast and routine when versions are preserved. The honest caveat: code rolls back; data often doesn’t.

A hotfix is the smallest possible fix, rushed through the same pipeline with every gate still on. Choosing between them, roll back or fix forward, is the first decision of every production fire.

Backward compatibility is what makes gradual rollouts and rollbacks possible. Old and new versions briefly coexist, so they must understand each other.

A breaking change ends that tolerance. Sometimes necessary, never casual.

You should now be able to hear “we rolled back at 2:40” and ask the veteran’s question. What did the bad version write while it was live?