Mission
By the end of this chapter, you can design a migration as controlled coexistence with business invariants, explicit state transitions, progressive exposure, real recovery, and proven decommission.
- Measurable outcome: Design a migration control plan that preserves explicit invariants through coexistence, cutover, rollback, and decommission; score at least 3 of 4.
- Prerequisites: Chapters 6, 7, 11, 15, 18, 25–27.
- Work product: A Migration State and Control Plan.
- Time: 80–105 minutes.
Before you read: Predict → Commit → Connect
Meridian Pay has copied all account records to a new regional datastore. Row counts match. Engineering proposes switching writes Friday night. Predict the strongest reason row-count equality is insufficient. Commit to an invariant that must remain true through cutover.
Connect to Chapter 6: data has state, ownership, ordering, and consistency behavior. Migration changes all four while the business continues to move.
A migration is a coexistence program
Teams often represent migration as two boxes and an arrow. The difficult program lives between the boxes: old and new contracts coexist, data changes during backfill, consumers upgrade at different speeds, teams need rollback, and operational ownership shifts. “Zero downtime” does not mean zero risk or zero user impact. Define it as observable service and integrity criteria, such as no unavailable user journey beyond the approved budget and no unaccounted data loss, duplication, or corruption.
Start with invariants: conditions that must remain true regardless of architecture:
- every accepted payment has one durable business identity;
- settlement is neither lost nor duplicated;
- authorization and ledger totals reconcile;
- customers see a monotonic, explainable state;
- audit history remains attributable;
- rollback does not reinterpret already committed events.
These are stronger than “new database healthy.” They make competing implementations comparable.
The exact states vary. What matters is that every transition has entry evidence, exit evidence, ownership, observation time, and a reversal or containment path.
Design coexistence deliberately
Inventory: Identify producers, consumers, schemas, API versions, events, batch jobs, reports, credentials, retention, regional restrictions, operational owners, and unknown clients. Traffic logs help but cannot prove a dormant quarterly process does not exist.
Compatibility: Decide how old and new contracts overlap. Additive changes are usually easier than destructive ones, but semantic changes can break consumers even when schemas parse. Version behavior, not merely syntax.
Backfill: Record source snapshot or watermark, transformation version, checksums or reconciliation logic, failures, retries, and late-arriving updates. Make the process resumable and idempotent where feasible.
Shadow and compare: Exercise the new path without making it authoritative. Compare outcomes at the business level, not only response codes. Understand why differences occur; do not normalize unexplained mismatches into an acceptable percentage.
Progressive traffic: Segment by risk and representativeness. A low-volume region may have unusual behavior and prove little about the primary population. Define exposure, observation duration, guardrails, and automatic/manual stop.
Authority transfer: Write when the new system becomes source of truth, how in-flight work resolves, which writes are allowed, and who decides conflict.
Decommission: Remove old writes, reads, credentials, jobs, dashboards, support paths, vendor spend, and assumptions. Archive required evidence and prove that no consumer remains. A powered-on fallback with no tested synchronization is not a rollback plan; it is a second stale system.
Rollback is a state transition, not a button
After the new system accepts writes, rollback must answer: which data returns, in what order, under which schema, with what conflict rule? Some changes are not cleanly reversible. For those, design forward recovery, traffic containment, read-only modes, or compensating transactions. Label them honestly.
Time rollback rehearsal against representative volume. Validate permissions, human roles, vendor availability, customer communication, and downstream recovery. Record the latest point at which rollback is safe; after it, the decision may become forward-fix or partial containment.
Decision rights: Who owns what?
- Architecture/Engineering owns migration design, contracts, data semantics, implementation, and technical evidence.
- Data owners own meaning, quality rules, retention, and reconciliation acceptance.
- SRE/Operations owns observability, production change control, recovery, and operational transfer.
- Security/Privacy/Compliance owns relevant control requirements, regional constraints, and evidence interpretation.
- Product/Business Operations owns acceptable customer and process impacts.
- The TPM owns the integrated state model, consumer migration, cross-team sequence, invariants, rehearsals, decision gates, communications, and decommission closure.
I do
For Meridian, I convert “row counts match” into three evidence layers:
- Structural: count, schema, checksum, referential integrity.
- Semantic: balances, settlement membership, state transitions, business identity.
- Operational: late updates, retries, reconciliation latency, alerts, and recovery.
I hold write cutover until unexplained semantic mismatches are zero for the defined population or explicitly dispositioned by authorized owners.
We do
During 10% traffic, the new service shows 0.03% more “completed” payments. Investigation finds the old service records timeout responses as pending even when the processor later confirms success. Which system is correct?
Together resist choosing by majority. Define the authoritative business event and reconciliation path.
Show the model answer
Model answer
Neither label is authoritative by itself. Reconcile against processor confirmation, ledger posting, idempotency identity, and settlement outcome. Classify each mismatch by cause. If the new service is semantically more accurate, update the invariant and downstream expectations deliberately; do not hide the change as migration noise. If evidence is incomplete, stop expansion while preserving current bounded traffic.
Rubric (0–4)
- 0: Assumes old or new is correct by default.
- 1: Requests more testing without defining authoritative evidence.
- 2: Names reconciliation sources but not decision/traffic action.
- 3: Defines business truth, mismatch classification, owner, and exposure decision.
- 4: Also addresses changed semantics, downstream contracts, historical comparability, and recovery.
You do
Build a Migration State and Control Plan:
| State/transition | Entry evidence | Invariants checked | Exposure | Observation window | Stop trigger | Recovery path | Decision owner |
|---|---|---|---|---|---|---|---|
| Shadow → 5% | Backfill reconciled; contract tests pass | Identity, balance, ordering | One low-risk segment | Full business cycle | Any unexplained integrity mismatch | Stop new writes; replay |
Add a consumer inventory, decommission checklist, and a “last safely reversible point.” Include one dormant or batch consumer deliberately; migrations often fail outside the visible request path.
Pause & Recall
Recall three invariant examples. From Chapter 15, why do consumers determine migration critical path? From Chapter 26, which journey SLI should govern traffic expansion? From Chapter 27, when does a mismatch become an incident?
Production lens
Run reconciliation continuously through at least one complete business cycle, not just during cutover. Freeze unrelated risky changes when appropriate, but avoid freezes so long that the migration becomes untestable against current software. Protect migration tooling like production software. Maintain change/version records so every transformed record can be explained.
Workplace artifact: migration cutover recommendation
Copyable cutover recommendation
Move [population] from [state] to [state] at [time]. Invariants [list] were verified by [evidence/window]. Known mismatches are [classified disposition]. Stop if [trigger]; recovery is [path] until [last reversible point]. Data authority during transition is [rule]. Decision owner: [role]; next expansion review: [time].
Chapter compression
A migration is a controlled coexistence of contracts, data states, consumers, and operators. Define business invariants, explicit transition states, representative observation, and real recovery. Transfer authority deliberately and prove decommission rather than abandoning the old path.
Retrieval deck
- Q: Why are matching row counts insufficient? A: They do not establish semantic correctness, ordering, business identity, or operational behavior.
- Q: What governs each migration transition? A: Entry/exit evidence, invariant checks, exposure, observation, stop trigger, recovery, and owner.
- Q: What is a shadow path? A: A non-authoritative execution used to compare behavior before controlling outcomes.
- Q: Why can rollback fail after writes move? A: State, schemas, ordering, and downstream actions may no longer be compatible with the old authority.
- Q: When is decommission complete? A: Consumers, data duties, credentials, operations, spend, and fallback assumptions are removed or transferred with proof.
Spaced review
- Now: Write three business invariants for the migration.
- +1 day: Redraw the migration state sequence and rollback transition from memory.
- +3 days: Define one transition's entry evidence, stop trigger, and recovery path.
- +7 days: Inventory one hidden, batch, or dormant consumer.
- +14 days: Compare predicted reconciliation classes with observed mismatches and revise the plan.