Moving a live business off Firestore with zero downtime

On a whiteboard, a database migration is a straight line. Export the old store, import the new one, point the app at it, go home. Then you try it under a business that is actually open. Somewhere out there it's four o'clock on a Tuesday. An account manager is sending an invoice, someone is dragging a deal into closed won, and a designer is logging the afternoon against a client. None of them agreed to a maintenance window, and no client-services company is going to stop billing for a weekend so the engineering can be tidy.

So the app had to keep reading and writing every second while we changed what it was reading from and writing to. Dozens of collections of live customer data, off Firestore and onto Aurora Postgres, with nobody on the other side of the screen noticing anything at all.

The failure that doesn't announce itself

Downtime is the risk everyone pictures, and it's the easy one, because an outage tells you it happened. The failure that keeps you up is a split brain: a window where some records live in the old store, some live in the new one, and a read can land on either. A deal gets updated in Aurora. A report reads the stale Firestore copy. Now the software is confidently wrong about a customer's business, and it stays confidently wrong until somebody happens to notice the numbers don't agree.

That made the requirement stricter than “no downtime.” At every instant, for every single record, exactly one store had to be the source of truth, and every reader had to agree on which one it was. You don't get there with a big-bang cutover and a held breath. You get there by moving in pieces small enough to check.

One door, so there's only one thing to change

What made this survivable was a decision taken long before the migration: nothing in the frontend talks to a database directly. Every request for app data goes through a single signed gateway, an HMAC-signed bridge sitting between the app and whatever happens to be storing the data. That discipline costs you a little friction on ordinary days. It paid for itself completely here. Exactly one place in the product answered the question “where does this collection live?”, and no call site anywhere else had an opinion about it. Change the answer there and the whole app follows.

From there we went wave by wave, one collection or a small cluster of related ones at a time, never the whole store at once. Every wave had the same three beats. Backfill the data into Aurora. Point the bridge's reads and writes for that collection at Aurora. Leave the old Firestore path wired up as a fallback that fires only on an error. Stripped down, a wave looked like this:

// during a wave, for ONE collection:
async function read(collection, orgId, id) {
  if (cutover.has(collection)) {
    try {
      return await aurora.read(collection, orgId, id);   // new source of truth
    } catch (err) {
      log.warn('aurora read failed, falling back', { collection, err });
      return await firestore.read(collection, orgId, id); // ON ERROR ONLY
    }
  }
  return firestore.read(collection, orgId, id);           // not cut over yet
}

The fallback is where migrations quietly go wrong. A dual read that reconciles two stores on every request sounds safer and is precisely how you manufacture a split brain, so ours does nothing of the kind. Once a collection is cut over, Aurora is the answer. Firestore gets consulted only if Aurora actually errors, and every one of those falls is logged loudly. A fallback nobody watches is a split brain with better manners. We wanted ours visible, rare, and temporary.

Verify first, retire second

A wave wasn't done when the app looked fine. Apps look fine all the time. A wave was done when the numbers said so: row counts and content checksums matching across both stores, the same record byte-for-byte in each. We ran that check in dev, then staging, then production, in that order, and only once a collection was provably identical did the Firestore path for it come out.

A migration is finished when the old store is gone, not when the new one works. Those are different days.

We ran the environments in that order because production is the one place where a mistake costs a real customer real data. It should be the last environment in which any given change has never been seen before. By the time a wave arrived there, that exact wave had already run clean twice.

Making the win hard to lose

Firestore is no longer the data store. What remains of it is small and chosen on purpose: authentication, hosting, the signed bridge, and a small identity mirror. None of that holds the operating data of a business. Everything a customer actually runs their company on lives in Aurora.

Which left one last problem. A year from now, a tired engineer adds “just one” Firestore read for some app data, and the whole thing starts creeping backwards. So the codebase enforces it. There's a build-time guard, and if bare Firestore access for app data ever reappears, the build fails. Walking this back now takes a deliberate act.

There was never a dramatic moment in any of it. No dark night, no status page, no countdown on a call. It happened in small increments, each one verified before the previous exit was sealed, until the old database was simply gone. Invoices kept going out the whole time. Deals kept moving, hours kept landing against clients, and the whole thing was built so that nobody on the other side of the screen would have cause to wonder where their data was living that week.