Automated Failover Is a Feature. DR Is a Discipline. Your Cloud Vendor Is Selling You the First.

May 7, 2026. 17:25 PDT. A room in an Amazon data centre in Northern Virginia overheated because multiple cooling units failed simultaneously. Within minutes, EC2 instances and EBS volumes on the affected racks lost power. Hardware was physically damaged.

Within the hour, Coinbase — one of the most technically sophisticated financial platforms on the internet — was down. Seven hours. Completely.

FanDuel went down mid-game during the Lakers vs Thunder Western Conference semifinal. CME Group’s trading platform reported issues. And Brian Armstrong, Coinbase’s CEO, posted on X the sentence that should be pinned to the wall of every cloud architecture review:

“We design our services to be redundant to downtime in any one AWS Availability Zone, and most of our systems worked this way — but not all.”

Most. But not all.

That word — all — is where the real conversation starts.

AWS multi-AZ is a genuine engineering achievement. Independent power feeds. Independent cooling. Independent physical infrastructure, separated by meaningful distance within a region. When a rack fails, a UPS blows, or a fibre cut takes out a single data centre building, multi-AZ works exactly as advertised.

For S3 buckets, read replicas, stateless API tiers, and session caches — it holds.

What AWS does not say in the sales deck:

• All six Availability Zones in us-east-1 share the same regional control plane
• They share the same IAM and STS endpoints
• They share the same internal DNS resolution path
• They share the same underlying network fabric that connects them

This is not a secret. It is simply never the headline.

In December 2021, us-east-1 suffered a major outage that took down CloudWatch, API Gateway, STS, Fargate, ECS, and EKS — simultaneously, across the entire region. Already-running instances were fine. But nothing could be launched, modified, or scaled. Multi-AZ did nothing, because the failure was above the AZ layer entirely. The control plane was gone. The AZs were irrelevant.

That was 2021. This is 2026. Same region. Same pattern. Different failure mode, identical outcome.

Here is the part the post-incident summaries glossed over.

Coinbase’s order book — the core matching engine that pairs buyers with sellers — cannot run simultaneously across multiple Availability Zones. This is not an oversight. It is not technical debt. It is physics.

Cross-AZ latency on AWS is 1–2 milliseconds round-trip. To maintain a consistent, sequentially-ordered book across two zones, every write must be acknowledged by both before it is committed. At trading volumes Coinbase operates, that 2ms becomes the ceiling on your throughput. You either accept the latency and lose institutional clients who co-locate for microsecond execution, or you accept a primary-zone model and accept that a zone failure takes the matching engine down.

There is no third option. Every serious financial exchange in the world runs a single-zone matching engine. The sophistication is not in eliminating that constraint — it is in knowing it exists, documenting it, and building explicit DR around it.

Coinbase knew. Armstrong confirmed it. What they did not have was a warm secondary that could take over the matching engine in a declared, tested, practiced way.

That is the gap. Not ignorance. Not negligence. Absent discipline.

AWS has no financial incentive to tell you that their multi-AZ feature does not cover your most critical workload.

This is not malice. It is business model.

The Well-Architected Framework checkbox says: “Are your workloads deployed across multiple Availability Zones?” You tick yes. The review passes. Nobody asks the follow-up question: “Which of those workloads has a stateful component that cannot actually failover, and what is your manual recovery procedure for that component?”

Because that question is uncomfortable, it surfaces architectural decisions that were never made, runbooks that were never written, and drills that were never run. It is easier for everyone in the room to leave multi-AZ ticked and move on.

Until 17:25 PDT on a Thursday in May.

Managed DR is not a product you buy. It is a set of decisions you make before an incident, tested repeatedly, until the decisions are muscle memory.

Explicit RPO and RTO, measured — not estimated. Not “we expect recovery within four hours.” When did you last fail over under realistic load and measure actual recovery time? If the answer is never, your RTO is a fiction.

Warm secondaries for stateful critical paths. The matching engine equivalent in your system — whatever cannot be automated — runs warm in the secondary site. Not cold. Warm means it is receiving replication, its configuration is current, and a human can declare a failover and have it running within a defined, tested window.

Runbooks that have been executed, not just written. A runbook that has never been run under pressure is a hypothesis. DR runbooks must be executed on a schedule, with real load, by the people who will execute them during an actual incident. Quarterly minimum.

Failure decision owned by your team, not your vendor. AWS will tell you when they believe a zone is impaired. They will not tell you to failover. They will not declare your outage for you. The decision to activate DR is yours. That means you need a declared threshold — a specific, pre-agreed condition — at which the on-call engineer calls the incident and starts the runbook without waiting for management approval.

A secondary that has no shared fate with the primary. Different region is better. Different provider is better still. Bare metal in a separate facility breaks every shared dependency — control plane, IAM, DNS, network fabric — completely. It is more work to operate. It is also the only architecture where a regional cloud failure cannot, by definition, take you down.

Your board asks: “Are we covered if AWS goes down?”

You say: “Yes, we’re multi-AZ.”

The outage happens. You’re down for seven hours.

The gap between those two moments is not an AWS problem. AWS delivered exactly what they sold. The gap is the distance between a feature and a discipline — and that distance was never measured, never tested, and never made visible to the people who needed to know it existed.

Multi-AZ is a feature. A good one. It covers the failure modes it was designed to cover and nothing beyond them.

DR is a discipline. It covers the failure modes that matter most — the ones that are rare, severe, and completely outside the vendor’s responsibility to resolve for you.

You need both. You cannot substitute one for the other.

1. List every stateful component in your critical path. For each one, answer: can it automatically failover to another zone or region? If the answer is no, you have a declared single point of failure. Own it.
2. Find your last measured RTO. Not estimated. Measured, from a drill, under load. If you cannot find it, you do not have one.
3. Identify your shared fate dependencies. If your primary and secondary both depend on the same DNS provider, the same IAM endpoint, or the same regional control plane — your secondary does not provide the independence you think it does.
4. Schedule a failover drill. Real load. Real team. Measure the time. Document what broke. Fix it. Repeat next quarter.

The cloud is infrastructure. Infrastructure fails. The question is not whether your vendor’s HA features will protect you — it is whether you have done the engineering work to protect yourself when they don’t.

Coinbase had not. For seven hours, their users could only refresh.

Do not be Coinbase.