6 Days Offline: What British Airways 2017 Outage Teaches About Failover Testing
Picture this. You build a backup system for one reason: so that when something breaks, the backup takes over and nobody notices. That's the whole promise of failover. It's insurance you hope to never use.
British Airways had that insurance. Two data centers, not one. That's the standard setup for exactly this kind of protection. On a Saturday morning in May 2017, they needed it. And it didn't work.
What happened
It started small. An engineer was doing maintenance at one of BA's data centers near Heathrow. At some point, the power supply got disconnected, then reconnected. That reconnection sent a power surge through the system.
That part alone shouldn't have grounded a single flight. Power problems happen. That's the entire reason a second data center exists, to pick up the load while the first one gets fixed.
But the second site didn't pick anything up. Whatever was supposed to happen automatically didn't happen. Both centers went down together, and BA's check-in systems, baggage systems, and website all failed at once.
The direct chaos, flights cancelled, planes grounded at Heathrow and Gatwick, lasted about three days. But the mess didn't really end there. Stranded bags, rebooked passengers, and refund claims kept the airline cleaning up for close to a week. Somewhere around 75,000 people had their travel plans wrecked. The final compensation bill came in at roughly £58 million.
The part that should worry every engineering team
Here's the detail that matters most, and it has nothing to do with the power surge itself.
A power surge is a normal, expected kind of failure. Data centers plan for it. What isn't normal is a failover system failing at the exact moment it's needed. That's not bad luck. That usually means the failover was never really tested under real conditions, only assumed to work because it looked fine on paper.
This is the gap a lot of teams have without realizing it. They build a backup system, run a quick check when it's first set up, and then never touch it again. Months or years go by. The main system changes. The backup doesn't get updated to match. Nobody notices, because nothing has forced a real failover to happen in the meantime.
Then one day something breaks for real, and the backup that was supposed to save the day turns out to be just as broken as everything else, just quietly, where nobody could see it.
How do you test a failover system properly
You don't just check that a backup exists. You force it to actually take over, on purpose, and watch what happens.
That means unplugging the primary system in a controlled way and confirming the backup handles real traffic, not just a health check ping. It means testing this on a schedule, not just once after setup. And it means testing the handoff itself, the actual moment of switching over, because that transition is usually where things break, not the backup system sitting idle.
Why do backup systems fail when they're needed
Mostly because they get tested once and then left alone. A backup that passed a test two years ago isn't proof of anything today. The main system has probably changed since then. New services got added. Configurations drifted. Nobody went back and re-tested the failover path against the current setup, because that kind of testing rarely feels urgent until the day it's the only thing standing between you and a very bad morning.
There's also a quieter risk here worth naming. BA's data center was managed by an outside contractor, and part of the disagreement afterward was about who was responsible for what. When you depend on a third party for infrastructure, your failover plan is only as good as your visibility into what they're actually doing, which is its own kind of risk worth testing for on its own.
What is disaster recovery testing in software
It's the practice of regularly proving your systems can survive a real failure, not just checking that a recovery plan exists on a document somewhere. Good disaster recovery testing simulates an actual outage, including the failover, the data integrity afterward, and how long it takes a system to be usable again. It's different from a backup check. A backup check confirms data exists somewhere. Disaster recovery testing confirms you can actually get back up and running with it.
How often should you test failover procedures
Often enough that a recent change to your main system can't quietly slip past your backup without anyone noticing. For most teams that means testing after any significant infrastructure change, plus a full scheduled test at least a few times a year regardless of whether anything changed. If your last real failover test was more than six months ago, you don't actually know if it still works. You're just hoping it does, the same way BA probably was.
The takeaway
BA didn't lose to a power surge. Plenty of companies survive those every year without anyone outside IT ever hearing about it. BA lost because the system built specifically to survive that exact situation didn't survive it.
This isn't just a story about data centers. It's the same pattern behind risk-based testing: the parts of a system nobody watches closely are usually the parts that fail hardest when they finally get called on. It's also close cousin to what happened during NATS' 2023 flight-planning failure and to Knight Capital's regression failure, two more cases where the failure lived in a part of the system that everyone assumed was fine.
If there's one honest question worth asking your own team after reading this: when was the last time you watched your failover take over, instead of just trusting that it would.