Google's Backup Generators Never Started in the London Outage That Hit Lloyds and HSBC

A fault in one utility cable cut power to a Google Cloud data center in London on Oct. 20 and disrupted services across the London region for more than six hours, Google said in its incident report.

A workload that runs in a single zone is built to go down with that zone, and Google’s account shows the damage did not stop there: workloads failing over to the other zones overloaded them.

Why did the generators stay off?

The site runs on two utility feeds, A and B, with diesel generators behind them, all started by a pair of programmable logic controllers. Engineers had opened the breaker on feed B for repairs after a generator test turned up problems, so feed A carried the whole site.

When a cable fault knocked out feed A at 6:53 a.m. London time, the controllers were still reading voltage on the live side of feed B’s open breaker. They took that reading as proof that utility power was available and never ordered the generators to start. The same reading made them block a manual transfer attempted remotely.

Batteries rated for about five minutes held the load until servers lost power at 7:00 a.m. Staff then moved the site onto a standby utility feed, and power began returning at 7:54 a.m.

Who felt it?

Revolut, Lloyds Banking Group and HSBC customers lost access to some services, according to published reports. Revolut told customers their money and data were safe, and HSBC later said its services had returned to normal. Royal Mail, Rightmove and NHS England drew spikes on outage trackers, and the video system for court hearings in England and Wales went down.

Engineers needed 6 hours and 8 minutes to clear the main impact. Virtual machines, Kubernetes clusters and databases without a standby copy went dark in the failed zone. The rush of workloads failing over then overloaded capacity elsewhere in the region, so workloads in healthy zones saw intermittent errors as well. Google did not say how many customers or workloads the outage reached.

Regulators are asking what comes next. The Financial Conduct Authority said it is working with Google and the affected firms. Treasury Select Committee chair Meg Hillier wrote to Bank of England Governor Andrew Bailey and Nikhil Rathi of the FCA asking how they will use the powers they received in July to oversee cloud providers serving UK firms.

What is Google changing?

Google reset and retested the controllers and the generators, then froze planned maintenance at its other sites built the same way. It is also checking controller settings at those sites, a review due Nov. 15.

Google’s next test, a full generator transfer with both utility feeds restored, is due Nov. 17.