When utility power failed at Google's europe-west2-b zone in London on the morning of 20 October, the UPS batteries did what they were meant to do and carried the load for about five minutes. The generators never received the order to start. Google published its updated incident report on 4 November and The Register wrote it up the next day: the data halls were without power for well over an hour, and it took a little more than six hours before most of Google's cloud services in the zone were healthy again.
A 2N Site Running On One Feeder
Google describes 2N utility power with two independent feeders, A and B, a standby utility connection for transferring the whole campus by hand, and a central high-voltage generator bank controlled by a dual-redundant PLC. In practice it was running on one leg. A quarterly generator load test on 14 October had turned up problems, the feeder B breaker was opened for repairs, and feeder A carried the entire critical load.
At 22:53 Pacific time on 19 October, 06:53 in London, an underground cable of the power supplier failed and took feeder A down. Then the controls let the site down. According to Google, the undervoltage sensing inputs of the PLC were wired on the line side of the feeder breakers. Feeder B was open at its breaker but still live upstream of it, so both PLCs, which run identical logic, saw healthy voltage and concluded that a good utility source was available. The logic did not check breaker position together with voltage, and no start command went out.
Staff on site tried a remote load transfer through the PLC, which refused because of the same live reading on feeder B. They then moved the load to the standby utility connection, but the batteries ran out during the changeover. Power returned to the data halls at 00:20 Pacific time, after the generator automation controllers had been reset and tested.
Forty Products And Some Slow Disks
Zonal resources in europe-west2-b, from Compute Engine virtual machines to zonal disks and databases without high availability, became unreachable. Some regional services did not fail over cleanly, and storage took a long time to recover after the unclean shutdown; Google's update lists about 40 products in the region as affected. The Register reports that customers including Lloyds, HSBC, Royal Mail and Revolut saw interruptions.
The fixes Google lists on the facility side are specific. The PLC was reconfigured on 20 and 26 October, and full load transfer tests with automatic generator start were run on the same days. A code review with the PLC vendor, a check of similar sites and repeat tests with both feeders back are scheduled for November. Emergency procedures for losing the active utility source have been rewritten and are now practised twice per shift, planned maintenance at sites with the same topology is frozen, and a second Google team is to review PLC code within a quarter after each facility handover.
The Test Nobody Writes Into The Schedule
Nothing in the report points at the engines. The generator plant was there, and once it was told to run, it ran. What failed was the chain that tells it to start, in a state the site was only in because of maintenance. Most acceptance tests open the incoming supply with everything else in its normal position. Fewer repeat the exercise with one feeder isolated, a breaker racked out or a transfer scheme left in manual, which is how many sites spend a good part of their maintenance weeks. A question worth adding to any commissioning plan: for each sensing point, which side of which breaker is it on, and what does the controller conclude when that breaker is open?
The other lesson is time. Five minutes of battery leaves no room for a manual workaround that has never been rehearsed, and Google's two drills per shift say as much. The generator side of these sites is on our data centers page; a very different approach to backup came up in Microsoft's Atlanta site without generators.
Sources: The Register, Google Cloud Service Health (incident report)
