So how did a power failure in one zone disrupt services across the region?

Transcript

1. On Tuesday morning in Britain, Revolut's card payments,
2. transfers and cash withdrawals were disrupted for hours.
3. Revolut told the BBC it was the result of an outage at Google's cloud in London.
4. A cloud is rented computers in data centres.
5. Google splits its London cloud into 3 zones, in separate data centres.
6. It says it designs zones
7. so a power, cooling or network failure in one is unlikely to hit the others.
8. Its database Spanner, Google says, keeps working if one zone fails.
9. At 7:08 on Tuesday morning, a power problem caused an outage in zone B.
10. Google later said it happened at a third-party facility.
11. But the trouble didn't stay in zone B.
12. Google's own status page warned of errors across the London region,
13. and listed 38 of its products as affected.
14. Revolut told customers about the problem at 7:35.
15. Google's first public update came 2 hours later.
16. It named the cause just after noon.
17. Some Lloyds, Halifax and HSBC customers also reported online banking problems.
18. Lloyds and HSBC said other organisations were affected, without naming Google.
19. Google closed the outage at 3 pm, after 7 hours and 52 minutes.
20. Revolut said its services were fully back that evening.
21. So how did a power failure in one zone disrupt services across the region?
22. Google hasn't explained it yet.
23. It has promised an analysis.
24. It has explained failures like this before.
25. July 2022, this same London region.
26. In a heatwave, the cooling failed in one zone.
27. At the start, Google accidentally routed its own internal traffic away
28. from all 3 zones, not just that one.
29. By Google's own record, that outage lasted almost 39 hours.
30. April 2023, Paris.
31. A water leak led to a fire that shut one data centre building.
32. Google found that copies of Spanner,
33. the database that runs most of the region's controls,
34. hadn't been spread correctly across the 3 buildings.
35. Both times, one zone failed, and the reason it spread was inside Google.
36. Tuesday's answer could be different.
37. And the banks?
38. British rules say a bank must keep each of its most important services within a limit,
39. even when a supplier fails.
40. Each bank sets that limit itself,
41. including a time limit before the disruption causes intolerable harm.
42. Whether Tuesday broke any of those limits hasn't been publicly established.
43. Parliament's Treasury Committee has asked regulators what caused this,
44. and how many firms were affected.
45. The first answers are due on 4 November.
46. 2 answers are owed.
47. Google's, on how a failure in one zone disrupted the region.
48. And the banks', on whether they stayed within their own limits.