Google engineer unplugged every fiber they could see and – surprise! – took down a chunk of the G-Cloud
This sounds like a RTFM error at hyperscale
Google Cloud has revealed the embarrassing reason for a partial outage at its us-central1-b region this week: An engineer unplugged it.
The incident spanned 07:41 to 11:52 PT on September 1, and saw a portion of the G-Cloud region experience “severe network degradation and resource isolation.”
“Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss,” states Google’s service health report.
The preliminary root cause on that page explains that Google designs its cloudy datacenters so they have “redundancy across multiple routing devices.”
“The system is designed to be resilient to all single device or fiber-path failures, and most double-or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power...
Copyright of this story solely belongs to www.theregister.com. To see the full text click HERE
