Google Cloud, June 2, 2019: One Maintenance Event Took the Network's Brain Offline
On Sunday, June 2, 2019, a routine maintenance event started in one Google physical location — and, because of two normally harmless misconfigurations and one software bug, it switched off the network control plane in several locations at once. Google's network kept running on stored instructions for a few minutes, then routes were withdrawn, capacity collapsed, and six US-facing regions spent between 3 hours 19 minutes and 4 hours 25 minutes congested. Recovery then took hours for a second reason: the control plane's configuration had been lost and had to be rebuilt by hand. This is the documented story, from Google's own incident report #19009.
The short answer
On Sunday, June 2, 2019, at 11:45 a.m. US/Pacific, a maintenance event — planned work on datacenter infrastructure, the kind Google runs constantly somewhere in the world — started in one physical location. It should have touched only that location. Instead, the automation behind it descheduled (switched off and removed from service) the jobs that run Google's network control plane — the software that tells the network where traffic should go — in several physical locations at once. Google's own incident report, Incident #19009 (issue summary dated June 6, 2019), says three things had to combine for that to happen: the control-plane jobs were configured to be stopped during maintenance instead of moved, the cluster managers running them were marked eligible for this relatively rare type of maintenance event, and the maintenance software itself had a bug that let it deschedule multiple independent clusters at once, even in different places. The network kept running for several minutes on its last known instructions — a design called fail-static — then BGP routing (the system networks use to tell each other which destinations they can reach) between specific impacted locations was withdrawn, capacity fell sharply, and projects in six regions hit heavy packet loss and congestion for between 3 hours 19 minutes and 4 hours 25 minutes, depending on the region. Engineers knew within two minutes. Full service did not resume until 16:10.
The company: what Google did
Google runs one of the largest networks on Earth. Alongside its own services — Search, YouTube, Gmail and the rest of what the report calls G Suite — it sells that infrastructure to other companies as Google Cloud: rented computers (Compute Engine), storage, databases, private connections into Google's network, and dozens of other services, organised into named regions such as us-east1 and us-west2. When a customer puts a business on Google Cloud, Google's network is not a supplier in the background. It is the road the business runs on.
Two pieces of machinery matter in this story, and the report explains both plainly. The first is the cluster management software. Inside any single physical location, Google's machines are divided into multiple logical clusters — groups of machines managed as a unit — each with its own dedicated cluster manager, so that one manager failing changes nothing. The cluster managers also automate maintenance events: power work, network upgrades, and similar jobs. Every job they run carries a label saying what should happen to it during maintenance: normally, either move it to a machine that is not under maintenance, or stop it and restart it afterwards.
The second is the network control plane. Think of it as the network's brain: the software that works out routes and tells the network hardware where to send traffic. The hardware that actually forwards packets is the data plane. Google deliberately runs the control plane under several different cluster managers in each location, so losing any one of them has, in the report's words, no impact on network capacity. Why this matters: this outage is not a story about one server failing. It is a story about the layer that was supposed to make single failures irrelevant — and about the automation that found a way to fail all of it at once.
The problem: three quiet faults, waiting for each other
The report is direct that this was, in its words, a major outage in both scope and duration, and that multiple failures combined to amplify the impact. Here are the three, in the report's own order — because the lesson is in the combination, not in any one of them.
First misconfiguration: the brain was set to stop, not move. The network control plane jobs, and the infrastructure supporting them, were configured to be stopped in the face of a maintenance event. For an ordinary job, stop-and-restart-later is a reasonable setting. For the software that runs the network, it means a maintenance event is allowed to take the brain offline and trust that it will come back.
Second misconfiguration: every copy was eligible for the same rare event. The multiple instances of cluster management software running the network control plane were marked as eligible for inclusion in a particular, relatively rare maintenance event type. Redundancy only protects you if the copies cannot all be selected by the same event. Here, they could.
The bug: one event could reach across locations. The software that initiates maintenance events had a specific bug allowing it to deschedule multiple independent software clusters at once — crucially, even if those clusters were in different physical locations. A maintenance event is supposed to be local, the way roadworks close one street, not every street a crew has ever worked on. The bug removed that boundary.
An analogy: imagine a building where the fire alarm in one room is wired, by mistake, to cut power to the control rooms on every floor — and the control-room staff have standing instructions to go home when the power cuts, instead of moving to a backup room. Each wiring choice looks harmless on its own, for months. The day the alarm is tested, all three are discovered in the same minute.
What this shows: the three-part combination from Google's report, in its own order. The outage needed all three at once — which is also why fixing any one of them properly breaks the chain.
What happened on June 2, 2019
All times are US/Pacific, as given in Google's report. Read the end times carefully: the dashboard header lists the incident as ending at 15:40, the report's narrative says full service resumed at 16:10, and some VPN gateways needed until 16:30. Those are per-service recovery times, not a contradiction to smooth over — this page keeps them separate, as the report does.
What this shows: the shape of the day — a two-minute detection, a 76-minute path to root cause, then a recovery measured in hours because the configuration itself had to be rebuilt.
11:45 — the event fires. A maintenance event starts in a single physical location. The automation builds its list of jobs to deschedule in that location — a list that includes the logical clusters running network control jobs. Those logical clusters, the report explains, also included network control jobs in other physical locations. The automation deschedules each in-scope cluster, taking down the network control jobs and their supporting infrastructure in multiple locations at once.
11:47–11:49 — fail-static runs out. Google designs for exactly this fear: even with a highly resilient control plane, the network is built to fail static — keep forwarding on its last known configuration, with no brain present, for a period of time. It works, for several minutes. Then BGP routing between specific impacted physical locations is withdrawn, network capacity drops significantly, some regions become inaccessible, and end-user impact appears in the 11:47–11:49 window. Engineers were alerted two minutes after the event began and engaged the protocols for the most significant production incidents.
The debugging fights the outage itself. The report is candid here: debugging was significantly hampered because the engineers' tools had to compete for the same congested network. Google had backup plans — including engineers travelling to secure facilities built to withstand catastrophic failures, and lowering the priority of less critical traffic to ease congestion — but using those backups added time. The scale of the outage, and the collateral damage to tooling, also made it hard at first to state precisely what was impacted and communicate it accurately.
13:01 — root cause, and the automation is stopped. By 13:01 the incident had been root-caused and engineers halted the maintenance automation. Then recovery hit its own second problem.
What this shows: why the outage did not start at 11:45 sharp. The stored-routes design absorbed the loss of the control plane briefly — then the withdrawals arrived, and the grace period ended everywhere at once.
14:03 to 16:10 — the rebuild. With all instances of the network control plane descheduled in several locations, its configuration data had been lost and needed to be rebuilt and redistributed — during a major network event, across multiple locations. The new configuration began rolling out at 14:03. In parallel, service teams redirected traffic away from the affected regions where they could. As each location's control plane was rescheduled and its configuration recreated, capacity returned: recovery started at 15:19, and the report's narrative records full service resumed at 16:10.
The damage
Where it hit. Customers may have seen increased latency, intermittent errors and connectivity loss in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1. Just as telling is where it did not: us-west1, and all European and Asian regions, did not experience regional network congestion. The blast radius was large — and bounded by geography, which is part of why traffic redirection to unaffected regions worked as a mitigation.
How deep it went, service by service (all figures from the report's detailed impact section). Compute Engine instances in us-east4, us-west2, northamerica-northeast1 and southamerica-east1 were inaccessible for the duration. Instance-to-instance packet loss in us-east1 reached up to 33 percent. Cloud Interconnect — the private links companies use to join their own networks to Google's — saw packet loss from 10 to 100 percent in affected regions, and 50 to 100 percent in the four hardest-hit regions from 11:45 to 16:10, with some BGP sessions going down. Cloud VPN gateways in those four regions were unreachable. Cloud Storage error rates tell the regional story in one line: us-west2 96.2 percent, southamerica-east1 79.3 percent, us-east4 62.4 percent, northamerica-northeast1 43.4 percent — against 0.7 percent in us-central1 and 1.2 percent in us-west1. Cloud Spanner in three regions was unavailable from 11:48 to 15:44, BigQuery job submissions briefly hit a 27 percent error rate in the first fifteen minutes, and Cloud Pub/Sub saw a 27 percent global drop in successful requests. The affected list also included App Engine, Cloud Endpoints, Cloud Console, Stackdriver Metrics and G Suite services in those regions — and, as the report notes, several non-Cloud Google services that could not fully redirect users elsewhere.
What the report does not do is put a dollar figure on the outage, and neither does this page. The measurable damage in the primary source is duration, reach and error rates — plus the fact that affected customers were directed to request SLA credits, the service-level agreement refunds cloud providers owe when availability promises are missed.
How they recovered
Step one: stop the cause. Once the failure was root-caused at 13:01, engineers halted the automation software responsible for the maintenance event. No recovery could outrun an automation that was still allowed to deschedule.
Step two: work around the broken network to fix it. Because the debugging tools themselves needed the congested network, engineers fell back on the backup plans the report describes: travelling to secure facilities designed for catastrophic failures, and deprioritising less critical traffic classes to reduce congestion. Service teams in parallel steered traffic away from affected regions toward regions still serving normally.
Step three: rebuild what was lost. The control plane could not simply be restarted: its configuration had been lost with the descheduling. Engineers rebuilt and redistributed that configuration for multiple locations, with rollout beginning at 14:03 — the single longest pole in the recovery, and the reason the gap between root cause (13:01) and capacity returning (15:19) is measured in hours.
Step four: bring capacity back location by location. As each location's control plane was rescheduled with recreated configuration, capacity returned from 15:19. Most services recovered with the network; the stragglers are named in the report — us-east4's VPN gateways at 15:40, and the VPN gateways in us-west2, northamerica-northeast1 and southamerica-east1 at 16:30, after extra work to move their control plane out of a fail-safe state it had entered to protect existing gateways from incorrect changes.
The fix: technology and controls that came out of it
The report's Prevention and Follow-Up section commits Google to fixes aimed, in its words, at eliminating not just these specific issues but the entire class of similar problems. Each fix maps to one of the faults above:
Stop the automation, then cage it. The descheduling automation was halted immediately, to be re-enabled only once safeguards exist to prevent descheduling jobs in multiple physical locations concurrently. That is the bug's fix at the source: a maintenance event's reach is limited before it starts, not discovered after.
Make the cluster manager refuse, whoever asks. Google committed to hardening the cluster management software so it rejects multi-location descheduling requests regardless of origin — defence in depth, so that even if some other tool develops the same bug, the request dies at the door. This is the difference between fixing the caller and fixing the system: only the second survives the next buggy caller.
The control plane must refuse this maintenance type. The network control plane software and its supporting infrastructure were to be reconfigured to handle maintenance correctly by rejecting requests of the type implicated in this incident — ending the stop-instead-of-move setting and the eligibility that made the first two faults dangerous.
Persist the configuration — so recovery stops being a rebuild. The control plane in each location was to be modified to persist its configuration, so it no longer needs to be rebuilt and redistributed if all its jobs are descheduled. The report quantifies the prize: this alone was expected to reduce recovery time by an order of magnitude. The outage's length was set less by the failure than by the rebuild; this fix attacks the length directly.
Hold fail-static longer. The network was to be updated to continue in fail-static mode for a longer period after losing its control plane, giving recovery an adequate window with no user impact. The June 2 grace period was several minutes; the incident needed much more.
Test the emergency path against this exact failure. Emergency tooling and procedures — including customer communication tooling — were to be reviewed, updated and tested for robustness against network failures of this kind, and continuous disaster-recovery testing extended to include this and other similarly catastrophic failures. The tools that failed the engineers were the ones that shared the network they were meant to repair; the fix is an out-of-band path that is exercised before it is needed.
What this means for you
Hyperscaler engineers
This is your case. Three transferable checks fall straight out of the report. First: audit what your automation is allowed to do, not just what it usually does — the bug mattered because the request it made was accepted; a cluster manager that rejects cross-location descheduling regardless of origin makes a whole class of caller bugs harmless. Second: list every job labelled stop-on-maintenance and ask, for each, what the world looks like if it stops everywhere at once — the control plane's setting was benign per-location and catastrophic in aggregate. Third: measure your fail-static window honestly and compare it with your rebuild time. Google's gap was minutes of grace against hours of rebuild, closed afterwards by persisting configuration. If your control plane lost its configuration today, which number would you be quoting in your own incident report?
Enterprise network engineers
Your scale is smaller; the shape is identical. Your version is the maintenance script that reboots a switch stack, the change window that touches both members of a redundant pair, the automation credential that can reach every site. The rules that would have broken this chain all fit an enterprise network: never let one job action both sides of a redundant pair, keep an out-of-band management path that does not ride the network it manages (Google's engineers ended up travelling to secure facilities — your equivalent is the console server and the management network you test quarterly, or the drive you take during the outage), and preserve device configuration somewhere the failure cannot reach, so recovery is a restore, not a rebuild from memory. And when a vendor sells you redundancy, ask the Google question: can a single event select all the copies at once?
HFT / low-latency engineers
Your control planes are smaller and faster, but the failure grammar is the same: a configuration or maintenance action with a wider reach than its author believed, applied to components that were supposed to be independent. The fail-static lesson cuts both ways in low-latency design — a trading or feed system that keeps running on stale state buys you minutes and charges you for them if the stale state is wrong, so the honest design question is the one Google had to answer afterwards: how long should it hold, and what, exactly, triggers it to stop? The other direct transfer is blast-radius testing: Google extended its disaster-recovery testing to include this failure class. Rehearsing the loss of the whole control layer — not one node of it — is the drill this incident argues for.
Try it yourself
Three free Packet Path tools build the habits this case rewards. Start with the BGP Toolkit: look up a real prefix and follow its origin and route — BGP withdrawal is the mechanism that turned a control-plane problem into lost capacity here, and seeing live routes makes that concrete. Then run two configurations through the Network Config Verifier — the misconfiguration half of this outage was settings that differed from what the design intended, and side-by-side comparison is how you catch a stop-versus-move style drift before an event does. Finally, paste a set of device configs into the Config to Topology Mapper and look at the dependency picture it draws: the question Google's report forces — what else does this one component take with it? — is much easier to answer from a map than from memory.
Sources
Primary source — Google Cloud Status Dashboard, Google Cloud Networking Incident #19009, issue summary dated June 6, 2019. Every fact, time, region, figure and prevention commitment on this page is taken from this report, including: the issue summary (regions, durations of 3 hours 19 minutes to 4 hours 25 minutes, affected services); the root cause (the two misconfigurations and the descheduling bug, the 11:45 event, fail-static behaviour, BGP withdrawal, impact from 11:47–11:49, root cause at 13:01, configuration rebuild from 14:03, capacity recovery from 15:19 and full service at 16:10); the prevention commitments; and the detailed per-service impact figures (Compute Engine, Interconnect, VPN, Storage, Spanner, BigQuery and Pub/Sub). Full text: https://status.cloud.google.com/incident/cloud-networking/19009
A note on end times, kept as the source presents them: the dashboard header records the incident as beginning 11:45 and ending 15:40 US/Pacific on June 2, 2019, while the report narrative records full service resumed at 16:10 and the detailed section records the remaining VPN gateways recovering at 16:30. This page presents them as per-service recovery times rather than choosing one. Google also published a shorter summary on June 4, 2019, before this full report; where the two differ in framing, this page follows the June 6 incident report, and figures that appear only in the June 4 summary are not used here.
- No single fault caused this outage: two normally-benign misconfigurations plus one bug in the maintenance automation had to line up before a local event could deschedule the network control plane in multiple physical locations.
- Fail-static bought minutes, not hours: the network ran without its control plane for several minutes, then BGP routing between specific impacted locations was withdrawn and capacity fell sharply — user impact began 11:47–11:49, two to four minutes after the 11:45 event.
- Recovery was slow for a separate reason: with every control-plane instance descheduled in several locations, its configuration data was lost and had to be rebuilt and redistributed — new configuration only began rolling out at 14:03, capacity from 15:19, full service at 16:10.
- The fixes Google committed to attack the class, not just the incident: reject multi-location descheduling in the cluster manager itself, make the control plane refuse this maintenance type, persist its configuration, and hold fail-static longer.