
Google has detailed a layered debugging effort that traced intermittent Google Kubernetes Engine (GKE) API timeouts to a Linux kernel TCP bug, after roughly 800 developers were blocked from deploying to production. What first looked like a connectivity issue in private cluster access turned out to involve three separate problems across GKE, Private Service Connect, and the Linux networking stack.
How the GKE API timeout problem surfaced
The issue appeared in January, when a customer running multiple GKE clusters with Private Service Connect (PSC) for private control-plane access saw intermittent timeouts as pods tried to reach the Kubernetes API server. According to Google’s account, packet captures showed something unusual: the control plane was replying to new TCP connection attempts with stale ACK packets from connections that had already been closed.
That made the failure harder to pin down. The symptom was an I/O timeout, but it could plausibly have been caused by several different layers in the stack. Google said engineers had to investigate GKE, PSC, networking, host networking, the kernel, and even the Linux upstream community before finding the full chain of causes.
Three bugs, three layers, one failure mode
Google’s investigation identified three interacting root causes. First, there was packet reordering in Private Service Connect. Second, Envoy’s half-close behavior prevented TCP connections from being properly closed. Third, a Linux kernel bug caused certain reordered FIN packets to be discarded.
That combination left some sockets stuck indefinitely in the FIN_WAIT_2 state, creating the conditions for connection failures that were difficult to reproduce consistently. In other words, each layer was behaving in a way that made sense on its own, but together they produced a failure that did not have a single obvious owner.
Why the debugging was so difficult
Google’s write-up emphasized that most network incidents can be fixed at one obvious layer: a firewall rule, a connection pool limit, or an MTU mismatch. This case was different because the same timeout symptom showed up whether engineers inspected the PSC layer, the kernel’s TCP state machine, or Envoy’s connection handling.
That overlap made it easy to chase partial explanations. A fix at one layer could improve the immediate symptom without correcting the underlying state-machine problem, which meant the team had to keep validating packet traces and rollout timing until the pieces lined up.
What Google changed to restore connectivity
The remediation unfolded over several weeks and involved three coordinated fixes. Google first rolled back a Private Service Connect change to restore service. It then corrected proxy protocol handling and added regression tests. Finally, the deeper kernel issue was fixed with a Linux patch authored by Eric Dumazet, formerly a software engineer at Google.
Rohan Karne, managing director at Google Cloud, framed the incident as an example of the less glamorous side of mission-critical platform engineering: working methodically through the obvious suspects and digging into packet captures until the kernel root cause becomes clear.
The kernel fix and where it landed
The Linux patch that addressed the TCP issue was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 long-term support branches. Google said the fix reached GKE customers in version 1.33.5 and later.
Justin Garrison, field CTO at Sidero, noted that the underlying bug had already been fixed upstream and that GKE was running a kernel from three years ago, version 6.1, which required backporting. That detail helps explain why the issue surfaced in a managed platform even though the kernel problem itself was not new.
What the case means for platform and SRE teams
The broader lesson from the incident is that failures in modern infrastructure often cross boundaries between application networking, proxies, cloud networking products, and the operating system kernel. Google’s account shows how a seemingly narrow API timeout can actually be the result of multiple independent behaviors reinforcing one another.
For site reliability teams, the case also highlights the importance of correlating packet-level evidence with rollout events and protocol state. A fix that restores service is not necessarily the same as a fix that restores correctness in the TCP state machine.
- Private Service Connect packet reordering contributed to connection instability.
- Envoy’s half-close handling left TCP connections from closing cleanly.
- A Linux kernel bug discarded reordered FIN packets, leaving sockets in FIN_WAIT_2.
- The final fix was a kernel patch, not just a cloud configuration change.
Why Google is treating this as more than a one-off outage
Google’s description of the incident suggests the company sees it as a useful example of how complex infrastructure bugs can span layers that teams normally debug separately. Guangba Yu, a postdoctoral researcher at the Chinese University of Hong Kong, pointed to the value of observing paths, protocol state, and rollout events together, while also noting that the anonymized case does not provide a full population failure rate or explain every API timeout.
That caveat matters. The report does not claim every timeout in GKE came from the same chain of bugs. Instead, it shows how one difficult incident was untangled by combining packet captures, state-machine analysis, and infrastructure change history.
Source: Original report
Was this helpful?
Explore more: DevOps Services More Cloud & DevOps Tech News
Last Modified: November 21, 2026 at 10:33 pm
1 views

