How to fix GKE cluster upgrade failed: timeout waiting for nodes to finish upgrading in Kubernetes
GKE cluster upgrade fails when nodes timeout during upgrade, pod eviction fails, or control plane migration hangs. Clusters revert to original version or get stuck mid-upgrade due to insufficient capacity or blocking workloads.
Quick fix
Fixes GKE cluster upgrade failed: timeout waiting for nodes to finish upgrading
- Step 1
- Step 2
- Step 3
- Step 4
- Step 5
# GKE cluster upgrade failed: timeout waiting for nodes to finish upgradingGKE cluster upgrade failed: timeout waiting for nodes to finish upgrading
What this error means
This error occurs when GKE cannot complete the cluster upgrade process within the allowed timeframe. During upgrades, Google replaces nodes with new versions but must first evict pods and ensure cluster stability. If pods fail to evict, nodes hang during draining, or control plane components cannot migrate, the upgrade times out and rolls back. This leaves your cluster in an inconsistent state.
Typical symptoms
- Cluster upgrade process hangs at 50% completion
- Error message: "Timeout waiting for nodes to upgrade"
- Some nodes report "Ready,SchedulingDisabled" indefinitely
- Pods stuck in pending or evicted state
- Control plane upgrade succeeds but node upgrade fails
- Cluster reverts to original version after timeout
- GKE console shows upgrade status as "In Progress" for hours
Common causes
- PodDisruptionBudget (PDB) preventing pod eviction during drain
- Insufficient cluster capacity to schedule evicted pods
- Local storage volumes preventing pod migration
- Long-running batch jobs refusing to terminate
- Node pool too small to handle rolling restart
- Persistent volume claim conflicts during node drain
- Control plane components taking too long to migrate
How to fix "GKE cluster upgrade failed: timeout waiting for nodes to finish upgrading"
First diagnostic step
Second diagnostic step
Third diagnostic step
Fourth diagnostic step
Fifth diagnostic step
Advanced notes
Additional notes and platform-specific considerations.
Frequently asked questions
- What does "GKE cluster upgrade failed: timeout waiting for nodes to finish upgrad" mean?
- This error occurs when GKE cannot complete the cluster upgrade process within the allowed timeframe. During upgrades, Google replaces nodes with new versions but must first evict pods and ensure cluster stability. If pods fail to evict, nodes hang during draining, or control plane components cannot migrate, the upgrade times out and rolls back.
- How do I fix "GKE cluster upgrade failed: timeout waiting for nodes to finish upgrad"?
- GKE cluster upgrade fails when nodes timeout during upgrade, pod eviction fails, or control plane migration hangs. Clusters revert to original version or get stuck mid-upgrade due to insufficient capacity or blocking workloads.
- What causes "GKE cluster upgrade failed: timeout waiting for nodes to finish upgrad"?
- PodDisruptionBudget (PDB) preventing pod eviction during drain Insufficient cluster capacity to schedule evicted pods Local storage volumes preventing pod migration
- Is the "GKE cluster upgrade failed: timeout waiting for nodes to finish upgrad" error serious?
- Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.
Official resources & further reading
- ๐Kubernetes Official Documentation
- ๐Google Cloud: Upgrading GKE Clusters(Official Docs)
- ๐Google Cloud: GKE Upgrade Troubleshooting(Official Docs)
- ๐ฌStack Overflow: GKE cluster upgrade issues(Stack Overflow)