Performing a Kubernetes cluster upgrade often feels like changing the engine of a plane while it is mid-flight. For mission-critical applications running on managed services like AWS EKS or Google GKE, even a few minutes of instability can lead to significant revenue loss. The default "in-place" rolling upgrade provided by cloud providers often falls short when dealing with complex stateful sets or legacy applications that do not handle SIGTERM signals gracefully.
To achieve a true zero-downtime Kubernetes cluster upgrade, you must move beyond the automated "Update" button. By implementing a blue/green node group migration strategy combined with strict PodDisruptionBudgets (PDBs), you can ensure that your workloads remain available and performant throughout the entire lifecycle of the version jump. This guide focuses on the architectural patterns required to migrate from older versions (like 1.28) to the latest stable releases (1.29 or 1.30) without dropping a single packet.
TL;DR — Avoid in-place node updates. Provision a new "Green" node group with the target version, use PodDisruptionBudgets to maintain minimum replica counts, and cordoning/draining the "Blue" group to force a controlled migration of workloads.
The Core Concept: Blue/Green Node Migration
💡 Analogy: Think of your cluster upgrade as a relay race. Instead of asking one runner to change their shoes while running (in-place upgrade), you bring a second runner (the Green node group) onto the track. They run alongside the first runner until they successfully hand over the baton (the workload), at which point the first runner can safely exit the track.
A Kubernetes cluster upgrade consists of two distinct phases: the Control Plane update and the Data Plane (worker nodes) update. While managed services like EKS and GKE handle the Control Plane with high availability, the Data Plane is where most downtime occurs. In a standard rolling update, the cloud provider terminates a node and starts a new one. If your application takes 2 minutes to boot but the health check only waits 30 seconds, you experience 90 seconds of "black hole" traffic.
Blue/Green migration solves this by decoupling the node lifecycle from the upgrade process. You create a completely new set of worker nodes running the target Kubernetes version. Because the old nodes still exist, you have an immediate rollback path if the new nodes fail to join the cluster or if the networking CNI plugin encounters issues with the new kernel version. This provides a safety net that in-place updates simply cannot match.
When to Choose Blue/Green Over Rolling Updates
You should adopt a Blue/Green strategy when your service level objective (SLO) requires 99.99% availability or higher. In my experience running production EKS clusters for financial services, the "Managed Node Group" update feature in AWS often triggers simultaneous reboots of too many nodes if the maxUnavailable setting is not perfectly tuned. This can lead to CPU throttling on the remaining nodes as they struggle to handle the suddenly redistributed load.
Metrics-backed adoption criteria include clusters where the "Time to Ready" for a pod exceeds 60 seconds or where applications maintain persistent local state (though rare in K8s, it happens with some legacy caches). If your Prometheus metrics show significant spikes in 5xx errors during previous kubectl drain operations, your application is likely not handling eviction signals correctly, making the controlled, manual pace of Blue/Green migration a necessity.
The Upgrade Architecture
The architecture involves maintaining two identical node groups with different labels. This allows you to use nodeSelector or affinity rules to precisely control which workloads move and when. During the transition, the Kubernetes Scheduler sees a massive increase in available capacity, which prevents the "Pending Pod" syndrome often seen during resource-constrained rolling updates.
[ Control Plane (v1.30) ]
|
+-----------------------------+
| |
[ Blue Node Group (v1.29) ] [ Green Node Group (v1.30) ]
| |
| (Draining...) | (Scaling Up...)
| |
[ Pod A ] ------------------> [ Pod A' ]
[ Pod B ] ------------------> [ Pod B' ]
Data flow remains uninterrupted because the Kubernetes Service and Ingress Controller (like NGINX or ALB Controller) automatically update their endpoint lists as pods transition. By using a PodDisruptionBudget, you prevent the kube-controller-manager from ever taking down more pods than your capacity can handle. For instance, if you have 10 replicas, a PDB with minAvailable: 8 ensures you never drop below 80% capacity during the migration.
Implementation Steps for EKS and GKE
Step 1: Define the PodDisruptionBudget
Before touching the nodes, you must protect your workloads. This YAML ensures that even during a heavy drain, the cluster maintains availability.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: my-web-serviceStep 2: Provision the Green Node Group
In EKS, create a new Managed Node Group with the new AMI version. In GKE, create a new Node Pool. Ensure the new nodes have the same IAM roles and security groups as the old ones. Verify they are Ready using the following command:
kubectl get nodes -L alpha.eksctl.io/nodegroup-name,cloud.google.com/gke-nodepoolStep 3: Cordon and Drain the Blue Group
This is the critical phase. Cordoning prevents new pods from being scheduled on the old nodes. Draining then gracefully evicts existing pods. If you have many nodes, run this in a loop or use a tool like k9s for manual oversight.
# Cordon the old nodes
kubectl get nodes -l version=v1.29 -o name | xargs -I {} kubectl cordon {}
# Gracefully drain (respecting PDBs)
kubectl drain -l version=v1.29 --ignore-daemonsets --delete-emptydir-data --timeout=300s⚠️ Common Mistake: Forgetting to exclude DaemonSets during a drain. Since DaemonSets run on every node, kubectl drain will fail unless you include the --ignore-daemonsets flag, as they cannot be rescheduled to other nodes.
Trade-offs and Decision Criteria
Choosing between Blue/Green and In-place upgrades is a matter of balancing cost against reliability. While Blue/Green is significantly safer, it temporarily doubles your compute costs during the transition period. For massive clusters (1,000+ nodes), this cost can be non-trivial, requiring a "rolling blue/green" where you migrate one subnet at a time.
| Feature | In-Place Rolling | Blue/Green Migration |
|---|---|---|
| Risk Level | High (Potential for capacity drops) | Low (Instant Rollback) |
| Additional Cost | Minimal | Temporary 100% Increase |
| Complexity | Low (Cloud-native automation) | Medium (Requires manual/scripted drain) |
| Downtime | Possible (if PDBs missing) | Near Zero |
If your application uses ReadWriteOnce Persistent Volumes (EBS/PD), Blue/Green can be tricky. The volume must be detached from the Blue node and reattached to the Green node, which inherently causes a few seconds of downtime for that specific pod. In these cases, ensure your application logic handles temporary storage disconnects or use ReadWriteMany storage like EFS or Filestore.
Pro-Tips for High-Scale Clusters
To further harden your Kubernetes cluster upgrade, implement PreStop Hooks. A preStop hook that executes a sleep 15 gives your Ingress controllers and Load Balancers enough time to propagate the "endpoint removal" signal before the pod actually dies. This prevents the race condition where a load balancer sends a request to a pod that has already received its SIGTERM.
Additionally, monitor your CoreDNS performance during the migration. As hundreds of pods restart simultaneously on the Green nodes, CoreDNS will experience a surge in lookup requests. I recommend scaling your CoreDNS replicas 2x before starting the drain to prevent DNS resolution failures, which are a common root cause of "mysterious" upgrade errors.
📌 Key Takeaways
- Always update the Control Plane before the worker nodes.
- Use
PodDisruptionBudgetsto enforce high availability at the application level. - Prefer Blue/Green node pools for a 100% rollback guarantee.
- Include
preStophooks to handle the delay in Load Balancer propagation.
Frequently Asked Questions
Q. How long does an EKS cluster upgrade typically take?
A. The Control Plane update usually takes 10–20 minutes. The worker node migration timing depends on your strategy; a Blue/Green migration of 50 nodes typically takes 15 minutes, whereas an in-place rolling update can take over an hour due to the sequential termination and boot cycles.
Q. What happens if a PodDisruptionBudget (PDB) is too restrictive?
A. If your PDB requires minAvailable: 10 but you only have 10 pods, the kubectl drain command will hang indefinitely. You must ensure that your deployment has enough replicas to satisfy the PDB while one pod is being evicted and rescheduled.
Q. Can I skip versions during a Kubernetes upgrade?
A. No, Kubernetes does not support skipping minor versions (e.g., you cannot go from 1.28 to 1.30 directly). You must upgrade to 1.29 first, ensure stability, and then proceed to 1.30. This applies to both the Control Plane and worker nodes.
Post a Comment