Rollouts, scaling and debugging

A Deployment's promise is that a new version reaches production without a moment when nothing is serving, that the number of pods follows the load, and that when something is wrong the evidence is one command away. Each promise has settings that make it true and defaults that make it false.

#Rolling updates

Change the image tag and the Deployment's controller rolls: it creates pods from the new ReplicaSet and removes pods from the old one, bounded by two numbers:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1          # how many extra pods may exist during the roll
    maxUnavailable: 0    # how many may be missing: zero means capacity never drops
minReadySeconds: 10      # a new pod counts as ready only after staying ready this long

With maxUnavailable: 0 a new pod must pass its readiness probe before an old one is removed, so the roll is only as safe as the probe — a readiness endpoint that returns 200 before the application can serve turns a rolling update into an outage with a green dashboard. minReadySeconds catches the pod that is ready for two seconds and then crashes. And progressDeadlineSeconds (default 600) marks a roll that is not progressing as failed, which is what alerts and kubectl rollout status report.

kubectl rollout undo deployment/orders-api scales the previous ReplicaSet back up — a rollback in seconds, with the old image, and it is the reason to keep image tags immutable: :latest on both ReplicaSets rolls back to the same thing. This repository's deploy script does the same dance without a cluster — a timestamped release directory, a health check, then the current symlink and an nginx reload, with the last five releases kept for --rollback — and the deployment-strategies lesson in the CI/CD course puts both in the family they belong to.

A Recreate strategy stops everything and then starts the new version: downtime, but the only correct choice when two versions cannot coexist (a schema change without expand-contract, a singleton consumer). And a rolling update is not a canary: every pod is either old or new, and the fraction of traffic on the new version is whatever the pod ratio happens to be. Canaries with traffic weights need a service mesh or an ingress that supports them.

#Scaling: the HPA and what it measures

The HorizontalPodAutoscaler adjusts replicas between a floor and a ceiling to hold a metric at a target:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: orders-api }
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
  behavior:
    scaleDown: { stabilizationWindowSeconds: 300 }

CPU utilisation is a percentage of the request, which is one more reason the request must be honest. It is the right signal for CPU-bound work and the wrong one for an I/O-bound service whose CPU stays flat while its latency climbs; for those, a custom metric through the metrics adapter — requests per second per pod from Micrometer, queue depth, Kafka consumer lag with KEDA — is the signal that actually tracks load. The stabilisation window stops the flapping where a burst scales up, the pods share the load, CPU drops, and the HPA scales down into the next burst.

Three things the HPA does not do. It does not scale faster than a pod starts, so a Spring Boot service with a 45-second start needs headroom in minReplicas, not a lower target. It does not add nodes — the cluster autoscaler does that when pods are unschedulable, on a delay of minutes. And it does not scale a StatefulSet database; that is the SQL course's replication lesson, not a replica count. The VerticalPodAutoscaler is the other axis, adjusting requests, and it is mostly useful in recommendation mode as a way to find out what a service really uses.

#PodDisruptionBudgets: surviving the cluster's own maintenance

A node drain — for an upgrade, a spot instance reclaim, a scale-down — evicts every pod on it. With three replicas that happen to sit on one node, a drain takes all three at once. A PodDisruptionBudget says how many may be missing during voluntary disruptions:

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  selector: { matchLabels: { app: orders-api } }
  minAvailable: 2

The drain then waits for replacements before taking the next pod. Pair it with a topology spread constraint or pod anti-affinity so the replicas are on different nodes in the first place; a PDB on three pods on one node protects you from the drain and not from the node dying.

#Debugging: the four commands

When a pod is not doing what it should, the order that finds most problems:

1. kubectl get pods — the status column is the first clue. Pending is scheduling (no node has the requested resources, or a volume cannot attach). ImagePullBackOff is the registry: a wrong tag, a missing pull secret. CrashLoopBackOff is the process exiting repeatedly; the delay between restarts grows to five minutes. Running with 0/1 ready is the readiness probe failing.
2. kubectl describe pod <name> — the Events at the bottom say why: FailedScheduling: insufficient memory, Liveness probe failed: HTTP 503, Back-off restarting failed container, and the Last State block with OOMKilled.
3. kubectl logs <name> --previous — the logs of the container that just died, which is what you need in a crash loop; without --previous you get the new, empty one. -c picks a container in a multi-container pod; -f follows.
4. kubectl exec -it <name> -- sh for a look inside, if the image has a shell — and kubectl debug with an ephemeral container for the Distroless images that do not. From there, the JVM lessons apply: jcmd, a thread dump, curl localhost:8080/actuator/health to see what the probe sees.

kubectl port-forward pod/<name> 8080:8080 reaches a pod from your laptop without exposing it, which is how you read its Actuator endpoints or its Prometheus metrics directly. And kubectl get events --sort-by=.lastTimestamp across the namespace is the timeline when it is not clear which object to describe.

#The failures you will actually meet

• CrashLoopBackOff with a clean log — the process exits before logging: a missing environment variable read at startup, a wrong ENTRYPOINT, a JVM flag the image's Java does not know. --previous and the exit code in describe are the evidence.
• Ready pods, no traffic — the Service's selector does not match the pod's labels (kubectl get endpoints <service> is empty), or the targetPort is not the port the container listens on.
• A roll that never finishes — the readiness probe fails on the new version and maxUnavailable: 0 refuses to remove the old; rollout status says so, and the fix is to read the new pod's logs, not to loosen the probe.
• Pending forever — a request larger than any node, or a PDB blocking the drain that would free room. describe names it.
• Working on one pod, failing on another — a config difference between environments, or a node-local problem (disk full, a bad DNS resolver); -o wide shows which node.