Configuration, probes and resources

A pod template says what image to run. Three other things decide whether the service actually works in the cluster: where its configuration and secrets come from, how the cluster learns whether it is alive and ready, and how much memory and CPU it is allowed — the last of which is the setting behind almost every "the pod keeps restarting" ticket.

#ConfigMaps and Secrets

A ConfigMap is key-value configuration as a cluster object; a Secret is the same with base64 encoding and, if the cluster is set up for it, encryption at rest and tighter access control. Either reaches the container as environment variables or as files:

env:
  - { name: SPRING_PROFILES_ACTIVE, valueFrom: { configMapKeyRef: { name: orders-config, key: profile } } }
  - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: orders-db, key: url } } }
volumeMounts:
  - { name: secrets, mountPath: /run/secrets, readOnly: true }
volumes:
  - { name: secrets, secret: { secretName: orders-db } }

Environment variables are read once at start; a mounted file updates in place when the ConfigMap changes (within a minute), which Spring can pick up with spring.config.import=configtree:/run/secrets/ and a refresh — and which is the argument for mounting secrets rather than injecting them, along with the Docker course's point that an environment variable is visible to every process and every kubectl describe. Base64 is not encryption: a Secret object committed to a Git repository is a plaintext credential with extra steps. The pattern that works is a secrets manager (Vault, the cloud's) synced into cluster Secrets by an operator, so the repository holds references and never values — the same rule as .env.example in this repository.

A change to a ConfigMap does not restart the pods that use it as environment variables. The usual trick is a checksum of the ConfigMap in a pod annotation, so a config change is a template change and rolls the Deployment.

#Probes: three questions, three endpoints

The kubelet asks a pod three things, and getting the endpoints right is the difference between a cluster that heals and one that kills healthy pods:

startupProbe:  { httpGet: { path: /actuator/health/liveness,  port: 8080 }, periodSeconds: 5, failureThreshold: 30 }
livenessProbe: { httpGet: { path: /actuator/health/liveness,  port: 8080 }, periodSeconds: 10, failureThreshold: 3 }
readinessProbe: { httpGet: { path: /actuator/health/readiness, port: 8080 }, periodSeconds: 5, failureThreshold: 2 }

The Actuator lesson separated liveness from readiness for exactly this: a liveness check that includes the database restarts your pods when the database is slow, which turns a database incident into a restart storm on top of it. Liveness should fail only when the JVM itself is wedged; readiness should fail when a dependency the pod needs is unavailable, so traffic goes elsewhere while it recovers. The startup probe with a generous threshold is what lets a Spring Boot application take 60 seconds on a throttled CPU without the liveness probe killing it at 30; before startup probes existed, initialDelaySeconds did that job badly.

Graceful shutdown is the readiness probe run backwards: on SIGTERM Spring Boot fails readiness first (server.shutdown=graceful makes the readiness state REFUSING_TRAFFIC), the endpoint list drops the pod, in-flight requests finish, then the process exits. terminationGracePeriodSeconds (default 30) must exceed the shutdown timeout, and a preStop sleep of a few seconds covers the window before every kube-proxy has seen the endpoint change.

#Requests and limits

resources:
  requests: { cpu: "1", memory: "1Gi" }
  limits:   { memory: "1Gi" }

A request is what the scheduler reserves: a pod is placed on a node with that much unallocated, and the CPU request is its guaranteed share under contention. A limit is what the kernel enforces: a memory limit is the cgroup limit the Java-in-a-container lesson's JVM reads and the kernel kills at; a CPU limit is a quota that throttles.

The two rules that follow, and that the community has argued to a consensus on: memory request equals limit, so the pod is never scheduled somewhere it will be killed for using what it needs; and a CPU request with no CPU limit, so the JVM sees a sensible processor count from the request and can burst into idle cores instead of being throttled at a quota — throttling produces the latency spikes the container lesson described, and a limit buys nothing on a node that is not contended. The exception is a multi-tenant node where one runaway pod must not starve the others; there, a limit is a fence, and a generous one.

#QoS: who gets killed first

From requests and limits the cluster assigns a quality of service class, and it is the order of eviction when a node runs short:

• Guaranteed — every container has requests equal to limits for both CPU and memory. Evicted last.
• Burstable — some requests, not all equal to limits. The recommended shape above is here.
• BestEffort — no requests or limits at all. Evicted first, and scheduled anywhere, and throttled to nothing under contention. A pod with no resources block is a pod you have told the cluster you do not care about.

#OOMKilled, and reading it

kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137. That is the kernel, not Java: the container's total memory hit the limit. The JVM's own OutOfMemoryError is a different event, exit code 1 with a stack trace, and it is why -XX:+ExitOnOutOfMemoryError matters — without it a heap-exhausted JVM stays up, fails no probe and serves nothing.

For OOMKilled, the checklist from the container lesson: is MaxRAMPercentage leaving room for non-heap; is direct memory or Metaspace growing (Native Memory Tracking); are there more threads than expected (each stack is off-heap); is the limit simply too small for the workload. kubectl top pod shows the working set against the limit, and a pod that sits at 95% of its limit is not fine, it is one allocation from a restart. Raise the limit and the request together, and find out why before the next one.