Kubernetes high availability: eight settings working together

Start with the cluster foundation.

The control plane manages the cluster; etcd stores cluster state. With stacked etcd, both run on the same nodes. A common HA setup for a self-managed cluster has at least three control-plane nodes behind a redundant API endpoint. Three etcd members require a majority of two and can therefore lose one member. The consensus protocol enforces that majority.

With managed Kubernetes, the provider operates part of this foundation. Check which components span zones, the availability commitments and who owns recovery. Existing pods may keep working when the control plane fails, while placing new pods, changing configuration and coordinating recovery may be blocked.

Check ingress or Gateway, DNS, container networking, storage and external dependencies too. A database needs its own availability design even when its operator runs redundantly. Zone-bound volumes can limit rescheduling. Also maintain backups and test restores.

Agree which failures the design needs to handle. Define the target in a Service Level Objective (SLO). Specify the intended recovery time, the Recovery Time Objective (RTO), and the acceptable data-loss window, the Recovery Point Objective (RPO). Losing one zone and losing an entire region are different scenarios.

Options for Highly Available Topology · etcd: failure tolerance

1. Replicas: enough copies and remaining capacity.

Choose enough replicas to handle the agreed load after a failure. This example runs three replicas. Losing one leaves two. Measure whether those two can handle peak traffic.

A Deployment sets the desired pod count through spec.replicas. Its ReplicaSet works to maintain that count. The application must support concurrent instances: check local sessions, files, locks and background jobs. Shared state can be stored outside the pods; that datastore needs its own availability design.

A single replica may update without planned interruption when surge capacity is available, but cannot tolerate failure of that one pod. Choose the count based on load, the importance of the application and the failures it needs to handle. An odd replica count does not give a normal API quorum or split-brain protection.

spec:
  replicas: 3

Practical checkMeasure whether two remaining replicas can handle the agreed peak load.

Deployments

2. PodDisruptionBudget: bound maintenance disruption.

Limit how many healthy pods may be removed during maintenance. A PodDisruptionBudget (PDB) does this for voluntary disruptions through the Eviction API, such as a node drain. Three healthy replicas with minAvailable: 2 normally allow one eviction. Use minAvailable or maxUnavailable and match the intended pods with the label selector.

The PDB does not constrain node or zone failure, Deployment rollouts, direct pod deletion or scaling through the scale subresource. A HorizontalPodAutoscaler (HPA) can therefore scale down despite the PDB. Configure the minimum replica count separately.

Also choose how to handle unhealthy pods. Omitting unhealthyPodEvictionPolicy uses IfHealthyBudget: a running pod that is not Ready may be evicted only when currentHealthy is at least desiredHealthy. If that budget is not met, such a pod can block a drain.

The example uses unhealthyPodEvictionPolicy: AlwaysAllow for this stateless API. Running, unhealthy pods may then be evicted even when too few healthy pods remain. Healthy pods still follow the PDB. The tradeoff: maintenance need not wait for the unhealthy pod, but it also loses the chance to recover on the same node. Choose the policy based on the application’s recovery behaviour.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: example-api
  namespace: ha-example
spec:
  minAvailable: 2
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: example-api

Practical checkCheck status.disruptionsAllowed. Test a drain with three healthy pods, a missing pod and a running pod that does not become Ready.

Pod disruptions·Configure a PodDisruptionBudget

3. Node anti-affinity: separate hosts for replicas.

Place replicas on separate nodes so one failed node does not take down every copy. Pod anti-affinity applies this to new pods with the same application label. The example uses topologyKey: kubernetes.io/hostname to spread them by host.

requiredDuringSchedulingIgnoredDuringExecution is a hard placement rule; preferredDuringSchedulingIgnoredDuringExecution is a preference. A hard rule leaves a pod Pending when no node meets the conditions. Spare resources, taints, tolerations and node selectors also matter. Three replicas need three eligible hosts here; an extra surge pod temporarily needs a fourth.

IgnoredDuringExecution means the rule does not automatically move existing pods when labels change. Check actual placement after a rollout and after maintenance.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: example-api
        topologyKey: kubernetes.io/hostname

Practical checkCheck the node column: each replica runs on a different eligible host.

Assigning Pods to Nodes

4. Zone spread: shared infrastructure can fail too.

Spread replicas across zones to limit shared failures. Separate hosts may still share a power supply or network connection. topologySpreadConstraints uses the topology.kubernetes.io/zone node label for this distribution. The labels must represent real, independent infrastructure.

With maxSkew: 1 and DoNotSchedule, the scheduler limits the difference relative to the eligible zones with the fewest pods. It counts pods matching the label selector, regardless of processing capacity. ScheduleAnyway makes placement a preference. Existing pods are not automatically moved.

nodeAffinityPolicy and nodeTaintsPolicy help determine which zones count. The example uses Honor for both. minDomains is deliberately not set to 3: that minimum can block replacement pods in two surviving zones. Omitting the field uses 1 in this calculation. During a zone failure, also check taints, tolerations and spare capacity on the remaining nodes.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    nodeAffinityPolicy: Honor
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: example-api

Practical checkTest both remaining capacity and Pending pods when one zone becomes unavailable.

Pod Topology Spread Constraints

5. Rolling updates: make the replacement usable first.

Start a usable replacement before removing old capacity. With a healthy starting state, RollingUpdate using maxUnavailable: 0 retains the desired available capacity. maxSurge: 1 allows one extra pod. minReadySeconds: 10 requires a new pod to stay Ready for ten continuous seconds before counting as Available.

The Deployment manages this separately from the PDB. Without room for the extra pod, a rollout may stall while old pods keep running. Terminating pods can temporarily consume resources too. Allow for that beyond maxSurge. progressDeadlineSeconds marks a stalled rollout; rollback needs separate handling.

Keep old and new versions compatible, including shared data and database schemas. On SIGTERM, the application should stop accepting new work and finish in-flight requests. terminationGracePeriodSeconds covers the entire shutdown, including any preStop hooks. Allow routing and endpoint changes to propagate and test the result under load.

spec:
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60

Practical checkRun an update under load and measure error rate, latency and available replicas.

Deployments·Pod lifecycle and termination

6. Resources: capacity for normal operation and disruption.

Reserve capacity for failures and updates too. The scheduler uses resource requests to place pods. Requests that are too low can leave a node overloaded at peak usage; requests that are too high hold capacity other pods could use. Measure normal consumption, startup peaks and failure behaviour.

CPU limits can cause throttling, reducing the CPU time available to the application. Exceeding a memory limit can trigger an out-of-memory (OOM) kill. This example omits a CPU limit. Shared tenancy, isolation requirements or cluster policy may still require an explicit CPU limit.

Keep room for system workloads, an extra pod during a rollout and rescheduling after a failure. A CPU-based HPA compares usage with the CPU request. Set that request from measurements. These requests and a memory-only limit do not give the pod Guaranteed QoS.

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi

Practical checkMeasure startup peaks, OOM kills, throttling and scheduling during a failure.

Resource Management for Pods and Containers·Horizontal Pod Autoscaling

7. Probes: separate startup, traffic and restart decisions.

Use separate checks for startup, serving traffic and restarting. A startupProbe allows time for initialization. Liveness and readiness checks wait until it succeeds. periodSeconds: 5 and failureThreshold: 24 give roughly two minutes of startup time here; probe scheduling and execution determine the actual timing.

Readiness determines whether a pod can receive traffic through an ordinary Service. A failed readiness probe does not restart the container. Liveness is for a process that cannot recover by itself, such as a deadlock. Enough failed checks restart the container. A startup probe that keeps failing can also trigger a restart.

Keep external databases and APIs out of liveness so a shared outage does not cause a restart wave. Readiness can check essential dependencies. Account for database failure potentially making every replica NotReady. Keep probes lightweight and choose timeouts and thresholds from measured behaviour.

startupProbe:
  httpGet: { path: /health/startup, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 24
readinessProbe:
  httpGet: { path: /health/ready, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /health/live, port: http }
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

Practical checkTest slow startup, a NotReady pod and an external dependency outage separately.

Liveness, Readiness, and Startup Probes

8. PriorityClass: make scarcity decisions explicit.

Agree which workloads get priority when capacity is scarce. A PriorityClass sets relative scheduling priority. It is a cluster-wide object that must exist beforehand. A pod references it through priorityClassName. Use an agreed scheme so each value has a clear meaning.

With preemptionPolicy: Never, the pod gets scheduling queue priority without displacing running pods of lower priority. The default PreemptLowerPriority can displace them under certain conditions. Kubernetes tries to respect a PDB during preemption, but may still exceed the budget.

A PriorityClass does not reserve CPU, memory or nodes. A high-priority pod can still remain Pending because of insufficient capacity or hard placement rules. Combine priority with sufficient capacity and restrict who may use high priorities.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: example-critical
value: 1000000
globalDefault: false
preemptionPolicy: Never
description: "Explicit priority without preemption"
# Pod spec: priorityClassName: example-critical

Practical checkTest queue behaviour and Pending reasons under scarcity without unintended displacement.

Pod Priority and Preemption

All eight settings in one manifest.

The reference manifest combines three replicas, three zones, a PDB with minAvailable: 2 and room for one extra pod during an update. The two surviving replicas must handle the agreed load. The application must support concurrent instances, the three probe endpoints and graceful shutdown.

Required hostname anti-affinity gives each replica its own node. Three replicas with maxSurge: 1 need at least four eligible worker nodes during an update. Check that zone placement, taints, requests and quotas allow the extra pod, including during a failure. Create the namespace and PriorityClass before starting the Deployment.

The registry.example.com image is a placeholder. Replace it with a reviewed image, preferably pinned by digest. Adapt resources, labels, probes and permissions to the application. The topology-spread node inclusion policies used here are stable since Kubernetes 1.33; check support in the target cluster.

# Illustrative stateless API reference, not a ready-to-apply application.
# Replace the placeholder image, health endpoints and resource values.
# Assumptions: three real zone domains and >=4 eligible worker nodes for surge.
# Two surviving replicas must handle the agreed load (verify by load testing).
# Topology-spread node inclusion policies are stable since Kubernetes 1.33.
apiVersion: v1
kind: Namespace
metadata:
  name: ha-example
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: example-critical
value: 1000000
globalDefault: false
preemptionPolicy: Never
description: "Explicit application priority without preemption"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-api
  namespace: ha-example
spec:
  replicas: 3
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/name: example-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: example-api
    spec:
      priorityClassName: example-critical
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/name: example-api
              topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          nodeAffinityPolicy: Honor
          nodeTaintsPolicy: Honor
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: example-api
      containers:
        - name: api
          # Placeholder: replace with a reviewed, digest-pinned application image.
          image: registry.example.com/team/example-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: temporary-files
              mountPath: /tmp
      volumes:
        - name: temporary-files
          emptyDir:
            sizeLimit: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: example-api
  namespace: ha-example
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: example-api
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: example-api
  namespace: ha-example
spec:
  minAvailable: 2
  # Healthy pods retain the budget; unhealthy Running pods need not block drain.
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: example-api

Align autoscaling with recovery and capacity.

A HorizontalPodAutoscaler (HPA) adjusts replica count to measured load. A CPU-utilization target compares CPU usage with the pods’ CPU requests. This requires working metrics and appropriate requests.

The HPA does not create nodes. Provide room through spare capacity, suitable node autoscaling and quotas. Scaling takes time. The example retains at least three replicas, for instance with minReplicas: 3 in an HPA. The PDB does not set that minimum. Use the same minimum with event-driven autoscaling through KEDA; scaling to zero does not fit this API.

When an HPA manages replica count, a GitOps controller should leave that field alone. Configure diff and sync behaviour accordingly and define which controller manages each field. Do not let two autoscalers independently control the same workload.

Measure error rate, latency and successful requests at the service boundary. Use Pending pods, Ready/Available replicas, PDB status, OOM kills and throttling to investigate disruptions.

Horizontal Pod Autoscaling

Test what stays available during a failure.

Check the manifests that will actually be deployed and compare them with live placement. Pods do not automatically inherit their node’s zone labels. Map the pods’ node column to the nodes’ zone labels. These commands only read status.

kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pods -n ha-example \
  -l app.kubernetes.io/name=example-api -o wide
kubectl get pdb example-api -n ha-example
kubectl describe pdb example-api -n ha-example
kubectl rollout status deployment/example-api -n ha-example
kubectl top pods -n ha-example \
  -l app.kubernetes.io/name=example-api

kubectl top shows current consumption; sizing requires historical and peak measurements. Plan disruption exercises in an agreed test environment, with monitoring and a recovery path. Record conditions and outcomes for each exercise.

Node failure

Remaining replicas serve the agreed traffic; errors and recovery remain within the objective.

Controlled drain

Evictions respect the PDB; recovery permits subsequent maintenance steps.

Zone failure

Traffic, compute, storage and dependencies are tested together; Pending reasons are understood.

Rollout under load

New pods become Available, existing requests drain and application versions can coexist.

Dependency failure

No liveness restart wave; degraded behaviour and recovery are predictable.

Data recovery

An executed restore demonstrates achievable RTO/RPO for the relevant datastore.

For each test, record the traffic still being served, the errors and the recovery time. Compare the results with the agreed target. Then adjust replicas, capacity or placement and repeat the test.