# Kubernetes high availability: eight settings working together

High availability for applications on Kubernetes.

Pascal van der Horst · 2026-10-30

A node fails. The application needs to keep running. That takes enough replicas, spread across nodes and zones, with capacity for recovery and updates that keep serving traffic.

The eight settings in this article use one stateless API as a running example, with three replicas and three zones. The cluster foundation comes first, followed by the application settings. Databases, singletons and batch workloads need their own choices.

Test what stays available during a failure. Measure how much traffic the application still handles and how long recovery takes.

## Start with the cluster foundation.

The control plane manages the cluster; etcd stores cluster state. With stacked etcd, both run on the same nodes. A common HA setup for a self-managed cluster has at least three control-plane nodes behind a redundant API endpoint. Three etcd members require a majority of two and can therefore lose one member. The consensus protocol enforces that majority.

With managed Kubernetes, the provider operates part of this foundation. Check which components span zones, the availability commitments and who owns recovery. Existing pods may keep working when the control plane fails, while placing new pods, changing configuration and coordinating recovery may be blocked.

Check ingress or Gateway, DNS, container networking, storage and external dependencies too. A database needs its own availability design even when its operator runs redundantly. Zone-bound volumes can limit rescheduling. Also maintain backups and test restores.

Agree which failures the design needs to handle. Define the target in a Service Level Objective (SLO). Specify the intended recovery time, the Recovery Time Objective (RTO), and the acceptable data-loss window, the Recovery Point Objective (RPO). Losing one zone and losing an entire region are different scenarios.

## 1. Replicas: enough copies and remaining capacity.

How many instances remain usable when one fails?

Choose enough replicas to handle the agreed load after a failure. This example runs three replicas. Losing one leaves two. Measure whether those two can handle peak traffic.

A Deployment sets the desired pod count through spec.replicas. Its ReplicaSet works to maintain that count. The application must support concurrent instances: check local sessions, files, locks and background jobs. Shared state can be stored outside the pods; that datastore needs its own availability design.

A single replica may update without planned interruption when surge capacity is available, but cannot tolerate failure of that one pod. Choose the count based on load, the importance of the application and the failures it needs to handle. An odd replica count does not give a normal API quorum or split-brain protection.

```yaml
spec:
  replicas: 3
```

**Check:** Measure whether two remaining replicas can handle the agreed peak load.

[Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)

## 2. PodDisruptionBudget: bound maintenance disruption.

How many pods may be evicted at the same time?

Limit how many healthy pods may be removed during maintenance. A PodDisruptionBudget (PDB) does this for voluntary disruptions through the Eviction API, such as a node drain. Three healthy replicas with minAvailable: 2 normally allow one eviction. Use minAvailable or maxUnavailable and match the intended pods with the label selector.

The PDB does not constrain node or zone failure, Deployment rollouts, direct pod deletion or scaling through the scale subresource. A HorizontalPodAutoscaler (HPA) can therefore scale down despite the PDB. Configure the minimum replica count separately.

Also choose how to handle unhealthy pods. Omitting unhealthyPodEvictionPolicy uses IfHealthyBudget: a running pod that is not Ready may be evicted only when currentHealthy is at least desiredHealthy. If that budget is not met, such a pod can block a drain.

The example uses unhealthyPodEvictionPolicy: AlwaysAllow for this stateless API. Running, unhealthy pods may then be evicted even when too few healthy pods remain. Healthy pods still follow the PDB. The tradeoff: maintenance need not wait for the unhealthy pod, but it also loses the chance to recover on the same node. Choose the policy based on the application’s recovery behaviour.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: example-api
  namespace: ha-example
spec:
  minAvailable: 2
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: example-api
```

**Check:** Check status.disruptionsAllowed. Test a drain with three healthy pods, a missing pod and a running pod that does not become Ready.

![PodDisruptionBudget: bound maintenance disruption.](https://horstconsultancy.nl/images/articles/ha/disruptions-en-light-wide.svg)

[Pod disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) · [Configure a PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)

## 3. Node anti-affinity: separate hosts for replicas.

Can one failed node take down several replicas?

Place replicas on separate nodes so one failed node does not take down every copy. Pod anti-affinity applies this to new pods with the same application label. The example uses topologyKey: kubernetes.io/hostname to spread them by host.

requiredDuringSchedulingIgnoredDuringExecution is a hard placement rule; preferredDuringSchedulingIgnoredDuringExecution is a preference. A hard rule leaves a pod Pending when no node meets the conditions. Spare resources, taints, tolerations and node selectors also matter. Three replicas need three eligible hosts here; an extra surge pod temporarily needs a fourth.

IgnoredDuringExecution means the rule does not automatically move existing pods when labels change. Check actual placement after a rollout and after maintenance.

```yaml
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: example-api
        topologyKey: kubernetes.io/hostname
```

**Check:** Check the node column: each replica runs on a different eligible host.

[Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

## 4. Zone spread: shared infrastructure can fail too.

Does the application remain useful when a failure domain is lost?

Spread replicas across zones to limit shared failures. Separate hosts may still share a power supply or network connection. topologySpreadConstraints uses the topology.kubernetes.io/zone node label for this distribution. The labels must represent real, independent infrastructure.

With maxSkew: 1 and DoNotSchedule, the scheduler limits the difference relative to the eligible zones with the fewest pods. It counts pods matching the label selector, regardless of processing capacity. ScheduleAnyway makes placement a preference. Existing pods are not automatically moved.

nodeAffinityPolicy and nodeTaintsPolicy help determine which zones count. The example uses Honor for both. minDomains is deliberately not set to 3: that minimum can block replacement pods in two surviving zones. Omitting the field uses 1 in this calculation. During a zone failure, also check taints, tolerations and spare capacity on the remaining nodes.

```yaml
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    nodeAffinityPolicy: Honor
    nodeTaintsPolicy: Honor
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: example-api
```

**Check:** Test both remaining capacity and Pending pods when one zone becomes unavailable.

![Zone spread: shared infrastructure can fail too.](https://horstconsultancy.nl/images/articles/ha/placement-en-light-wide.svg)

[Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)

## 5. Rolling updates: make the replacement usable first.

Can a new version start before old capacity is removed?

Start a usable replacement before removing old capacity. With a healthy starting state, RollingUpdate using maxUnavailable: 0 retains the desired available capacity. maxSurge: 1 allows one extra pod. minReadySeconds: 10 requires a new pod to stay Ready for ten continuous seconds before counting as Available.

The Deployment manages this separately from the PDB. Without room for the extra pod, a rollout may stall while old pods keep running. Terminating pods can temporarily consume resources too. Allow for that beyond maxSurge. progressDeadlineSeconds marks a stalled rollout; rollback needs separate handling.

Keep old and new versions compatible, including shared data and database schemas. On SIGTERM, the application should stop accepting new work and finish in-flight requests. terminationGracePeriodSeconds covers the entire shutdown, including any preStop hooks. Allow routing and endpoint changes to propagate and test the result under load.

```yaml
spec:
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60
```

**Check:** Run an update under load and measure error rate, latency and available replicas.

![Rolling updates: make the replacement usable first.](https://horstconsultancy.nl/images/articles/ha/rollout-en-light-wide.svg)

[Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) · [Pod lifecycle and termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)

## 6. Resources: capacity for normal operation and disruption.

Can recovery and an update still fit when load is high?

Reserve capacity for failures and updates too. The scheduler uses resource requests to place pods. Requests that are too low can leave a node overloaded at peak usage; requests that are too high hold capacity other pods could use. Measure normal consumption, startup peaks and failure behaviour.

CPU limits can cause throttling, reducing the CPU time available to the application. Exceeding a memory limit can trigger an out-of-memory (OOM) kill. This example omits a CPU limit. Shared tenancy, isolation requirements or cluster policy may still require an explicit CPU limit.

Keep room for system workloads, an extra pod during a rollout and rescheduling after a failure. A CPU-based HPA compares usage with the CPU request. Set that request from measurements. These requests and a memory-only limit do not give the pod Guaranteed QoS.

```yaml
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi
```

**Check:** Measure startup peaks, OOM kills, throttling and scheduling during a failure.

[Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) · [Horizontal Pod Autoscaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)

## 7. Probes: separate startup, traffic and restart decisions.

Which check stops traffic and which restarts a container?

Use separate checks for startup, serving traffic and restarting. A startupProbe allows time for initialization. Liveness and readiness checks wait until it succeeds. periodSeconds: 5 and failureThreshold: 24 give roughly two minutes of startup time here; probe scheduling and execution determine the actual timing.

Readiness determines whether a pod can receive traffic through an ordinary Service. A failed readiness probe does not restart the container. Liveness is for a process that cannot recover by itself, such as a deadlock. Enough failed checks restart the container. A startup probe that keeps failing can also trigger a restart.

Keep external databases and APIs out of liveness so a shared outage does not cause a restart wave. Readiness can check essential dependencies. Account for database failure potentially making every replica NotReady. Keep probes lightweight and choose timeouts and thresholds from measured behaviour.

```yaml
startupProbe:
  httpGet: { path: /health/startup, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 24
readinessProbe:
  httpGet: { path: /health/ready, port: http }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /health/live, port: http }
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
```

**Check:** Test slow startup, a NotReady pod and an external dependency outage separately.

![Probes: separate startup, traffic and restart decisions.](https://horstconsultancy.nl/images/articles/ha/probes-en-light-wide.svg)

[Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/)

## 8. PriorityClass: make scarcity decisions explicit.

Which workload gets priority when capacity is scarce?

Agree which workloads get priority when capacity is scarce. A PriorityClass sets relative scheduling priority. It is a cluster-wide object that must exist beforehand. A pod references it through priorityClassName. Use an agreed scheme so each value has a clear meaning.

With preemptionPolicy: Never, the pod gets scheduling queue priority without displacing running pods of lower priority. The default PreemptLowerPriority can displace them under certain conditions. Kubernetes tries to respect a PDB during preemption, but may still exceed the budget.

A PriorityClass does not reserve CPU, memory or nodes. A high-priority pod can still remain Pending because of insufficient capacity or hard placement rules. Combine priority with sufficient capacity and restrict who may use high priorities.

```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: example-critical
value: 1000000
globalDefault: false
preemptionPolicy: Never
description: "Explicit priority without preemption"
# Pod spec: priorityClassName: example-critical
```

**Check:** Test queue behaviour and Pending reasons under scarcity without unintended displacement.

[Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)

## All eight settings in one manifest.

The reference manifest combines three replicas, three zones, a PDB with minAvailable: 2 and room for one extra pod during an update. The two surviving replicas must handle the agreed load. The application must support concurrent instances, the three probe endpoints and graceful shutdown.

Required hostname anti-affinity gives each replica its own node. Three replicas with maxSurge: 1 need at least four eligible worker nodes during an update. Check that zone placement, taints, requests and quotas allow the extra pod, including during a failure. Create the namespace and PriorityClass before starting the Deployment.

The registry.example.com image is a placeholder. Replace it with a reviewed image, preferably pinned by digest. Adapt resources, labels, probes and permissions to the application. The topology-spread node inclusion policies used here are stable since Kubernetes 1.33; check support in the target cluster.

[YAML reference](https://horstconsultancy.nl/downloads/kubernetes-ha-reference.yaml)

## Align autoscaling with recovery and capacity.

A HorizontalPodAutoscaler (HPA) adjusts replica count to measured load. A CPU-utilization target compares CPU usage with the pods’ CPU requests. This requires working metrics and appropriate requests.

The HPA does not create nodes. Provide room through spare capacity, suitable node autoscaling and quotas. Scaling takes time. The example retains at least three replicas, for instance with minReplicas: 3 in an HPA. The PDB does not set that minimum. Use the same minimum with event-driven autoscaling through KEDA; scaling to zero does not fit this API.

When an HPA manages replica count, a GitOps controller should leave that field alone. Configure diff and sync behaviour accordingly and define which controller manages each field. Do not let two autoscalers independently control the same workload.

Measure error rate, latency and successful requests at the service boundary. Use Pending pods, Ready/Available replicas, PDB status, OOM kills and throttling to investigate disruptions.

## Test what stays available during a failure.

Check the manifests that will actually be deployed and compare them with live placement. Pods do not automatically inherit their node’s zone labels. Map the pods’ node column to the nodes’ zone labels. These commands only read status.

```bash
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pods -n ha-example \
  -l app.kubernetes.io/name=example-api -o wide
kubectl get pdb example-api -n ha-example
kubectl describe pdb example-api -n ha-example
kubectl rollout status deployment/example-api -n ha-example
kubectl top pods -n ha-example \
  -l app.kubernetes.io/name=example-api
```

- **Node failure:** Remaining replicas serve the agreed traffic; errors and recovery remain within the objective.

- **Controlled drain:** Evictions respect the PDB; recovery permits subsequent maintenance steps.

- **Zone failure:** Traffic, compute, storage and dependencies are tested together; Pending reasons are understood.

- **Rollout under load:** New pods become Available, existing requests drain and application versions can coexist.

- **Dependency failure:** No liveness restart wave; degraded behaviour and recovery are predictable.

- **Data recovery:** An executed restore demonstrates achievable RTO/RPO for the relevant datastore.

For each test, record the traffic still being served, the errors and the recovery time. Compare the results with the agreed target. Then adjust replicas, capacity or placement and repeat the test.

## Sources

Verified on 30 October 2026. Consult the official documentation for the Kubernetes version in use.

- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)

- [Pod disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)

- [Configure a PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)

- [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

- [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)

- [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)

- [Liveness, Readiness, and Startup Probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/)

- [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)

- [Options for Highly Available Topology](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/)

- [Horizontal Pod Autoscaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)

- [Pod lifecycle and termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)

- [etcd: failure tolerance](https://etcd.io/docs/v3.6/faq/#what-is-failure-tolerance)