Five AKS clusters, one Git repository

Five clusters, two hard boundaries

A cluster running well today says little about its state six months from now. Manual changes accumulate, versions drift and the reason behind a setting disappears. Across five clusters, that drift quickly becomes structural. The entire platform layer is therefore declarative: what is in Git is what should be running.

The landscape contains two cluster families. infra provides management capabilities such as identity, central observability and CI runners. worker runs workloads and related platform services. In addition to non-production and production, a vnext cluster receives new components and versions first.

Non-production and production each have a dedicated Argo CD installation. This adds some operational work, but creates a concrete boundary: a failure, upgrade or overly broad permission in the non-production control plane cannot directly affect production. Non-production assumes that data may leak, so only anonymised data is allowed there. Production processes real data; data must not leak from that environment.

The repository is the deployment model

The repository contains more than 3,700 files. Nearly 3,000 are vendored Helm charts; the actual platform logic spans roughly 770 files. The fixed structure matters more than the file count:

applications/
  infrastructure/<component>/<cluster>/
  platform/<component>/<cluster>/
  applications/<workload>/<cluster>/
applicationsets/
  base/
  overlays/nonprod/
  overlays/prod/
argocd/
bootstrap/<stage>/
projects/
clusters/

Each component and cluster combination has its own build root. It contains kustomization.yaml, helm-values.yaml and argocd-app-config.yaml. The final file is deployment metadata for the ApplicationSet: name, namespace, sync wave, target cluster, stage and Git revision.

Shared values are sometimes repeated deliberately. A build remains inside its own directory and a single folder shows the complete state for that cluster. Reviewers do not need to follow hidden inheritance across several layers.

Charts are vendored

Upstream Helm charts are stored in the repository at an exact version. Kustomize combines the chart, additional manifests and patches in a single build. Rendering needs no network access and produces a reproducible result. Previous chart versions remain available while any cluster still uses them.

Onboarding without a central inventory list

Terraform creates one root Application per environment. From that point, Argo CD manages itself and the rest of the platform. The bootstrap applies AppProjects and the stage-specific ApplicationSets. Thirteen ApplicationSets each use a Git generator to discover configuration files.

Every discovered argocd-app-config.yaml file produces an Application with a predictable name, such as infra-prod-cert-manager. Argo CD then runs kustomize build --enable-helm, compares the result with the target cluster and synchronises only the differences.

Three types of ApplicationSet

• Local components: infrastructure, platform services and workloads from this repository.
• Cluster-configuration driven: infra configuration determines which instance from another repository is connected.
• External workload repositories: teams maintain their deployment configuration; the platform repository controls the allowed AppProject and destination.

This keeps operators and instances separate. The Keycloak and RabbitMQ operators belong to the platform; their instances live with the service responsible for them.

Ordering and environment separation live in code

Platform components have hard dependencies. Gateway API CRDs must exist before Istio and gateways can start. The required order is therefore stored as a syncWave alongside the component. The ApplicationSet adds it as an annotation to the generated Application.

ApplicationSets are defined once in a shared base. Its stage selector deliberately uses the invalid value REPLACE_ME. Only the non-production or production overlay injects a valid stage. Rendering the base by mistake therefore selects nothing. Future clusters also remain as .template files until registered with Argo CD.

Generated Applications use automated sync with prune and selfHeal. Manual changes are reverted and resources removed from Git are removed from the cluster. Controller or webhook mutations are narrowly captured in ignoreDifferences, preventing required changes from causing a permanent sync loop.

Boundaries before a manifest reaches the cluster

Four AppProjects separate bootstrap, infrastructure, platform services and applications. Each project defines allowed repositories, clusters, namespaces and cluster-scoped resource types. A workload cannot create a ClusterRole when its project does not permit it, even if that manifest is present in the workload repository.

Argo CD uses deny-all-by-default RBAC. Users sign in through Entra ID; group membership determines project access and elevated permissions are activated through PIM. Secrets follow one route: Azure Key Vault, through Workload Identity and External Secrets Operator, into a Kubernetes Secret. Git contains only the reference.

The pull request is the first policy boundary

Every changed cluster directory is rendered. YAML and schema validation, path checks, Gitleaks, RBAC validation, Trivy and Kyverno CLI then inspect the actual output. A manifest diff makes the real effect of a chart update visible; one changed version line can produce hundreds of changed manifest lines.

• BuildKustomize per cluster directory
• SchemaYAML and Kubernetes resources
• SecretsGitleaks before merge
• PolicyKyverno CLI and Trivy
• RBACAppProjects and permissions
• ReviewRendered manifest diff

A platform layer of CNCF building blocks

The infrastructure layer contains 24 components. Not every cluster receives the same set: the cluster role determines which operators and platform services are required.

Observability is centralised. Alloy sends metrics, logs and traces from every cluster to the LGTM backend on infra. This keeps local agents small and enables analysis across cluster boundaries.

Updates progress towards production in controlled stages

Renovate tracks chart and image versions, but never changes all clusters at once. Each component and environment receives a separate pull request. Versions land on vnext and non-production first. Production follows only after a release has been available for at least seven days. The exact version remains pinned per cluster.

How this relates to Haven+

Haven+ is a reference architecture and implementation for additional platform services on a Haven Kubernetes environment. Its direction aligns with this design: GitOps, open-source components and repeatable delivery of monitoring, identity, certificates, databases, policy and secrets. This is an architecture comparison, not a formal claim of compliance.

There is substantial overlap: Alloy, cert-manager, CloudNativePG, ExternalDNS, External Secrets, Grafana, Loki, Mimir, Tempo, Istio and Keycloak appear in both landscapes. The implementation differs where the target platform requires it. Logs and traces use Loki and Tempo rather than ECK. Secrets remain in Key Vault instead of being encrypted in Git. Kyverno, KEDA, RabbitMQ, Redis and Actions Runner Controller are additional choices.

The Kubernetes patterns are portable. The Azure integrations are not automatically portable.

The implementation is explicitly Azure-specific through AKS, Azure DNS, Key Vault, Workload Identity and Azure Container Registry. An exit strategy must make that distinction explicit: GitOps and Kubernetes patterns can move, while cloud integrations require replacement.

Practical lessons from this design

1. Make the cluster directory the unit. Keep chart, values and deployment metadata together.
2. Let ApplicationSets discover. The generator does not change when a cluster or component is added.
3. Separate the control planes. Production gains a boundary that does not rely on RBAC alone.
4. Make builds reproducible. Vendor charts and pin every version per cluster.
5. Express the baseline as code. Policy and CI re-evaluate agreements for every change.
6. Keep secrets out of Git. Use an external secret store with workload identity.

The design is not finished. Components and reference architectures continue to evolve. The lasting value is in the operating model: every change is reviewable, traceable and reversible. For software that residents rely on every day, that is a basic requirement.