From d010a5595f5d9b4ede841f3c044205054e288f36 Mon Sep 17 00:00:00 2001 From: Baptiste Bonnot Date: Mon, 15 Sep 2025 13:34:15 +0200 Subject: [PATCH] Adding base overlays patterns --- ...pository-Handling-Multiple-Environments.md | 250 +++++++++++++++++- 1 file changed, 248 insertions(+), 2 deletions(-) diff --git a/Guide-Repository-Handling-Multiple-Environments.md b/Guide-Repository-Handling-Multiple-Environments.md index b57165e..1398035 100644 --- a/Guide-Repository-Handling-Multiple-Environments.md +++ b/Guide-Repository-Handling-Multiple-Environments.md @@ -250,8 +250,254 @@ to be loaded for a given environments. In this example we are going to create a directory structure that allow us to *Have shared resources definition and environments specific one. -Taking back our echo-server example. +Taking back our echo-server example we are going to show how spawn different +resources definitions base on the environments. Here the diagram of what we are +trying to achieve. -XXX +```mermaid +flowchart TD + %% Node styles + classDef file fill:#ffffff,stroke:#9CA3AF,stroke-width:1px,color:#374151,rounded:10px; + classDef var fill:#FEF9C3,stroke:#F59E0B,stroke-width:1px,color:#78350F,rounded:8px; + classDef overlay fill:#DCFCE7,stroke:#22C55E,stroke-width:2px,color:#166534,rounded:10px; + classDef base fill:#E0F2FE,stroke:#3B82F6,stroke-width:2px,color:#1E40AF,rounded:10px; + classDef flux fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95,rounded:10px; + + %% Git repository + subgraph GitRepo["📂 Git Repository"] + subgraph Base["Base (shared resources)"] + D1["Deployment.yaml"]:::file + HR["HelmRelease.yaml"]:::file + KB["kustomization.yaml"]:::base + end + + subgraph Overlays["Overlays (tenant-specific)"] + subgraph Dev["🟢 Dev Overlay"] + P1["patch-replicas.yaml"]:::file + KD["kustomization.yaml"]:::overlay + end + subgraph Prod["🔴 Prod Overlay"] + P2["patch-resources.yaml"]:::file + KP["kustomization.yaml"]:::overlay + end + end + end + + %% Flux Kustomization + subgraph Flux["⚡ FluxCD"] + KOverlay["Kustomization CR → overlays/${TENANT_NAMESPACE}"]:::flux + end + + %% Kubernetes Cluster + subgraph Cluster["🖥️ Kubernetes Cluster"] + NS["${TENANT_NAMESPACE} namespace"]:::var + APP["myapp resources"]:::file + end + + %% Connections + KOverlay --> Dev + KOverlay --> Prod + Dev --> Base + Prod --> Base + Dev --> APP + Prod --> APP + APP --> NS +``` +In the given example we are going on the tenant-tpl-dev increase the number of +replicas for the echo server. While we change the resources value for the +production. This structure allow us to avoid duplicating the helmrelease.yaml +base definition and only modify the value we want for base on each environment. + +We end up with that structure for our echo-server. + +```shell +tenant-tpl/echo-server(main)$ tree +. +├── base +│   ├── helmrelease.yaml +│   └── kustomization.yaml +├── ks.yaml +└── overlays + ├── tenant-tpl + │   ├── kustomization.yaml + │   └── patch-resources.yaml + └── tenant-tpl-dev + ├── kustomization.yaml + └── patch-replicas.yaml +``` +##### Analysis and proof +Our goal was to define a main helmrelease for the echo server and change some +parameters according to the environments. We increase the number of replicas +for dev and change the amount of resources for prod. + +First look to the flux side to proof reconciliation and versioning. + +*tenant-tpl* +```shell +(venv) euler@HAL:~/.../tenant-tpl/echo-server(main)$ flux get all -n tenant-tpl +NAME REVISION SUSPENDED READY MESSAGE +gitrepository/tenant-repos main@sha1:3f1afa0f False True stored artifact for revision 'main@sha1:3f1afa0f' + +NAME REVISION SUSPENDED READY MESSAGE +helmrepository/bjw-s False True Helm repository is Ready + +NAME REVISION SUSPENDED READY MESSAGE +helmchart/tenant-tpl-echo-server 3.2.1 False True pulled 'app-template' chart with version '3.2.1' + +NAME REVISION SUSPENDED READY MESSAGE +helmrelease/echo-server 3.2.1 False True Helm upgrade succeeded for release tenant-tpl/echo-server.v2 with chart app-template@3.2.1 + +NAME REVISION SUSPENDED READY MESSAGE +kustomization/echo-server main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/repos-sync main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/tenant-apps main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/tenant-config main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/vars main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +``` + +*tenant-tpl-dev* + +```shell +(venv) euler@HAL:~/.../tenant-tpl/echo-server(main)$ flux get all -n tenant-tpl-dev +NAME REVISION SUSPENDED READY MESSAGE +gitrepository/tenant-repos main@sha1:3f1afa0f False True stored artifact for revision 'main@sha1:3f1afa0f' + +NAME REVISION SUSPENDED READY MESSAGE +helmrepository/bjw-s False True Helm repository is Ready + +NAME REVISION SUSPENDED READY MESSAGE +helmchart/tenant-tpl-dev-echo-server 3.2.1 False True pulled 'app-template' chart with version '3.2.1' + +NAME REVISION SUSPENDED READY MESSAGE +helmrelease/echo-server 3.2.1 False True Helm upgrade succeeded for release tenant-tpl-dev/echo-server.v2 with chart app-template@3.2.1 + +NAME REVISION SUSPENDED READY MESSAGE +kustomization/echo-server main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/repos-sync main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/tenant-apps main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/tenant-config main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +kustomization/vars main@sha1:3f1afa0f False True Applied revision: main@sha1:3f1afa0f +``` + +Both are at the same versioning on `main@sha1:3f1afa0f'` Now inspecting the +echo-server helmrelease. + +```shell +$ flux trace hr echo-server -n tenant-tpl-dev + + Object: HelmRelease/echo-server + Namespace: tenant-tpl-dev + Status: Managed by Flux + --- + Kustomization: echo-server + Namespace: tenant-tpl-dev + Target: tenant-tpl-dev + Path: ./echo-server/overlays/tenant-tpl-dev + Revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + Status: Last reconciled at 2025-09-15 13:13:04 +0200 CEST + Message: Applied revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + --- + GitRepository: tenant-repos + Namespace: tenant-tpl-dev + URL: https://git.kvant.cloud/phoenix-oss/tenant-tpl + Branch: main + Revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + Status: Last reconciled at 2025-09-15 13:12:46 +0200 CEST + Message: stored artifact for revision 'main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046' + +$ flux trace hr echo-server -n tenant-tpl + + Object: HelmRelease/echo-server + Namespace: tenant-tpl + Status: Managed by Flux + --- + Kustomization: echo-server + Namespace: tenant-tpl + Target: tenant-tpl + Path: ./echo-server/overlays/tenant-tpl + Revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + Status: Last reconciled at 2025-09-15 13:12:21 +0200 CEST + Message: Applied revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + --- + GitRepository: tenant-repos + Namespace: tenant-tpl + URL: https://git.kvant.cloud/phoenix-oss/tenant-tpl + Branch: main + Revision: main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046 + Status: Last reconciled at 2025-09-15 13:12:07 +0200 CEST + Message: stored artifact for revision 'main@sha1:3f1afa0f19882e2c6acdf318b6a6d0195e24d046' + +``` +As you can see in the `path` hold a different value based on the environments. +Thanks to our main kustomization that defined the path using +`${TENANT_NAMESPACE}` value. Each overlays include the base but applied a patch +on it before sending it to kube. + +Let's verify that our patch are correctly applied. On dev we wanted 4 replicas. +```shell +$ oc get pods -n tenant-tpl-dev + NAME READY STATUS RESTARTS AGE + echo-server-5bd6b558d6-4nngj 1/1 Running 0 10m + echo-server-5bd6b558d6-65zlb 1/1 Running 0 10m + echo-server-5bd6b558d6-ks2fq 1/1 Running 0 3d19h + echo-server-5bd6b558d6-mz2wf 1/1 Running 0 3d19h + echo-server-5bd6b558d6-rm64f 1/1 Running 0 10m +``` +4 replicas ✔ + +On tenant-tpl we wanted to increase the resources. + +*DEV* +```shell + +$ oc get pods -n tenant-tpl-dev -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{.name}{" CPU Requests: "}{.resources.requests.cpu}{" Memory Requests: "}{.resources.requests.memory}{"\n"}{end}{end}' + echo-server-5bd6b558d6-4nngj + app CPU Requests: 10m Memory Requests: 64Mi + echo-server-5bd6b558d6-65zlb + app CPU Requests: 10m Memory Requests: 64Mi + echo-server-5bd6b558d6-ks2fq + app CPU Requests: 10m Memory Requests: 64Mi + echo-server-5bd6b558d6-mz2wf + app CPU Requests: 10m Memory Requests: 64Mi + echo-server-5bd6b558d6-rm64f + app CPU Requests: 10m Memory Requests: 64Mi +``` +*PROD* + +```shell +$ oc get pods -n tenant-tpl -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}{.name}{" CPU Requests: "}{.resources.requests.cpu}{" Memory Requests: "}{.resources.requests.memory}{"\n"}{end}{end}' + echo-server-6cc5465c7d-5kg5r + app CPU Requests: 100m Memory Requests: 128Mi + echo-server-6cc5465c7d-dsmb6 + app CPU Requests: 100m Memory Requests: 128Mi + +``` + +We confirm that we have different resources. Our patch properly work and we now +achieve to have a comon base and patching fields according to the environments +✔ + +### Find the file reference: + +## Base +- [helmrelease.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/base/helmrelease.yaml) +- [kustomization.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/base/kustomization.yaml) + +## Main Kustomization +- [ks.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/ks.yaml) + +## Overlays + +### tenant-tpl +- [kustomization.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/overlays/tenant-tpl/kustomization.yaml) +- [patch-resources.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/overlays/tenant-tpl/patch-resources.yaml) + +### tenant-tpl-dev +- [kustomization.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/overlays/tenant-tpl-dev/kustomization.yaml) +- [patch-replicas.yaml](https://git.kvant.cloud/phoenix-oss/tenant-tpl/-/blob/main/echo-server/overlays/tenant-tpl-dev/patch-replicas.yaml) + +### Link Reference +- [FLUX Repository Structure ](https://fluxcd.io/flux/guides/repository-structure/#repository-structure-2) +