ArgoCD App of Apps for infra provisioning
You’re finally done bootstrapping your cluster with minimum required components, like ArgoCD, and your ingress or even multiple ingresses. At this point you want to stop deploying anything to your cluster using kubectl apply and similar. You want to use your ArgoCD deployment to deploy using the GitOps approach. This includes using GitOps for infrastructure provisioning, for example to deploy grafana.
ArgoCD has an App of Apps pattern for installing a set of apps to your cluster using various methods like helm charts, kustomize, plain manifests etc. The general idea is to create a single repo which contains declarative application references to other repos (helm charts etc.), which is then added to ArgoCD, and in turn deploys the applications.
Let’s start by creating a repo with the file /apps/Chart.yaml. This serves as the “main” application in ArgoCD, essentially the App of the App of Apps.
apiVersion: v2
name: applications
description: Applications
type: application
version: 0.1.0
appVersion: "1.0"
If the repository is a private repository, create a SSH key pair, and add the public key as a GitHub Deploy Key. Base64 encode the private key, and add it as a GitHub Repository Secret to your bootstrapping repository (the repository where you deploy ArgoCD to your cluster). Finally, add the following to your CD pipeline in your bootstrapping repository. $GH_SSHPRIV is your base64 encoded private key, and $GH_PATH is the path to your repository (e.g. EldarBorge/aks-poc).
# Add Infra GitOps repo for further configuration
echo "$GH_SSHPRIV" | base64 -d | ./argocd repo add git@github.com:$GH_PATH \
--insecure-skip-server-verification --ssh-private-key-path /dev/stdin
./argocd app create infra-gitops \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc \
--repo git@github.com:$GH_PATH \
--path apps
./argocd app sync infra-gitops
sleep 30
./argocd app sync -l argocd.argoproj.io/instance=infra-gitops
Now that the App of the App of Apps is created, and automatically added to ArgoCD, we can start defining applications which will be automatically deployed with GitOps. The procedure is to create the namespace, and add a yaml-file to the templates-folder which defines the repo and sets any custom values required.
Let’s deploy Grafana as an example. Create /apps/templates/namespaces.yaml which instructs ArgoCD to create a new namespace for grafana. Sync-wave -1 tells ArgoCD this resource needs to be deployed first.
apiVersion: v1
kind: Namespace
metadata:
name: grafana
annotations:
argocd.argoproj.io/sync-wave: "-1"
Next up is creating /apps/templates/grafana.yaml. This tells ArgoCD we want to deploy a helm chart, where to locate the chart, and which values to define. In this example, I’m using Key Vault CSI Driver to use existing Key Vault secrets as the login for Grafana, and create an IngressRoute to enable public access to Grafana.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: grafana
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
chart: grafana
repoURL: https://grafana.github.io/helm-charts
targetRevision: "*"
helm:
releaseName: grafana
values: |
extraSecretMounts:
- name: secrets-store-inline
mountPath: /mnt/secrets-store
readOnly: true
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "grafana-kv"
admin:
existingSecret: grafana-admin
userKey: admin-user
passwordKey: admin-password
serviceAccount:
name: grafana
extraObjects:
- apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: grafana
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host(`grafana.eldar.cloud`)
services:
- name: grafana
port: 80
- apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: grafana-kv
spec:
provider: azure
secretObjects:
- secretName: grafana-admin
type: Opaque
data:
- objectName: <key vault secret name for username>
key: admin-user
- objectName: <key vault secret name for password>
key: admin-password
parameters:
clientID: "<azure ad workload identity client id>"
keyvaultName: "<key vault name>"
objects: |
array:
- |
objectName: <key vault secret name for username>
objectType: secret
- |
objectName: <key vault secret name for password>
objectType: secret
tenantId: "<tenant guid>"
destination:
server: "https://kubernetes.default.svc"
namespace: grafana