Traefik as ingress for Azure Kubernetes Service

In my previous AKS posts I’ve used Azure Application Gateway Ingress Controller (AGIC) for ingress, but you may instead want to use Traefik. Traefik has a larger community and better documentation, and is cloud agnostic. In this post I’ll show how to bootstrap the cluster with Traefik, including exposing the Traefik dashboard with a secret from Azure Key Vault. The secret is synced using Azure Key Vault CSI driver and Azure authentication is provided by Azure AD Workload Identity.

Assuming you have your favorite CD pipeline set up with the possibility to pass variables and secrets, the first step is to enable OIDC issuer on the AKS cluster for usage with Azure AD Workload Identity.

az extension add --name aks-preview > /dev/null
az extension update --name aks-preview > /dev/null
az aks update -n $AKS_NAME -g $AZURE_RG --enable-oidc-issuer > /dev/null

This next step could be automated, but I opted not to, as it would require minimum Application.ReadWrite.All to Microsoft Graph API. What you need to do is create an Azure AD Application, grab the OIDC issuer URL, and create the federated credential. See Microsoft docs for federation example, and use traefik as namespace and traefik as service account name. OIDC issuer URL can be retrieved using Azure CLI.

az aks show --resource-group $AZURE_RG --name $AKS_NAME --query "oidcIssuerProfile.issuerUrl" -otsv

Next step is to install the Azure Key Vault Provider for Secrets Store CSI Driver, passing the value “syncSecret.enabled=true” to enable the functionality to sync Key Vault secrets to Kubernetes secrets, which is needed for usage with Basic auth in Traefik Middleware.

# Download helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

# Install Key Vault CSI Driver 
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm upgrade --install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system \
--set "secrets-store-csi-driver.syncSecret.enabled=true"

Next up is generating the basic auth credentials and placing it in a Key Vault secret if it doesn’t exist. The script checks if the secret exists, and if it doesn’t exist, randomly generates a password and uses htpasswd to generate the basic auth credentials (with traefikadmin as username). Then, the script generates two Key Vault secrets, one with the password in clear text and one with the htpasswd credentials.

Afterwards, it creates the traefik namespace and generates the SecretProviderClass which syncs the Key Vault secret with a Kubernetes secret using Workload Identity. $WID_CLIENTID is the Client ID of the previously created AAD App which is federated with the traefik namespace in your cluster. The AAD App has to have secrets get permission to the Key Vault.

# Generate Traefik dashboard login and add to Key Vault 
kvname=$(az keyvault list --query "[?resourceGroup == '$AZURE_RG'].name" --output tsv)
az keyvault secret show --name traefikpw --vault-name $kvname --query "attributes.created" -otsv
if [ $? -ne 0 ]
then 
  apt install apache2-utils
  pw=$(tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' </dev/urandom | head -c 13  ; echo)
  htpw=$(htpasswd -nb traefikadmin $pw)
  az keyvault secret set --name TraefikAdmin --vault-name $kvname --value $pw > /dev/null
  az keyvault secret set --name traefikpw --vault-name $kvname --value $htpw > /dev/null
fi

# Generate secret sync from Key Vault using AAD Workload Identity
secretsyncvar="apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: traefik-azure-sync
  namespace: traefik
spec:
  provider: azure
  secretObjects:                                 
    - secretName: traefikpw
      type: Opaque
      data:
        - objectName: traefikpw
          key: users
  parameters:
    clientID: \"$WID_CLIENTID\"
    keyvaultName: \"$KV_NAME\"
    objects: |
      array:
        - |
          objectName: traefikpw
          objectType: secret
    tenantId: \"$AZURE_TENANT\""

# Deploy k8s secret synced from key vault
kubectl create namespace traefik --dry-run=client -o json | kubectl apply -f -
echo "$secretsyncvar" | kubectl apply -f -

To deploy Traefik, we need to create a yaml-file called traefik-values.yaml which will be used to pass values to the helm deployment. While mounting the Kubernetes secret to the Traefik pod doesn’t really make sense, it’s required because Azure Key Vault CSI Driver doesn’t sync a Key Vault secret to a Kubernetes secret unless mounted to a pod, see “NOTE” in the documentation.

deployment:
  additionalVolumes:
    - name: traefikpw
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "traefik-azure-sync"

additionalVolumeMounts:
  - name: traefikpw
    mountPath: "/mnt/traefikauth" 
    readOnly: true

With the values file in place, we can deploy Traefik. $TRAEFIK_IP is an Azure Public IP to be used for Traefik. This Public IP has to be created beforehand and should reside in $AZURE_RG. The AKS identity needs Network Contributor to the Resource Group (if you’re using Bicep, it’s the <aks-resource>.identity.principalId identity). $DNS is your domain name, for example eldar.cloud.

The final piece of the puzzle is the manifests for exposing the Traefik dashboard and tying it all together. The first manifest is the Middleware using the Kubernetes secret which is synced from Key Vault, while the second is the actual IngressRoute which routes requests for yourdomain.com/api and yourdomain.com/dashboard/ to the Traefik dashboard.

# Install traefik 
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm upgrade --install traefik traefik/traefik --namespace traefik \
-f traefik-values.yaml --set "service.spec.loadBalancerIP=$TRAEFIK_IP" \
--set "service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-resource-group=$AZURE_RG"

# Generate traefik dashboard manifests
traefikDashboardIngress="apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-dashboard-basicauth
  namespace: traefik
spec:
  basicAuth:
    secret: traefikpw
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
    - match: Host(\`traefik.$DNS\`) && (PathPrefix(\`/api\`) || PathPrefix(\`/dashboard\`))
      kind: Rule
      middlewares:
        - name: traefik-dashboard-basicauth
          namespace: traefik
      services:
        - name: api@internal
          kind: TraefikService"

# Deploy traefik ingress
echo "$traefikDashboardIngress" | kubectl apply -f -

Please note that as of writing, Azure AD Workload Identity is in preview and shouldn’t be used in production — but it beats using Azure AD Pod Identity and is considered its successor.

One Reply to “Traefik as ingress for Azure Kubernetes Service”

Leave a Reply

Your email address will not be published.