Traefik as ingress for Azure Kubernetes Service
In my previous AKS posts I’ve used Azure Application Gateway Ingress Controller (AGIC) for ingress, but you may instead want to use Traefik. Traefik has a larger community and better documentation, and is cloud agnostic. In this post I’ll show how to bootstrap the cluster with Traefik, including exposing the Traefik dashboard with a secret from Azure Key Vault. The secret is synced using Azure Key Vault CSI driver and Azure authentication is provided by Azure AD Workload Identity.
Assuming you have your favorite CD pipeline set up with the possibility to pass variables and secrets, the first step is to enable OIDC issuer on the AKS cluster for usage with Azure AD Workload Identity.
az extension add --name aks-preview > /dev/null
az extension update --name aks-preview > /dev/null
az aks update -n $AKS_NAME -g $AZURE_RG --enable-oidc-issuer > /dev/null
This next step could be automated, but I opted not to, as it would require minimum Application.ReadWrite.All to Microsoft Graph API. What you need to do is create an Azure AD Application, grab the OIDC issuer URL, and create the federated credential. See Microsoft docs for federation example, and use traefik as namespace and traefik as service account name. OIDC issuer URL can be retrieved using Azure CLI.
az aks show --resource-group $AZURE_RG --name $AKS_NAME --query "oidcIssuerProfile.issuerUrl" -otsv
Next step is to install the Azure Key Vault Provider for Secrets Store CSI Driver, passing the value “syncSecret.enabled=true” to enable the functionality to sync Key Vault secrets to Kubernetes secrets, which is needed for usage with Basic auth in Traefik Middleware.
# Download helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
# Install Key Vault CSI Driver
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm upgrade --install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system \
--set "secrets-store-csi-driver.syncSecret.enabled=true"
Next up is generating the basic auth credentials and placing it in a Key Vault secret if it doesn’t exist. The script checks if the secret exists, and if it doesn’t exist, randomly generates a password and uses htpasswd to generate the basic auth credentials (with traefikadmin as username). Then, the script generates two Key Vault secrets, one with the password in clear text and one with the htpasswd credentials.
Afterwards, it creates the traefik namespace and generates the SecretProviderClass which syncs the Key Vault secret with a Kubernetes secret using Workload Identity. $WID_CLIENTID is the Client ID of the previously created AAD App which is federated with the traefik namespace in your cluster. The AAD App has to have secrets get permission to the Key Vault.
# Generate Traefik dashboard login and add to Key Vault
kvname=$(az keyvault list --query "[?resourceGroup == '$AZURE_RG'].name" --output tsv)
az keyvault secret show --name traefikpw --vault-name $kvname --query "attributes.created" -otsv
if [ $? -ne 0 ]
then
apt install apache2-utils
pw=$(tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' </dev/urandom | head -c 13 ; echo)
htpw=$(htpasswd -nb traefikadmin $pw)
az keyvault secret set --name TraefikAdmin --vault-name $kvname --value $pw > /dev/null
az keyvault secret set --name traefikpw --vault-name $kvname --value $htpw > /dev/null
fi
# Generate secret sync from Key Vault using AAD Workload Identity
secretsyncvar="apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: traefik-azure-sync
namespace: traefik
spec:
provider: azure
secretObjects:
- secretName: traefikpw
type: Opaque
data:
- objectName: traefikpw
key: users
parameters:
clientID: \"$WID_CLIENTID\"
keyvaultName: \"$KV_NAME\"
objects: |
array:
- |
objectName: traefikpw
objectType: secret
tenantId: \"$AZURE_TENANT\""
# Deploy k8s secret synced from key vault
kubectl create namespace traefik --dry-run=client -o json | kubectl apply -f -
echo "$secretsyncvar" | kubectl apply -f -
To deploy Traefik, we need to create a yaml-file called traefik-values.yaml which will be used to pass values to the helm deployment. While mounting the Kubernetes secret to the Traefik pod doesn’t really make sense, it’s required because Azure Key Vault CSI Driver doesn’t sync a Key Vault secret to a Kubernetes secret unless mounted to a pod, see “NOTE” in the documentation.
deployment:
additionalVolumes:
- name: traefikpw
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "traefik-azure-sync"
additionalVolumeMounts:
- name: traefikpw
mountPath: "/mnt/traefikauth"
readOnly: true
With the values file in place, we can deploy Traefik. $TRAEFIK_IP is an Azure Public IP to be used for Traefik. This Public IP has to be created beforehand and should reside in $AZURE_RG. The AKS identity needs Network Contributor to the Resource Group (if you’re using Bicep, it’s the <aks-resource>.identity.principalId identity). $DNS is your domain name, for example eldar.cloud.
The final piece of the puzzle is the manifests for exposing the Traefik dashboard and tying it all together. The first manifest is the Middleware using the Kubernetes secret which is synced from Key Vault, while the second is the actual IngressRoute which routes requests for yourdomain.com/api and yourdomain.com/dashboard/ to the Traefik dashboard.
# Install traefik
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm upgrade --install traefik traefik/traefik --namespace traefik \
-f traefik-values.yaml --set "service.spec.loadBalancerIP=$TRAEFIK_IP" \
--set "service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-resource-group=$AZURE_RG"
# Generate traefik dashboard manifests
traefikDashboardIngress="apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: traefik-dashboard-basicauth
namespace: traefik
spec:
basicAuth:
secret: traefikpw
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: traefik
spec:
entryPoints:
- web
routes:
- match: Host(\`traefik.$DNS\`) && (PathPrefix(\`/api\`) || PathPrefix(\`/dashboard\`))
kind: Rule
middlewares:
- name: traefik-dashboard-basicauth
namespace: traefik
services:
- name: api@internal
kind: TraefikService"
# Deploy traefik ingress
echo "$traefikDashboardIngress" | kubectl apply -f -
Please note that as of writing, Azure AD Workload Identity is in preview and shouldn’t be used in production — but it beats using Azure AD Pod Identity and is considered its successor.
One Reply to “Traefik as ingress for Azure Kubernetes Service”