Azure Application Gateway for Containers - First Look
Microsoft recently announced a preview of Azure Application Gateway for Containers, so we thought we'd have a look at the early release and see what it's all about.
The product is described as "the evolution" of the Application Gateway Ingress Controller (AGIC) product, and provides an enterprise-grade ingress solution for Kubernetes services running in Azure. Fundamentally it operates in a similar fashion by linking Kubernetes CRDs to ARM resources, however, there are a number of architectural differences between the products:
The Azure resource is now a new Application Gateway for Containers instance rather than an Application Gateway
It is now possible to let the controller manage the Azure resources directly rather than having to spin this up prior to the controller installation
A bunch of new CRD types exist that offer support for advanced ingress services, such as mTLS authentication, Gateway API integration and other traffic management features
Sounds good doesn't it? Let's dive in.
Fundamentals
The Microsoft documentation covers the basics on the new components. Let's summarise how this all hangs together:
The
ALB Controllerruns within the cluster and is responsible for deployment (optionally) and configuration of theApplication Gateway for Containers(AGC) ARM resourceThe controller is integrated with
Azure Workload Identitywhich enables the controller to authorize against the Azure API to update ARM resourcesIngress endpoints (known as
Frontends) are provided via the AGC resource which is configured with one or more service endpoints within the clusterThe AGC instance is associated with one or more delegated subnets (known as
Associations) that provide connectivity from external clients
It’s worth mentioning that the Application Gateway for Containers is not a direct replacement for the Application Gateway. It doesn’t provide any WAF capabilities and currently won’t support private Frontend resources, although there is support for TLS policy in the current preview.
Deployment
Before starting, it's worth noting that the AGC offering is only available in selected regions at the moment, so ensure that you are deploying into one of these before you start. For our tests, we used the UK South region.
The Microsoft documentation provides a walkthrough of the deployment steps, and we'd recommend having a good scan through these to understand the process. We'll touch on the salient points below, but we'll assume that you'll use the official documentation to get the controller up and running in your cluster.
Azure Prerequisites
The ALB controller has some pre-requisites that are covered in the official documentation:
A number of provider registrations are required before the ALB resource are available via the Azure API. To register via the Azure CLI:
# Sign in to your Azure subscription.
SUBSCRIPTION_ID='<your subscription id>'
az login
az account set --subscription $SUBSCRIPTION_ID
# Register required resource providers on Azure.
az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.Network
az provider register --namespace Microsoft.NetworkFunction
az provider register --namespace Microsoft.ServiceNetworkingThe controller needs at least one dedicated Azure subnet that has connectivity into the AKS cluster. This subnet requires the
Microsoft.ServiceNetworking/trafficControllersdelegation, and can’t be used for any other services.We need to create a user-managed identity which is then federated to the controller service account. The identity requires the following permissions:
| Role | Target |
|---|---|
Reader |
AKS managed (nodes) resource group |
AppGw for Containers Configuration Manager |
AKS managed (nodes) resource group |
Network Contributor |
ALB subnet |
The identity should also have a service account federation linked to the AKS OIDC Issuer. By default, the subject would be
system:serviceaccount:azure-alb-system:alb-controller-sa.Azure Workload Identity must be installed and configured on the cluster before installing the ALB controller. This can be installed via the CLI or via Helm, as per our recent blog.
Installing the ALB Controller
The controller installation comes down to a simple helm command if all the pre-requisites are completed:
helm install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \
--version 0.4.023971 \
--set albController.podIdentity.clientID=<YOUR_UMI_CLIENT_ID>We can confirm all is running by checking the controller pods and the new CRDs:
kubectl get pods -n azure-alb-system NAME READY STATUS RESTARTS AGE alb-controller-65d4b898bd-7qhrc 1/1 Running 0 88m alb-controller-bootstrap-657b5d9c47-4v8q4 1/1 Running 0 90m kubectl get crds | grep -e alb.networking.azure.io -e networking.k8s.io applicationloadbalancer.alb.networking.azure.io 2023-08-16T07:31:01Z backendtlspolicies.alb.networking.azure.io 2023-08-16T07:31:01Z frontendtlspolicies.alb.networking.azure.io 2023-08-16T07:31:01Z gatewayclasses.gateway.networking.k8s.io 2023-08-16T07:31:01Z gateways.gateway.networking.k8s.io 2023-08-16T07:31:01Z healthcheckpolicy.alb.networking.azure.io 2023-08-16T07:31:01Z httproutes.gateway.networking.k8s.io 2023-08-16T07:31:01Z ingressextension.alb.networking.azure.io 2023-08-16T07:31:01Z referencegrants.gateway.networking.k8s.io 2023-08-16T07:31:01Z routepolicies.alb.networking.azure.io 2023-08-16T07:31:01Z
Custom resources
If you want to follow along, make sure you replace the host.example.com references with your public DNS names - especially important when using Cert-Manager to validate your certificate requests!
The ALB Resource
So far we've configured our cluster ready to consume AGC instances, but we haven't actually configured one. Whilst we could deploy our Azure resource outside of Kubernetes (the BYOD method), one of the killer features of the new controller is that it can take care of deploying our Azure resources for us, so let's do that.
At this point it's worth checking that you've provided the necessary RBAC permissions to your managed identity (we covered these earlier). If you do get issues with the controller it's most likely going to be related to this.
The YAML below, will create a new ApplicationLoadBalancer instance in a dedicated namespace. Make sure you replace the subnet ID for your delegated subnet in the associations section.
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: alb-ingress-system
---
apiVersion: alb.networking.azure.io/v1
kind: ApplicationLoadBalancer
metadata:
name: alb-ingress
namespace: alb-ingress-system
spec:
associations:
- <INSERT_YOUR_ALB_SUBNET_ID_HERE>
EOF
namespace/alb-test-infra created
applicationloadbalancer.alb.networking.azure.io/alb-ingress createdIf we keep an eye on the ALB controller logs, we should see the controller spin up the AGC resource in our target subscription and associate with the ALB subnet:
kubectl logs svc/alb-controller -n azure-alb-system -f ... - Creating Application Gateway for Containers resource **** from CRD **** - Successfully created Application Gateway for Containers resource **** - Getting association **** for Application Gateway for Containers resource **** - Association not found for Application Gateway for Containers resource **** - Creating association **** for Application Gateway for Containers resource **** ...
Using the Ingress API
Now we have our controller up and running, let's explore what happens when we create an Ingress resource linked to the controller.
Let's deploy the Microsoft demo into a new
test-infranamespace:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: test-infra
---
apiVersion: v1
kind: Service
metadata:
labels:
app: echo
name: echo
namespace: test-infra
spec:
ports:
- port: 80
protocol: TCP
targetPort: 3000
selector:
app: echo
sessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: echo
name: echo
namespace: test-infra
spec:
replicas: 1
selector:
matchLabels:
app: echo
template:
metadata:
labels:
app: echo
spec:
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20220815-e21d1a4
name: echo
lifecycle:
preStop:
exec:
command: ["sleep", "10"]
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
EOF
namespace/test-infra created
service/echo created
deployment.apps/echo createdNow let's deploy an ingress. In this example, we're using Cert-Manager and External-DNS to apply SSL certificates and DNS hostnames respectively. Your issuers and challenge types may well differ, but this should give you an idea:
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-01
namespace: test-infra
annotations:
alb.networking.azure.io/alb-name: alb-ingress
alb.networking.azure.io/alb-namespace: alb-ingress-system
cert-manager.io/acme-challenge-type: dns01
cert-manager.io/cluster-issuer: letsencrypt-azure
spec:
ingressClassName: azure-alb-external
tls:
- hosts:
- test.example.com
secretName: ingress-01-tls-secret
rules:
- host: test.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: echo
port:
number: 80
EOFThe SSL certificates are issued and we can test our application by hitting the external URL (make sure you change the test.example.com in these examples if you want to test).
It's great that we can use the Ingress API with the same tools we've always used and get the same results. The other thing that has quickly become apparent is the improvement in performance over the AGIC implementation - the new resource clearly updates much faster than the AGIC implementation does.
Using the Gateway API
So far, we haven't done anything that we couldn't have achieved with the AGIC solution. This changes when we start exploring the Gateway API, which is new to AGC and would appear to be one of the key benefits in using the new controllers.
Whilst the Gateway API is still under development (the AGC supports API version v1beta), it has been adopted by a number of key CNCF vendor products and is likely to replace the legacy Ingress API in the near (ish) future.
In the new model, HTTPRoute resources are roughly analogous with Ingress resources, whereas the Gateway is a new concept and relates to the Layer 7 routing specification, allowing a myriad of use cases for the API.
Let's do a simple test to mirror the Ingress API example we looked at previously. For this to work, we need to make sure that our Cert-Manager deployment is up-to-date, and has the Gateway API (experimental) support enabled. We also need to update and enable External-DNS to support our HTTPRoute object.
With that done, let's run the equivalent deployment:
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: gateway-01
namespace: test-infra
annotations:
alb.networking.azure.io/alb-name: alb-ingress
alb.networking.azure.io/alb-namespace: alb-ingress-system
cert-manager.io/acme-challenge-type: dns01
cert-manager.io/cluster-issuer: letsencrypt-azure
spec:
gatewayClassName: azure-alb-external
listeners:
- name: http
hostname: test.example.com
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: All
tls:
mode: Terminate
certificateRefs:
- name: gateway-01-tls-secret
kind: Secret
group: ""
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: https-route
namespace: test-infra
spec:
parentRefs:
- name: gateway-01
rules:
- backendRefs:
- name: echo
port: 80
EOFThere's a bit of lag between configuring the Gateway and the resource returning Programmed status (i.e. the Azure implementation is complete) but only around 60 seconds at most during the tests - still a marked improvement over AGIC and understandable considering the backend updates that are taking place.
We can check the status of the HTTPRoute to ensure that all is well (jq is only here to make the JSON pretty!):
kubectl get httproute/https-route -n test-infra -o jsonpath='{.status}' | jq .
{
"parents": [
{
"conditions": [
{
"lastTransitionTime": "2023-08-16T15:36:03Z",
"message": "",
"observedGeneration": 1,
"reason": "ResolvedRefs",
"status": "True",
"type": "ResolvedRefs"
},
{
"lastTransitionTime": "2023-08-16T15:36:03Z",
"message": "Route is Accepted",
"observedGeneration": 1,
"reason": "Accepted",
"status": "True",
"type": "Accepted"
},
{
"lastTransitionTime": "2023-08-16T15:36:04Z",
"message": "Application Gateway for Containers resource has been successfully updated.",
"observedGeneration": 1,
"reason": "Programmed",
"status": "True",
"type": "Programmed"
}
],
"controllerName": "alb.networking.azure.io/alb-controller",
"parentRef": {
"group": "gateway.networking.k8s.io",
"kind": "Gateway",
"name": "gateway-01"
}
}
]
}Closing thoughts
First impressions of the new AGC implementation are favourable, and there are some clear advantages to implementing this solution over the legacy AGIC pattern. We particularly like the zero-touch Azure Gateway for Containers ARM resource deployment model which feels like a big step towards a true cloud-native service. Whilst the BYOD approach is supported, there probably aren't many use cases where you would want to use it - possibly in highly-regulated environments that require strict separation of duties.
The support for both Ingress and Gateway APIs means that you can adopt the technology now and move towards the Gateway API once it matures. It's a good feature and should prove popular with those that have already built up patterns around the Ingress API.
To temper the enthusiasm a little, we probably won't be rushing to deploy the controller into production environments just yet. The Helm charts are quite limited at the moment, and the fluid nature of the Gateway API is likely to result in a high number of code updates in the short term. We would also want to test some of the more advanced features and see whether these would replace or compliment the more mature features of existing products such as Istio and Contour.
Azure Application Gateway for Containers is definitely a product we'll be keeping a close eye on over the coming months and is likely to be the go-to ingress solution for AKS customers as it moves towards a GA release.
