Microsoft recently announced a preview of Azure Application Gateway for Containers, so we thought we'd have a look at the early release and see what it's all about.

The product is described as "the evolution" of the Application Gateway Ingress Controller (AGIC) product, and provides an enterprise-grade ingress solution for Kubernetes services running in Azure. Fundamentally it operates in a similar fashion by linking Kubernetes CRDs to ARM resources, however, there are a number of architectural differences between the products:

• The Azure resource is now a new Application Gateway for Containers instance rather than an Application Gateway

• It is now possible to let the controller manage the Azure resources directly rather than having to spin this up prior to the controller installation

• A bunch of new CRD types exist that offer support for advanced ingress services, such as mTLS authentication, Gateway API integration and other traffic management features

Sounds good doesn't it? Let's dive in.

Fundamentals

The Microsoft documentation covers the basics on the new components. Let's summarise how this all hangs together:

• The ALB Controller runs within the cluster and is responsible for deployment (optionally) and configuration of the Application Gateway for Containers (AGC) ARM resource

• The controller is integrated with Azure Workload Identity which enables the controller to authorize against the Azure API to update ARM resources

• Ingress endpoints (known as Frontends) are provided via the AGC resource which is configured with one or more service endpoints within the cluster

• The AGC instance is associated with one or more delegated subnets (known as Associations) that provide connectivity from external clients

It’s worth mentioning that the Application Gateway for Containers is not a direct replacement for the Application Gateway. It doesn’t provide any WAF capabilities and currently won’t support private Frontend resources, although there is support for TLS policy in the current preview.

Deployment

Before starting, it's worth noting that the AGC offering is only available in selected regions at the moment, so ensure that you are deploying into one of these before you start. For our tests, we used the UK South region.

The Microsoft documentation provides a walkthrough of the deployment steps, and we'd recommend having a good scan through these to understand the process. We'll touch on the salient points below, but we'll assume that you'll use the official documentation to get the controller up and running in your cluster.

Azure Prerequisites

The ALB controller has some pre-requisites that are covered in the official documentation:

• A number of provider registrations are required before the ALB resource are available via the Azure API. To register via the Azure CLI:

• The controller needs at least one dedicated Azure subnet that has connectivity into the AKS cluster. This subnet requires the Microsoft.ServiceNetworking/trafficControllers delegation, and can’t be used for any other services.

• We need to create a user-managed identity which is then federated to the controller service account. The identity requires the following permissions:

• The identity should also have a service account federation linked to the AKS OIDC Issuer. By default, the subject would be system:serviceaccount:azure-alb-system:alb-controller-sa.

• Azure Workload Identity must be installed and configured on the cluster before installing the ALB controller. This can be installed via the CLI or via Helm, as per our recent blog.

Installing the ALB Controller

The controller installation comes down to a simple helm command if all the pre-requisites are completed:

We can confirm all is running by checking the controller pods and the new CRDs:

Custom resources

If you want to follow along, make sure you replace the host.example.com references with your public DNS names - especially important when using Cert-Manager to validate your certificate requests!

The ALB Resource

So far we've configured our cluster ready to consume AGC instances, but we haven't actually configured one. Whilst we could deploy our Azure resource outside of Kubernetes (the BYOD method), one of the killer features of the new controller is that it can take care of deploying our Azure resources for us, so let's do that.

At this point it's worth checking that you've provided the necessary RBAC permissions to your managed identity (we covered these earlier). If you do get issues with the controller it's most likely going to be related to this.

The YAML below, will create a new ApplicationLoadBalancer instance in a dedicated namespace. Make sure you replace the subnet ID for your delegated subnet in the associations section.

If we keep an eye on the ALB controller logs, we should see the controller spin up the AGC resource in our target subscription and associate with the ALB subnet:

Using the Ingress API

Now we have our controller up and running, let's explore what happens when we create an Ingress resource linked to the controller.

• Let's deploy the Microsoft demo into a new test-infra namespace:

• Now let's deploy an ingress. In this example, we're using Cert-Manager and External-DNS to apply SSL certificates and DNS hostnames respectively. Your issuers and challenge types may well differ, but this should give you an idea:

The SSL certificates are issued and we can test our application by hitting the external URL (make sure you change the test.example.com in these examples if you want to test).

It's great that we can use the Ingress API with the same tools we've always used and get the same results. The other thing that has quickly become apparent is the improvement in performance over the AGIC implementation - the new resource clearly updates much faster than the AGIC implementation does.

Using the Gateway API

So far, we haven't done anything that we couldn't have achieved with the AGIC solution. This changes when we start exploring the Gateway API, which is new to AGC and would appear to be one of the key benefits in using the new controllers.

Whilst the Gateway API is still under development (the AGC supports API version v1beta), it has been adopted by a number of key CNCF vendor products and is likely to replace the legacy Ingress API in the near (ish) future.

In the new model, HTTPRoute resources are roughly analogous with Ingress resources, whereas the Gateway is a new concept and relates to the Layer 7 routing specification, allowing a myriad of use cases for the API.

Let's do a simple test to mirror the Ingress API example we looked at previously. For this to work, we need to make sure that our Cert-Manager deployment is up-to-date, and has the Gateway API (experimental) support enabled. We also need to update and enable External-DNS to support our HTTPRoute object.

With that done, let's run the equivalent deployment:

There's a bit of lag between configuring the Gateway and the resource returning Programmed status (i.e. the Azure implementation is complete) but only around 60 seconds at most during the tests - still a marked improvement over AGIC and understandable considering the backend updates that are taking place.

We can check the status of the HTTPRoute to ensure that all is well (jq is only here to make the JSON pretty!):

Closing thoughts

First impressions of the new AGC implementation are favourable, and there are some clear advantages to implementing this solution over the legacy AGIC pattern. We particularly like the zero-touch Azure Gateway for Containers ARM resource deployment model which feels like a big step towards a true cloud-native service. Whilst the BYOD approach is supported, there probably aren't many use cases where you would want to use it - possibly in highly-regulated environments that require strict separation of duties.

The support for both Ingress and Gateway APIs means that you can adopt the technology now and move towards the Gateway API once it matures. It's a good feature and should prove popular with those that have already built up patterns around the Ingress API.

To temper the enthusiasm a little, we probably won't be rushing to deploy the controller into production environments just yet. The Helm charts are quite limited at the moment, and the fluid nature of the Gateway API is likely to result in a high number of code updates in the short term. We would also want to test some of the more advanced features and see whether these would replace or compliment the more mature features of existing products such as Istio and Contour.

Azure Application Gateway for Containers is definitely a product we'll be keeping a close eye on over the coming months and is likely to be the go-to ingress solution for AKS customers as it moves towards a GA release.

Useful Links

• Microsoft - AGC Installation

• BlakYaks - Azure Workload Identity

• Kubernetes - Gateway API