AKS Network: Azure CNI Overlay & Azure CNI Powered by Cilium

Short blog article on a couple of noteworthy features in Azure AKS that recently became generally available. Azure CNI Overlay & Azure CNI Powered by Cilium.

AKS CNI

Up until recently you only really had 2 options for your Kubernetes CNI in AKS, Kubenet or Azure CNI. With Kubenet your Nodes received an IP address from your subnet and your pods received an IP address from a logically different address space to the Azure virtual network subnet of the nodes:

This is nice and simple and only requires a few IP addresses from your subnet. It does come with some limitations though and the Microsoft best practice docs gently steer you toward Azure CNI for production workloads. With Azure CNI every pod on a node gets an IP address from the subnet:

Not only do the pods consume IP addresses from the subnet but the service pre-provisions all of the IPs from the subnet when nodes come online. So if you have set your maximum pods per node to 100 then AKS will pre-provision 100 IPs when each node comes online. The threat of IP exhaustion is very real if you didn't get your calculator out and plan your IP addressing properly.

Azure CNI Overlay

Azure CNI Overlay essentially uses the same approach as Kubenet and uses a logically different address space for your pods:

So we now have a production grade option with a much simpler design. Quite a big win. It gets us around some of the limitations of Kubenet such as Windows nodepool support, but it does have it's own limitations such as not being able to use Application Gateway Ingress Controller (AGIC).

Azure CNI Powered by Cilium

This is Microsoft's next-gen offering for AKS CNI backed by Cilium. It is functionality equivalent to Azure CNI and can be deployed in 2 modes, dynamic IP assignment (like Azure CNI) or overlay (like Azure CNI Overlay). This is likely to be the go-to for new cluster designs but, as always, it has it's own limitations. For example; it only supports Linux nodepools and doesn't currently work with sidecar-based service meshes like Istio (fairly hefty issue but looks like a fix is in the post).

Conclusion

Some significant improvements have been made to the CNI options for AKS, but one thing remains unchanged: getting your CNI configuration correct is still one of the most critical design decisions for AKS.