Test limit of Pods regarding IP allocation with VPC CNI

When using VPC CNI, there are a number of things that can influence the number of pods an instance can have. Here, I won't go into details about the Kubernetes recommended limit of 100 pods per instance, neither than checking cpu/memory limits (i'll use very small pods) in order to focus on the IP allocation limits.

Our goal here will be to uses as many IP as possible on an instance.

How this works

Amazon EKS supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes. By default every instance can have multiple Elastic Network Interface (ENI), and are attached one by default called the primary ENI. Any additional network interface attached to the instance is called a secondary network interface. Each network interface can be assigned multiple private IP addresses.

The Amazon VPC Container Network Interface (CNI) plugin for Kubernetes is deployed with each of your Amazon EC2 nodes in a Daemonset with the name aws-node.

• Responsible for creating network interfaces and attaching the network interfaces to Amazon EC2 instances, assigning secondary IP addresses to network interfaces, and maintaining a warm pool of IP addresses on each node for assignment to Kubernetes pods when they are scheduled.
• When the number of pods running on the node exceeds the number of addresses that can be assigned to a single network interface, the plugin starts allocating a new network interface, as long as the maximum number of network interfaces for the instance aren't already attached
• you can tweak default behaviour if needed see https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

The number of network interfaces, and the number of IP allowed in each network interface depend on the instance type, see IP addresses per network interface per instance type

For instance, for a c5d.4xlarge we have:

Instance type	Maximum network interfaces	Private IPv4 addresses per interface	IPv6 addresses per interface
c5d.4xlarge	8	30	30

The Instances is using the 8 ENI available using 30 IP in each so the max usable IP would be : 8ENI * (30-1)@ +2 = 234 @IP availables 30*8-2 = 238 pods

You can chack for each instance type the maximum number of pods in eni-max-pods.txt

We can also see this information from the node itself:

kubectl describe node ip-10-0-83-246.eu-west-1.compute.internal | grep -i pods
  pods:                        234

If you are in this configuration, and you execute the 2048-deployment.yaml you'll see that you can apply up to 225 pods and that the node has 234 pods (including other workloads present on my nodes such as daemonsets (aws-node, cloudwatch-agent, aws-for-fluent-bit, ...)

Using Custom Networking

You can follow the tutorial on eksworkshop.com VPC CNI allows you to have custom networking. By default, when new network interfaces are allocated for pods, ipamD uses the node's primary network interface's security groups and subnet. You might want your pods to use a different security group or subnet, within the same VPC as your control plane security group. Examples:

• There are a limited number of IP addresses available in a subnet. This might limit the number of pods that can be created in the cluster. Using different subnets for pods allows you to increase the number of available IP addresses.
• For security reasons, your pods must use different security groups or subnets than the node's primary network interface.

For using Custom Networking, we will need to configure the vcp-cni plugin to uses custom network configuration

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

this needs latests versions of cni-plugin (I have 1.7.8) kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Then we need to create a ENIConfig Custom Ressource Definition for each subnet we want to schedule pods in.

We recommend creating a file with name=AZ specifying the subnet, and security group to uses:

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: <us-west-2a>
spec: 
  securityGroups: 
    - <sg-0dff111a1d11c1c11>
  subnet: <subnet-011b111c1f11fdf11>

When Using custom networking, the formula to calculate the number of Pods is different, because each instance will keep 1 ENI for IP routing, and not available for Pod scheduling:

maxPods = (number of interfaces - 1) * (max IPv4 addresses per interface - 1) + 2

In our case we have: (8-1) * (30-1) + 2 = 205 pods

By playing with the replicas field of the 2048-deployment.yaml file, I can push my instance with 205 pods

k describe node ip-10-0-83-246.eu-west-1.compute.internal | grep -i pods
  pods:                        234
  pods:                        234
Non-terminated Pods:          (205 in total)

In this case we see that the instance still considere to have a max pods to 234. It may be useful in this case to update the bootstrap arguments used to configure your nodes adding for instance :

--use-max-pods false --kubelet-extra-args '--max-pods=<205>'

If you have any nodes in your cluster that had pods placed on them before you completed this procedure, you should terminate them. Only new nodes that are registered with the k8s.amazonaws.com/eniConfig label use the new custom networking feature.