Kubernetes networking on EKS is one of the areas where teams most often run into problems they did not anticipate. Pods running out of IP addresses. Load balancers being created in the wrong subnets. Ingress not routing traffic because an annotation is missing or wrong. These problems are fixable but they are much easier to avoid if you understand the architecture before you build.
In this article I will walk through how EKS networking works end to end, how to configure the VPC CNI for production pod density, and how to set up the AWS Load Balancer Controller with proper Ingress annotations.
How the VPC CNI Works
EKS uses the Amazon VPC CNI plugin by default. Unlike overlay networks used by other Kubernetes distributions, the VPC CNI assigns real VPC IP addresses to pods. Each pod gets an IP from the subnet where its node lives. This means pods are directly routable within the VPC and from on-premises networks connected via VPN or Direct Connect.
The CNI pre-allocates secondary ENIs and IP addresses on each node to minimize pod startup latency. The number of pods a node can run is limited by the number of network interfaces and IP addresses the instance type supports. A t3.medium supports 3 ENIs with 6 IPs each, for a maximum of 17 pods including the system pods. An m5.xlarge supports 4 ENIs with 15 IPs each, supporting up to 58 pods.
For clusters with many small pods, this per-instance limit becomes a bottleneck before CPU or memory is exhausted. The solution is prefix delegation, which assigns a /28 CIDR prefix to each ENI slot rather than individual IPs, multiplying available pod IPs by 16.
resource "aws_eks_cluster" "main" {
name = "production"
role_arn = aws_iam_role.eks_cluster.arn
version = "1.30"
vpc_config {
subnet_ids = concat(var.private_subnet_ids, var.public_subnet_ids)
security_group_ids = [aws_security_group.eks_cluster.id]
endpoint_private_access = true
endpoint_public_access = true
public_access_cidrs = var.allowed_cidr_blocks
}
enabled_cluster_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
tags = { Environment = "production" }
}
resource "aws_eks_node_group" "main" {
cluster_name = aws_eks_cluster.main.name
node_group_name = "main"
node_role_arn = aws_iam_role.eks_node.arn
subnet_ids = var.private_subnet_ids
instance_types = ["m5.xlarge"]
scaling_config {
desired_size = 3
min_size = 2
max_size = 10
}
update_config {
max_unavailable = 1
}
labels = {
role = "application"
}
}
resource "aws_eks_addon" "vpc_cni" {
cluster_name = aws_eks_cluster.main.name
addon_name = "vpc-cni"
addon_version = "v1.18.1-eksbuild.1"
resolve_conflicts_on_update = "OVERWRITE"
service_account_role_arn = aws_iam_role.vpc_cni.arn
configuration_values = jsonencode({
env = {
ENABLE_PREFIX_DELEGATION = "true"
WARM_PREFIX_TARGET = "1"
}
})
}ENABLE_PREFIX_DELEGATION = true activates /28 prefix assignment. On an m5.xlarge with prefix delegation, each of the 4 ENIs can hold 1 prefix of 16 IPs, giving you up to 234 pods per node instead of 58. For clusters running many small workloads, this is the most impactful single configuration change you can make.
Subnet Tagging for Load Balancer Discovery
The AWS Load Balancer Controller discovers which subnets to place load balancers in using specific subnet tags. Without these tags, load balancer creation fails or places resources in the wrong subnets.
resource "aws_subnet" "public" {
count = length(var.azs)
vpc_id = var.vpc_id
cidr_block = var.public_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = {
Name = "public-${var.azs[count.index]}"
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/elb" = "1"
}
}
resource "aws_subnet" "private" {
count = length(var.azs)
vpc_id = var.vpc_id
cidr_block = var.private_cidrs[count.index]
availability_zone = var.azs[count.index]
tags = {
Name = "private-${var.azs[count.index]}"
"kubernetes.io/cluster/${var.cluster_name}" = "shared"
"kubernetes.io/role/internal-elb" = "1"
}
}The kubernetes.io/role/elb = 1 tag on public subnets tells the Load Balancer Controller to use these subnets for internet-facing load balancers. The kubernetes.io/role/internal-elb = 1 tag on private subnets is used for internal load balancers. The cluster tag is required on all subnets the cluster uses.
AWS Load Balancer Controller
resource "aws_iam_role" "load_balancer_controller" {
name = "aws-load-balancer-controller"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = {
Federated = aws_iam_openid_connect_provider.eks.arn
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" = "system:serviceaccount:kube-system:aws-load-balancer-controller"
"${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud" = "sts.amazonaws.com"
}
}
}]
})
}
resource "helm_release" "aws_load_balancer_controller" {
name = "aws-load-balancer-controller"
repository = "https://aws.github.io/eks-charts"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
version = "1.8.1"
set {
name = "clusterName"
value = aws_eks_cluster.main.name
}
set {
name = "serviceAccount.annotations.eks\.amazonaws\.com/role-arn"
value = aws_iam_role.load_balancer_controller.arn
}
set {
name = "replicaCount"
value = "2"
}
set {
name = "region"
value = var.region
}
set {
name = "vpcId"
value = var.vpc_id
}
}Ingress Configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
namespace: production
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}, {"HTTP": 80}]'
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789:certificate/abc123
alb.ingress.kubernetes.io/healthcheck-path: /health
alb.ingress.kubernetes.io/healthcheck-interval-seconds: "30"
alb.ingress.kubernetes.io/healthy-threshold-count: "2"
alb.ingress.kubernetes.io/unhealthy-threshold-count: "3"
alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:123456789:regional/webacl/main/abc123
spec:
rules:
- host: app.yourdomain.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 8080
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 3000target-type: ip routes traffic directly to pod IPs rather than to node IPs. This bypasses kube-proxy and NodePort services, reducing latency and allowing the ALB health checks to target pods directly. It requires the VPC CNI since pods need routable VPC IPs, which is why these two pieces work together.
Attaching the WAF ACL ARN at the Ingress level applies WAF rules to all traffic entering through this ALB, covering every service behind it in a single annotation.
Closing Thoughts
EKS networking has more moving parts than most AWS services, but the pieces fit together logically once you understand why each one exists. The VPC CNI gives pods real VPC IPs, enabling direct routing and ALB target-type ip. Prefix delegation removes the pod density ceiling. Subnet tags tell the Load Balancer Controller where to put load balancers. The Ingress annotations connect your application to the ALB with the specific routing and security behavior you need.
Get the subnet tags right before you start deploying workloads. Enable prefix delegation if you run many small pods. Use target-type ip on all Ingress resources. These three decisions eliminate the majority of EKS networking problems teams encounter in production.
Enjoy the cloud.
Osama
#AWS #EKS #Kubernetes #CloudArchitecture #NetworkEngineering #CloudNative #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudComputing #DevOps #ContainerOrchestration #VPC #LoadBalancer #TechBlog #CloudInfrastructure #K8s #CNI #Helm
Technology Geek 
Leave a comment