Most teams copy a VPC configuration from a blog post or the AWS console defaults and move on. That works until it does not. A poorly designed VPC creates security exposure, makes troubleshooting painful, and limits your ability to scale or connect to other networks later without major rework.
I have seen production environments where everything ran in public subnets because setting up NAT gateways felt like extra work. I have seen security groups that allow all traffic between every service because someone could not figure out which ports were needed. These are the configurations that make incidents worse and audits uncomfortable.
In this article I will walk through designing a production VPC from scratch. We will cover subnet segmentation, NAT gateway placement, security group strategy, flow logs, and VPC peering. Everything built with Terraform.
The Three-Tier Subnet Model
A production VPC needs at minimum three tiers of subnets in each availability zone.
Public subnets hold resources that need direct internet access: load balancers, NAT gateways, and bastion hosts if you use them. Resources in public subnets have a route to an internet gateway. They can receive inbound connections from the internet if their security group allows it.
Private subnets hold your application tier: ECS tasks, Lambda functions in a VPC, EC2 instances running application servers. Resources here have outbound internet access through a NAT gateway but cannot receive inbound connections from the internet directly. All inbound traffic comes through the load balancer in the public subnet.
Data subnets hold your databases, ElastiCache clusters, and any other data stores. Resources here have no outbound internet access at all. They can only receive connections from the private subnet. This isolation means a compromised application server cannot phone home with your data even if the attacker establishes a shell.
locals {
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
public_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
private_subnets = ["10.0.11.0/24", "10.0.12.0/24", "10.0.13.0/24"]
data_subnets = ["10.0.21.0/24", "10.0.22.0/24", "10.0.23.0/24"]
}
resource "aws_vpc" "main" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = { Name = "production-vpc" }
}
resource "aws_subnet" "public" {
count = length(local.azs)
vpc_id = aws_vpc.main.id
cidr_block = local.public_subnets[count.index]
availability_zone = local.azs[count.index]
map_public_ip_on_launch = false
tags = { Name = "public-${local.azs[count.index]}", Tier = "public" }
}
resource "aws_subnet" "private" {
count = length(local.azs)
vpc_id = aws_vpc.main.id
cidr_block = local.private_subnets[count.index]
availability_zone = local.azs[count.index]
tags = { Name = "private-${local.azs[count.index]}", Tier = "private" }
}
resource "aws_subnet" "data" {
count = length(local.azs)
vpc_id = aws_vpc.main.id
cidr_block = local.data_subnets[count.index]
availability_zone = local.azs[count.index]
tags = { Name = "data-${local.azs[count.index]}", Tier = "data" }
}map_public_ip_on_launch = false on the public subnets is intentional. Auto-assigning public IPs to every resource launched in a public subnet is a common misconfiguration. Assign Elastic IPs explicitly only to resources that actually need them.
Internet Gateway and NAT Gateways
resource "aws_internet_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = { Name = "production-igw" }
}
resource "aws_eip" "nat" {
count = length(local.azs)
domain = "vpc"
tags = { Name = "nat-eip-${local.azs[count.index]}" }
}
resource "aws_nat_gateway" "main" {
count = length(local.azs)
allocation_id = aws_eip.nat[count.index].id
subnet_id = aws_subnet.public[count.index].id
tags = { Name = "nat-${local.azs[count.index]}" }
depends_on = [aws_internet_gateway.main]
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.main.id
}
tags = { Name = "public-rt" }
}
resource "aws_route_table" "private" {
count = length(local.azs)
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.main[count.index].id
}
tags = { Name = "private-rt-${local.azs[count.index]}" }
}
resource "aws_route_table" "data" {
vpc_id = aws_vpc.main.id
tags = { Name = "data-rt" }
}
resource "aws_route_table_association" "public" {
count = length(local.azs)
subnet_id = aws_subnet.public[count.index].id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "private" {
count = length(local.azs)
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private[count.index].id
}
resource "aws_route_table_association" "data" {
count = length(local.azs)
subnet_id = aws_subnet.data[count.index].id
route_table_id = aws_route_table.data.id
}One NAT gateway per availability zone is the production configuration. A single NAT gateway is cheaper but creates a single point of failure and a cross-AZ traffic bottleneck. If the AZ hosting your NAT gateway goes down, all private subnet resources in the other AZs lose outbound internet access. Three NAT gateways cost about $100 per month more but remove that risk entirely.
The data subnet route table has no default route. There is no path out to the internet from that tier. Databases can only communicate within the VPC, which is exactly what you want.
Security Groups
Security groups are stateful firewalls. A rule allowing inbound traffic on port 443 automatically allows the return traffic without a separate outbound rule. Design them around application roles, not IP ranges.
resource "aws_security_group" "alb" {
name = "alb-sg"
vpc_id = aws_vpc.main.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
description = "HTTPS from internet"
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
description = "HTTP redirect from internet"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
description = "All outbound"
}
tags = { Name = "alb-sg" }
}
resource "aws_security_group" "app" {
name = "app-sg"
vpc_id = aws_vpc.main.id
ingress {
from_port = 8080
to_port = 8080
protocol = "tcp"
security_groups = [aws_security_group.alb.id]
description = "App traffic from ALB only"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
description = "All outbound"
}
tags = { Name = "app-sg" }
}
resource "aws_security_group" "database" {
name = "database-sg"
vpc_id = aws_vpc.main.id
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.app.id]
description = "PostgreSQL from app tier only"
}
tags = { Name = "database-sg" }
}Referencing security group IDs in ingress rules rather than CIDR ranges is the correct pattern for internal traffic. The ALB security group ID as the source for the app security group means only traffic originating from the ALB can reach your application. No CIDR range to maintain, no risk of IP collisions after a VPC expansion.
The database security group has no egress rule at all. Databases do not initiate outbound connections. Leaving egress open on database security groups is a common mistake that provides an unnecessary outbound path if the instance is compromised.
VPC Flow Logs
Flow logs capture metadata about every network connection in your VPC. They are essential for security investigations, troubleshooting connectivity issues, and understanding your actual traffic patterns.
resource "aws_cloudwatch_log_group" "vpc_flow_logs" {
name = "/aws/vpc/flow-logs"
retention_in_days = 30
kms_key_id = aws_kms_key.logs.arn
}
resource "aws_flow_log" "main" {
vpc_id = aws_vpc.main.id
traffic_type = "ALL"
iam_role_arn = aws_iam_role.flow_logs.arn
log_destination = aws_cloudwatch_log_group.vpc_flow_logs.arn
tags = { Name = "production-vpc-flow-logs" }
}
resource "aws_iam_role" "flow_logs" {
name = "vpc-flow-logs-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "vpc-flow-logs.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy" "flow_logs" {
name = "vpc-flow-logs-policy"
role = aws_iam_role.flow_logs.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
]
Resource = "*"
}]
})
}A useful CloudWatch Insights query when investigating a connectivity issue or suspected intrusion:
fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action, bytes
| filter action = "REJECT"
| filter dstAddr like "10.0."
| sort @timestamp desc
| limit 100This query shows all rejected connections to resources inside your VPC. A sudden spike in rejections from an external IP is a signal worth investigating. Consistent rejections to a specific internal port you did not know was being probed tells you about misconfigurations in your application.
VPC Endpoints for AWS Services
By default, API calls from your private subnets to AWS services like S3, DynamoDB, Secrets Manager, and SSM travel through the NAT gateway, out to the public internet, and back in. This adds latency, NAT gateway data processing costs, and routes sensitive traffic through the public internet unnecessarily.
VPC endpoints route these calls privately within the AWS network:
resource "aws_vpc_endpoint" "s3" {
vpc_id = aws_vpc.main.id
service_name = "com.amazonaws.${var.region}.s3"
vpc_endpoint_type = "Gateway"
route_table_ids = concat(
aws_route_table.private[*].id,
[aws_route_table.data.id]
)
tags = { Name = "s3-endpoint" }
}
resource "aws_vpc_endpoint" "secrets_manager" {
vpc_id = aws_vpc.main.id
service_name = "com.amazonaws.${var.region}.secretsmanager"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private[*].id
security_group_ids = [aws_security_group.vpc_endpoints.id]
private_dns_enabled = true
tags = { Name = "secretsmanager-endpoint" }
}
resource "aws_vpc_endpoint" "ssm" {
vpc_id = aws_vpc.main.id
service_name = "com.amazonaws.${var.region}.ssm"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private[*].id
security_group_ids = [aws_security_group.vpc_endpoints.id]
private_dns_enabled = true
tags = { Name = "ssm-endpoint" }
}
resource "aws_security_group" "vpc_endpoints" {
name = "vpc-endpoints-sg"
vpc_id = aws_vpc.main.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = [aws_security_group.app.id]
description = "HTTPS from app tier"
}
tags = { Name = "vpc-endpoints-sg" }
}S3 and DynamoDB use Gateway endpoints, which are free and work by adding routes to your route tables. All other AWS services use Interface endpoints, which create elastic network interfaces in your subnets and cost about $7.50 per month per AZ per endpoint. The cost is worth it for services your application calls frequently, especially Secrets Manager and SSM Parameter Store.
Closing Thoughts
A well-designed VPC is invisible in daily operations. Traffic flows where it should, security groups block what they should, and when something goes wrong the flow logs tell you exactly what happened. A poorly designed VPC is a constant source of friction: mysterious connectivity failures, security reviews that turn up excessive permissions, and refactoring projects that nobody wants to touch because changing subnets requires rebuilding resources.
The patterns here are not complicated. Three subnet tiers, one NAT gateway per AZ, security groups referencing security groups rather than CIDR ranges, flow logs always enabled, and VPC endpoints for the AWS services you call most. Build this once correctly and it will carry you through years of growth without major rework.
Enjoy the cloud.
Osama
#AWS #VPC #NetworkEngineering #CloudArchitecture #CloudSecurity #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudNative #CloudComputing #DevOps #BackendEngineering #NetworkDesign #TechBlog #CloudInfrastructure #SRE #SystemDesign #SecurityEngineering #CloudNetworking
Technology Geek 
Leave a comment