Kafka is one of the most capable event streaming platforms available, and one of the most operationally demanding to run yourself. Broker configuration tuning, ZooKeeper replacement with KRaft, partition leadership rebalancing after broker failures, TLS certificate management, and version upgrades that require rolling restarts across the cluster. Amazon MSK takes on all of that and gives you a fully compatible Kafka cluster you manage through standard Kafka tooling.

In this article I will walk through provisioning a production MSK cluster with Terraform, configuring producers and consumers with the settings that matter for reliability, and using MSK Serverless for teams that want Kafka without cluster management.

Provisioning MSK with Terraform

resource "aws_msk_configuration" "main" {
  name              = "production-msk-config"
  kafka_versions    = ["3.6.0"]
  server_properties = <<-EOF
    auto.create.topics.enable=false
    default.replication.factor=3
    min.insync.replicas=2
    num.partitions=6
    log.retention.hours=168
    log.retention.bytes=107374182400
    log.segment.bytes=1073741824
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    replica.fetch.max.bytes=10485760
    message.max.bytes=10485760
  EOF
}

resource "aws_msk_cluster" "main" {
  cluster_name           = "production-kafka"
  kafka_version          = "3.6.0"
  number_of_broker_nodes = 6

  broker_node_group_info {
    instance_type   = "kafka.m5.xlarge"
    client_subnets  = var.private_subnet_ids
    security_groups = [aws_security_group.msk.id]

    storage_info {
      ebs_storage_info {
        volume_size            = 1000
        provisioned_throughput {
          enabled           = true
          volume_throughput = 250
        }
      }
    }
  }

  configuration_info {
    arn      = aws_msk_configuration.main.arn
    revision = aws_msk_configuration.main.latest_revision
  }

  encryption_info {
    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
    encryption_at_rest_kms_key_arn = aws_kms_key.msk.arn
  }

  client_authentication {
    sasl {
      iam = true
    }
  }

  open_monitoring {
    prometheus {
      jmx_exporter  { enabled_in_broker = true }
      node_exporter { enabled_in_broker = true }
    }
  }

  logging {
    broker_logs {
      cloudwatch_logs {
        enabled   = true
        log_group = aws_cloudwatch_log_group.msk.name
      }
      s3 {
        enabled = true
        bucket  = aws_s3_bucket.msk_logs.bucket
        prefix  = "msk-logs/"
      }
    }
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

Six broker nodes across three AZs gives you two brokers per AZ. With default.replication.factor=3 and min.insync.replicas=2, each partition has three replicas spread across AZs, and producers require acknowledgment from at least two replicas before a write is confirmed. This configuration survives the loss of an entire AZ without data loss or write failures.

auto.create.topics.enable=false is a production requirement. With auto-creation enabled, a misconfigured producer can create topics with wrong partition counts or replication factors that are expensive to fix later. Manage topic creation explicitly through your infrastructure code.

IAM authentication removes the need for username/password or certificate-based client authentication. Producers and consumers use their IAM role and AWS Signature V4 signing. No credentials to rotate, no certificate expiry to track.

Creating Topics

resource "aws_msk_cluster_policy" "main" {
  cluster_arn = aws_msk_cluster.main.arn
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { AWS = aws_iam_role.msk_admin.arn }
      Action    = ["kafka-cluster:*"]
      Resource  = "${aws_msk_cluster.main.arn}/*"
    }]
  })
}

resource "kafka_topic" "orders" {
  name               = "orders"
  replication_factor = 3
  partitions         = 18

  config = {
    "cleanup.policy"       = "delete"
    "retention.ms"         = "604800000"
    "min.insync.replicas"  = "2"
    "compression.type"     = "lz4"
  }
}

resource "kafka_topic" "order_events_dlq" {
  name               = "orders.DLQ"
  replication_factor = 3
  partitions         = 6

  config = {
    "cleanup.policy" = "delete"
    "retention.ms"   = "2592000000"
  }
}

Producer Configuration

from kafka import KafkaProducer
import boto3
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

def create_msk_producer(bootstrap_servers: list) -> KafkaProducer:
    def oauth_cb(oauth_config):
        auth_token, expiry_ms = MSKAuthTokenProvider.generate_auth_token("us-east-1")
        return auth_token, expiry_ms / 1000

    return KafkaProducer(
        bootstrap_servers    = bootstrap_servers,
        security_protocol    = "SASL_SSL",
        sasl_mechanism       = "OAUTHBEARER",
        sasl_oauth_token_provider = oauth_cb,
        acks                 = "all",
        retries              = 10,
        retry_backoff_ms     = 200,
        max_in_flight_requests_per_connection = 1,
        enable_idempotence   = True,
        compression_type     = "lz4",
        batch_size           = 65536,
        linger_ms            = 10,
        buffer_memory        = 67108864
    )

acks=all combined with enable_idempotence=True is the configuration that prevents data loss and duplicates. acks=all waits for all in-sync replicas to acknowledge the write. Idempotent producers assign sequence numbers to messages, allowing brokers to deduplicate retries. max_in_flight_requests_per_connection=1 ensures messages within a partition arrive in order even during retries. These three settings together provide exactly-once semantics at the producer level.

MSK Serverless

resource "aws_msk_serverless_cluster" "main" {
  cluster_name = "dev-kafka-serverless"

  vpc_config {
    subnet_ids         = var.private_subnet_ids
    security_group_ids = [aws_security_group.msk.id]
  }

  client_authentication {
    sasl {
      iam { enabled = true }
    }
  }

  tags = { Environment = "development" }
}

MSK Serverless is the right choice for development environments, staging clusters with variable load, and teams that want Kafka compatibility without managing cluster sizing. You pay per GB ingested and egressed with no broker nodes to provision. The tradeoff is that it does not support all Kafka configurations and has throughput limits that matter at scale. For production with predictable high throughput, provisioned MSK remains the better option.

Closing Thoughts

MSK gives you a production-ready Kafka cluster without the operational burden of managing brokers, ZooKeeper or KRaft, certificate authorities, and rolling upgrades. The IAM authentication model integrates cleanly with your existing AWS identity infrastructure. The provisioned throughput on EBS volumes handles high-throughput workloads without the performance unpredictability of burst storage.

Disable auto topic creation. Use acks=all with idempotent producers. Set min.insync.replicas=2 with replication factor 3. Monitor consumer lag as your primary operational health metric. These four decisions cover the reliability surface that matters most in production Kafka deployments.

Enjoy the cloud.

Osama

---

#AWS #MSK #ApacheKafka #EventStreaming #CloudArchitecture #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudComputing #DataEngineering #RealTimeData #StreamProcessing #TechBlog #CloudInfrastructure #Microservices #EventDriven #BackendEngineering #SRE #DevOps