OpenSearch handles two fundamentally different workloads in most AWS environments. The first is full-text search for application features: product search, document search, autocomplete. The second is log and metric analytics: ingesting application logs, CloudWatch exports, and VPC flow logs so engineering teams can query them in near real time.

Both workloads run on the same OpenSearch engine but have different index design requirements, different retention policies, and different query patterns. Getting the configuration right for one does not automatically get it right for the other. In this article I will walk through a production OpenSearch setup that handles both workloads with Terraform.

Cluster Sizing and Architecture

OpenSearch clusters in production need at minimum three dedicated master nodes and two data nodes. The master nodes manage cluster state, shard allocation, and routing. They should never handle indexing or search requests. Data nodes do the actual work.

For log analytics workloads with high ingest rates, use UltraWarm nodes for older data. UltraWarm stores data on S3-backed storage at a fraction of the cost of SSD-backed hot nodes while still supporting queries. Move data from hot to UltraWarm after 7 days and to cold storage after 30 days.

resource "aws_elasticsearch_domain" "main" {
  domain_name           = "production-search"
  elasticsearch_version = "OpenSearch_2.13"

  cluster_config {
    instance_type            = "r6g.large.search"
    instance_count           = 3
    dedicated_master_enabled = true
    dedicated_master_type    = "m6g.large.search"
    dedicated_master_count   = 3
    zone_awareness_enabled   = true

    zone_awareness_config {
      availability_zone_count = 3
    }

    warm_enabled = true
    warm_type    = "ultrawarm1.medium.search"
    warm_count   = 2
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 100
    throughput  = 250
    iops        = 3000
  }

  vpc_options {
    subnet_ids         = slice(var.private_subnet_ids, 0, 3)
    security_group_ids = [aws_security_group.opensearch.id]
  }

  encrypt_at_rest {
    enabled    = true
    kms_key_id = aws_kms_key.opensearch.arn
  }

  node_to_node_encryption {
    enabled = true
  }

  domain_endpoint_options {
    enforce_https       = true
    tls_security_policy = "Policy-Min-TLS-1-2-2019-07"
  }

  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = false
    master_user_options {
      master_user_arn = aws_iam_role.opensearch_admin.arn
    }
  }

  log_publishing_options {
    cloudwatch_log_group_arn = aws_cloudwatch_log_group.opensearch_index_slow.arn
    log_type                 = "INDEX_SLOW_LOGS"
  }

  log_publishing_options {
    cloudwatch_log_group_arn = aws_cloudwatch_log_group.opensearch_search_slow.arn
    log_type                 = "SEARCH_SLOW_LOGS"
  }

  log_publishing_options {
    cloudwatch_log_group_arn = aws_cloudwatch_log_group.opensearch_error.arn
    log_type                 = "ES_APPLICATION_LOGS"
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

Three data nodes across three availability zones gives you high availability. If one AZ goes down, the cluster continues operating with the other two. Primary and replica shards are distributed across AZs automatically when zone awareness is enabled.

gp3 volumes give you configurable throughput and IOPS independently of volume size. For write-heavy log ingest workloads, set throughput to 250 MB/s and IOPS to 3000 minimum. Monitor EBSIOBalance and EBSThroughputBalance metrics and increase these values if you see write throttling.

Index Templates for Application Logs

Index templates define settings and mappings that apply automatically to new indices matching a pattern. For time-series log data, you want a new index created each day with consistent settings.

PUT _index_template/application-logs
{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "30s",
      "index.lifecycle.name": "app-logs-policy",
      "index.lifecycle.rollover_alias": "app-logs",
      "index.codec": "best_compression"
    },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "@timestamp":  { "type": "date" },
        "level":       { "type": "keyword" },
        "service":     { "type": "keyword" },
        "trace_id":    { "type": "keyword" },
        "message":     { "type": "text",    "analyzer": "standard" },
        "duration_ms": { "type": "integer" },
        "status_code": { "type": "short" },
        "user_id":     { "type": "keyword" },
        "endpoint":    { "type": "keyword" },
        "error": {
          "type": "object",
          "properties": {
            "type":    { "type": "keyword" },
            "message": { "type": "text" }
          }
        }
      }
    }
  }
}

dynamic = strict prevents new fields from being added to the index automatically. This stops mapping explosions where applications log unexpected JSON structures that add hundreds of unmapped fields, consuming significant memory in the cluster. Any field not in the mapping returns an error on index. This forces you to be deliberate about what you index.

refresh_interval = 30s delays making new documents searchable until 30 seconds after indexing. The default is 1 second. For log analytics where near-real-time is acceptable, a longer refresh interval significantly reduces write pressure on the cluster by batching segment refreshes.

Index Lifecycle Management

PUT _plugins/_ism/policies/app-logs-policy
{
  "policy": {
    "description": "Manage application log index lifecycle",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1d",
              "min_size": "10gb"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": { "min_index_age": "7d" }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          { "warm_migration": {} },
          { "replica_count": { "number_of_replicas": 0 } }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": { "min_index_age": "90d" }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [ { "delete": {} } ],
        "transitions": []
      }
    ]
  }
}

Removing replicas on warm indices saves storage. Warm data on UltraWarm nodes is already durable because it is backed by S3. A replica on UltraWarm doubles your storage cost without adding meaningful durability. Set replicas to 0 as soon as an index transitions to warm.

Ingestion with Kinesis Firehose

resource "aws_kinesis_firehose_delivery_stream" "app_logs" {
  name        = "app-logs-to-opensearch"
  destination = "opensearch"

  opensearch_configuration {
    domain_arn            = aws_elasticsearch_domain.main.arn
    role_arn              = aws_iam_role.firehose_opensearch.arn
    index_name            = "app-logs"
    index_rotation_period = "OneDay"
    buffering_interval    = 60
    buffering_size        = 5
    retry_duration        = 300

    s3_backup_mode = "FailedDocumentsOnly"

    s3_configuration {
      role_arn           = aws_iam_role.firehose_opensearch.arn
      bucket_arn         = aws_s3_bucket.firehose_backup.arn
      buffering_interval = 300
      buffering_size     = 64
      compression_format = "GZIP"
    }

    vpc_config {
      subnet_ids         = var.private_subnet_ids
      security_group_ids = [aws_security_group.firehose.id]
      role_arn           = aws_iam_role.firehose_opensearch.arn
    }
  }
}

s3_backup_mode = FailedDocumentsOnly means Firehose writes to S3 only when OpenSearch rejects a document. This gives you a recovery path for failed documents without paying to store all log data twice. Check the S3 backup bucket regularly in production. A large number of failed documents indicates a mapping mismatch between what your application is sending and what the index template expects.

Key Metrics to Watch

ClusterStatus.red means at least one primary shard is unassigned. This is a critical alert. Queries against affected indices will fail. The most common cause is a data node going down with no replica to promote, or running out of disk space causing OpenSearch to block indexing.

FreeStorageSpace below 20 percent is the threshold to act before OpenSearch automatically sets indices to read-only. Add more storage or trigger index deletion before hitting that wall.

JVMMemoryPressure above 85 percent for sustained periods causes garbage collection pauses that affect both indexing and search latency. If you see this consistently, either reduce the JVM heap allocation per node or add more data nodes.

Closing Thoughts

OpenSearch rewards careful upfront design more than most AWS services. Index templates with explicit mappings, lifecycle policies that move data to cheaper tiers automatically, and cluster sizing with dedicated masters are not optional extras for production. They are the foundation that keeps the cluster stable as data volume grows.

Set your index templates before you start ingesting. Enable lifecycle management from day one. Watch FreeStorageSpace and JVMMemoryPressure as your primary health indicators. Do those three things and your cluster will handle years of log data and search traffic without major operational incidents.

Enjoy the cloud.

Osama

---

#AWS #OpenSearch #ElasticSearch #SearchEngineering #LogAnalytics #CloudArchitecture #Terraform #InfrastructureAsCode #DataEngineering #AmazonWebServices #SolutionsArchitect #CloudComputing #CloudNative #BackendEngineering #Observability #TechBlog #CloudInfrastructure #Kinesis #Analytics #DevOps