Getting an OKE cluster running takes 20 minutes. Getting it production-ready takes considerably longer because the defaults that make it easy to start are not the same defaults that make it safe to run. The API server is publicly accessible by default. Node pools share subnets with system workloads. Pods can mount service account tokens with overly broad scope. None of this belongs in production.

This post covers a production OKE cluster with Terraform: private API server endpoint, separated node pool topology by workload type, NSG isolation between tiers, OCI Workload Identity for pod-level IAM access without stored credentials, Pod Security Admission enforcement, and cluster autoscaler configuration.

Step 1: Private Cluster with Separated Node Pools

resource "oci_containerengine_cluster" "production" {
  compartment_id     = var.compartment_id
  kubernetes_version = "v1.30.1"
  name               = "production-cluster"
  vcn_id             = var.vcn_id

  endpoint_config {
    is_public_ip_enabled = false
    subnet_id            = var.api_endpoint_subnet_id
    nsg_ids              = [oci_core_network_security_group.api_endpoint_nsg.id]
  }

  options {
    kubernetes_network_config {
      pods_cidr     = "10.244.0.0/16"
      services_cidr = "10.96.0.0/16"
    }
    service_lb_subnet_ids = [var.load_balancer_subnet_id]
    add_ons {
      is_kubernetes_dashboard_enabled = false
      is_tiller_enabled               = false
    }
  }

  image_policy_config {
    is_policy_enabled = true
    key_details { kms_key_id = var.image_signing_key_id }
  }

  cluster_pod_network_options { cni_type = "OCI_VCN_IP_NATIVE" }
}

# System node pool - kube-system workloads only
resource "oci_containerengine_node_pool" "system" {
  cluster_id         = oci_containerengine_cluster.production.id
  compartment_id     = var.compartment_id
  kubernetes_version = "v1.30.1"
  name               = "system-pool"

  node_config_details {
    size    = 3
    nsg_ids = [oci_core_network_security_group.system_nodes_nsg.id]
    placement_configs {
      availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
      subnet_id           = var.system_node_subnet_id
      fault_domains       = ["FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"]
    }
  }

  node_shape = "VM.Standard.E4.Flex"
  node_shape_config { ocpus = 4; memory_in_gbs = 32 }
  node_source_details { image_id = data.oci_core_images.ol8.images[0].id; source_type = "IMAGE"; boot_volume_size_in_gbs = 100 }
  initial_node_labels { key = "node-role"; value = "system" }
}

# Application node pool - production workloads
resource "oci_containerengine_node_pool" "application" {
  cluster_id         = oci_containerengine_cluster.production.id
  compartment_id     = var.compartment_id
  kubernetes_version = "v1.30.1"
  name               = "application-pool"

  node_config_details {
    size    = 3
    nsg_ids = [oci_core_network_security_group.app_nodes_nsg.id]
    placement_configs {
      availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
      subnet_id           = var.app_node_subnet_id
      fault_domains       = ["FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"]
    }
  }

  node_shape = "VM.Standard.E4.Flex"
  node_shape_config { ocpus = 8; memory_in_gbs = 64 }
  node_source_details { image_id = data.oci_core_images.ol8.images[0].id; source_type = "IMAGE"; boot_volume_size_in_gbs = 100 }
  initial_node_labels { key = "node-role"; value = "application" }
}

Step 2: OCI Workload Identity

OCI Workload Identity is the OKE equivalent of AWS IRSA. Pods authenticate to OCI services using their Kubernetes service account identity, mapped to OCI IAM dynamic groups. No credentials in secrets, no long-lived API keys mounted into pods.

resource "oci_identity_dynamic_group" "orders_pods" {
  compartment_id = var.tenancy_ocid
  name           = "orders-api-pods"
  description    = "OKE pods running as the orders-api service account"
  matching_rule  = "All {resource.type = 'workloadIdentities', resource.compartment.id = '${var.compartment_id}'}"
}

resource "oci_identity_policy" "orders_pod_policy" {
  compartment_id = var.compartment_id
  name           = "orders-api-pod-policy"
  statements = [
    "Allow dynamic-group orders-api-pods to use secret-family in compartment id ${var.compartment_id} where target.secret.name = 'orders-api-*'",
    "Allow dynamic-group orders-api-pods to use queues in compartment id ${var.compartment_id} where target.queue.name = 'orders-*'"
  ]
}

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders-api
  namespace: orders
  annotations:
    oci.oraclecloud.com/workload-identity-enabled: "true"
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: orders-api
      containers:
        - name: orders-api
          image: me-jeddah-1.ocir.io/namespace/orders-api:1.2.3
          env:
            - name: OCI_RESOURCE_PRINCIPAL_VERSION
              value: "2.2"
            - name: OCI_RESOURCE_PRINCIPAL_REGION
              value: me-jeddah-1

import oci, os

# Pod authenticates automatically via Workload Identity - no credentials needed
signer      = oci.auth.signers.EphemeralResourcePrincipalSigner()
vaults      = oci.vault.VaultsClient(config={}, signer=signer)

secret_bundle = vaults.get_secret_bundle_by_name(
    secret_name="orders-api-db-password",
    vault_id=os.environ["VAULT_ID"]
).data

Step 3: Pod Security Admission

apiVersion: v1
kind: Namespace
metadata:
  name: orders
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        fsGroup: 1001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: orders-api
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            capabilities:
              drop: ["ALL"]

Step 4: Cluster Autoscaler

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: cluster-autoscaler
          image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.0
          command:
            - ./cluster-autoscaler
            - --cloud-provider=oci
            - --nodes=2:20:ocid1.nodepool.oc1..your-app-pool-ocid
            - --scale-down-delay-after-add=5m
            - --scale-down-unneeded-time=10m
            - --scale-down-utilization-threshold=0.5
            - --expander=least-waste
            - --skip-nodes-with-local-storage=false
            - --v=4

Set --scale-down-delay-after-add=5m to prevent the autoscaler from removing nodes immediately after adding them. Without this, a traffic spike triggers a scale-out followed by an immediate scale-in before the new nodes are fully utilized, causing unnecessary churn and wasting provisioning cost.

Use --expander=least-waste for workloads with heterogeneous resource requests. It selects the node group that wastes the least CPU and memory when adding a node, keeping cluster utilization higher compared to the default random expander.

Step 5: NetworkPolicy for Namespace Isolation

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-namespace-isolation
  namespace: orders
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - port: 8080
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders
    # Allow DNS
    - ports:
        - port: 53
          protocol: UDP
    # Allow OCI API endpoints
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8

Operational Notes

The image policy config with a KMS signing key enforces that only images signed with your key can run in the cluster. Any unsigned image is rejected at admission. Build image signing into your OCI DevOps pipeline after push and before deployment, and distribute the verification public key to the cluster via the image policy configuration.

System and application node pools should be in different subnets with different NSGs. This allows you to apply more restrictive NSG rules to the application pool and control which node pools have egress access to specific OCI services. Mixing system and application workloads on the same nodes makes it harder to apply least-privilege networking at the node level.

Regards,
Osama

#OCI #OracleCloud #OKE #Kubernetes #Terraform #CloudSecurity #DevOps #PlatformEngineering #TechBlog #Oracle #CloudNative #K8s #IaC #WorkloadIdentity #PodSecurity #ClusterAutoscaler #OracleCloudInfrastructure #ContainerSecurity #DevSecOps #ZeroTrust