OCI Instance Pools and Autoscaling: Building a Production-Grade Compute Scaling Architecture with Terraform

Vertical scaling on OCI is straightforward: stop the instance, change the shape, start it again. It works but it does not solve the problem you face at 9am on a Monday when traffic doubles in ten minutes and you need twenty more instances, not one bigger one. That is horizontal scaling, and doing it properly on OCI requires understanding how Instance Configurations, Instance Pools, and Autoscaling Configurations work together.

Most teams get to instance pools quickly. They read the docs, create a pool with a fixed size, and think they are done. What they miss is the autoscaling layer on top, the load balancer backend set attachment that makes the pool actually serve traffic, the health check configuration that removes unhealthy instances before they receive requests, and the custom metric path that scales on application-level signals instead of just CPU.

This post covers all of it: the full Terraform implementation of a production autoscaling group behind a load balancer, health checks, scaling policies using both metric-based and schedule-based triggers, and custom metric publishing so you can scale on queue depth or request latency instead of raw CPU utilization.

How the Components Fit Together

Before writing any Terraform, the relationship between the three core resources matters.

An Instance Configuration is a template. It defines the compute shape, the OS image, the boot volume size, the VCN subnet placement, the cloud-init script, and any attached block volumes. The Instance Configuration itself does not run anything. It is a snapshot of how an instance should be created.

An Instance Pool uses that template to create and manage a group of identically configured instances. The pool maintains a target size, handles replacements when an instance becomes unhealthy, and integrates with the OCI Load Balancer to register and deregister instances automatically as they join or leave the pool.

An Autoscaling Configuration sits on top of the pool and adjusts the target size based on rules you define. It can scale out when CPU exceeds a threshold, scale in when it drops, and follow a fixed schedule for predictable load patterns.

Step 1: Instance Configuration

The cloud-init script inside the instance configuration is where you install your application, configure the OCI Monitoring agent for custom metrics, and register the instance with your configuration management system. Keep it idempotent.

data "oci_core_images" "ol8_image" {
compartment_id = var.compartment_id
operating_system = "Oracle Linux"
operating_system_version = "8"
shape = "VM.Standard.E4.Flex"
sort_by = "TIMECREATED"
sort_order = "DESC"
filter {
name = "display_name"
values = ["^.*Oracle-Linux-8.*$"]
regex = true
}
}
resource "oci_core_instance_configuration" "app_instance_config" {
compartment_id = var.compartment_id
display_name = "orders-api-instance-config-v${var.app_version}"
instance_details {
instance_type = "compute"
launch_details {
compartment_id = var.compartment_id
display_name = "orders-api-node"
shape = "VM.Standard.E4.Flex"
shape_config {
ocpus = 2
memory_in_gbs = 16
}
source_details {
source_type = "image"
image_id = data.oci_core_images.ol8_image.images[0].id
boot_volume_size_in_gbs = 50
}
create_vnic_details {
subnet_id = var.app_subnet_id
assign_public_ip = false
nsg_ids = [var.app_nsg_id]
hostname_label_prefix = "orders-api"
}
metadata = {
ssh_authorized_keys = var.ssh_public_key
user_data = base64encode(templatefile("${path.module}/templates/cloud-init.yaml", {
app_version = var.app_version
compartment_id = var.compartment_id
region = var.region
monitoring_enabled = "true"
}))
}
defined_tags = {
"Operations.Environment" = "production"
"Operations.Application" = "orders-api"
"Operations.ManagedBy" = "terraform"
}
}
}
}

The cloud-init template at templates/cloud-init.yaml:

#cloud-config
runcmd:
# Install OCI Unified Monitoring Agent for custom metrics
- dnf install -y oracle-cloud-agent
- systemctl enable oracle-cloud-agent
- systemctl start oracle-cloud-agent
# Install the application
- mkdir -p /opt/orders-api
- dnf install -y python3.11 python3.11-pip
- pip3.11 install orders-api==${app_version}
# Configure the application
- |
cat > /etc/orders-api/config.yaml <<EOF
environment: production
compartment_id: ${compartment_id}
region: ${region}
metrics_namespace: custom_orders_api
EOF
# Start the application
- systemctl enable orders-api
- systemctl start orders-api
write_files:
- path: /etc/systemd/system/orders-api.service
content: |
[Unit]
Description=Orders API Service
After=network.target
[Service]
Type=simple
User=app
ExecStart=/usr/local/bin/orders-api serve
Restart=always
RestartSec=5
Environment=CONFIG_FILE=/etc/orders-api/config.yaml
[Install]
WantedBy=multi-user.target

Step 2: Instance Pool

resource "oci_core_instance_pool" "orders_api_pool" {
compartment_id = var.compartment_id
instance_configuration_id = oci_core_instance_configuration.app_instance_config.id
display_name = "orders-api-pool"
size = 2
placement_configurations {
availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
primary_subnet_id = var.app_subnet_id
fault_domains = ["FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"]
}
placement_configurations {
availability_domain = data.oci_identity_availability_domains.ads.availability_domains[1].name
primary_subnet_id = var.app_subnet_id_ad2
fault_domains = ["FAULT-DOMAIN-1", "FAULT-DOMAIN-2", "FAULT-DOMAIN-3"]
}
load_balancers {
backend_set_name = oci_load_balancer_backend_set.orders_api_backend.name
load_balancer_id = oci_load_balancer.orders_lb.id
port = 8080
vnic_selection = "PrimaryVnic"
}
defined_tags = {
"Operations.Environment" = "production"
"Operations.Application" = "orders-api"
}
}

Two placement configurations across two availability domains with all three fault domains specified in each. This spreads instances evenly across the physical failure domains within each AD. A single hardware failure affecting one fault domain takes out at most one third of your capacity in one AD, not all of it.

The load_balancers block registers the pool with the load balancer backend set automatically. When the pool adds an instance, OCI registers it with the backend set. When it removes one, OCI deregisters it before terminating the instance so it drains connections cleanly.

Step 3: Load Balancer and Health Check

resource "oci_load_balancer" "orders_lb" {
compartment_id = var.compartment_id
display_name = "orders-api-lb"
shape = "flexible"
subnet_ids = [var.public_subnet_id]
is_private = false
shape_details {
minimum_bandwidth_in_mbps = 10
maximum_bandwidth_in_mbps = 400
}
defined_tags = {
"Operations.Environment" = "production"
"Operations.Application" = "orders-api"
}
}
resource "oci_load_balancer_backend_set" "orders_api_backend" {
name = "orders-api-backend-set"
load_balancer_id = oci_load_balancer.orders_lb.id
policy = "LEAST_CONNECTIONS"
health_checker {
protocol = "HTTP"
port = 8080
url_path = "/health"
interval_ms = 10000
timeout_in_millis = 3000
retries = 3
return_code = 200
response_body_regex = ".*\"status\":\"healthy\".*"
}
session_persistence_configuration {
cookie_name = "orders_session"
disable_fallback = false
}
}
resource "oci_load_balancer_listener" "orders_https" {
load_balancer_id = oci_load_balancer.orders_lb.id
name = "orders-https-listener"
default_backend_set_name = oci_load_balancer_backend_set.orders_api_backend.name
port = 443
protocol = "HTTP"
ssl_configuration {
certificate_name = oci_load_balancer_certificate.orders_cert.certificate_name
verify_peer_certificate = false
protocols = ["TLSv1.2", "TLSv1.3"]
cipher_suite_name = "oci-wider-compatible-ssl-cipher-suite-v1"
}
connection_configuration {
idle_timeout_in_seconds = 60
}
}

The health checker uses response_body_regex to validate the response body, not just the HTTP status code. Your /health endpoint should return a JSON payload that confirms the application is ready to serve traffic, not just that the process is running. A process can be alive but unable to connect to the database, which makes it unhealthy from a request-serving perspective even though it returns 200.

Step 4: Metric-Based Autoscaling

The default metric-based autoscaling policy uses CPU utilization. This works for CPU-bound workloads but misses the mark for I/O-bound services where CPU stays low while request queues build up.

resource "oci_autoscaling_auto_scaling_configuration" "orders_api_asc" {
compartment_id = var.compartment_id
display_name = "orders-api-autoscaling"
is_enabled = true
cool_down_in_seconds = 300
auto_scaling_resources {
id = oci_core_instance_pool.orders_api_pool.id
type = "instancePool"
}
policies {
display_name = "cpu-scale-out"
policy_type = "threshold"
capacity {
initial = 2
min = 2
max = 20
}
rules {
display_name = "scale-out-on-high-cpu"
action {
type = "CHANGE_COUNT_BY"
value = 2
}
metric {
metric_type = "CPU_UTILIZATION"
threshold {
operator = "GT"
value = 75
}
}
}
rules {
display_name = "scale-in-on-low-cpu"
action {
type = "CHANGE_COUNT_BY"
value = -1
}
metric {
metric_type = "CPU_UTILIZATION"
threshold {
operator = "LT"
value = 25
}
}
}
}
}

The cool_down_in_seconds = 300 prevents the autoscaler from firing again within five minutes of the last scaling action. Without this, a sudden traffic spike triggers a scale-out, the new instances come online, CPU drops, a scale-in fires immediately, the instances terminate, CPU climbs again, and you get an oscillation loop. Five minutes gives new instances time to warm up and take on load before the next evaluation.

Scale out by 2, scale in by 1. Always scale out faster than you scale in. The cost of having one extra instance for a few minutes is trivial compared to the cost of serving degraded traffic because you removed capacity too aggressively.

Step 5: Custom Metric Autoscaling

CPU-based scaling is not enough for most production services. A better signal is often active request queue depth or response latency percentile. If your application publishes custom metrics to OCI Monitoring, you can scale on those instead.

Here is how the application publishes a custom metric from Python:

python

import oci
import json
from datetime import datetime, timezone
def publish_queue_depth_metric(queue_depth: int, compartment_id: str):
config = oci.config.from_file()
monitoring_client = oci.monitoring.MonitoringClient(
config,
service_endpoint="https://telemetry-ingestion.{}.oraclecloud.com".format(config["region"])
)
metric_data = oci.monitoring.models.PostMetricDataDetails(
metric_data=[
oci.monitoring.models.MetricDataDetails(
namespace="custom_orders_api",
compartment_id=compartment_id,
name="RequestQueueDepth",
dimensions={
"environment": "production",
"application": "orders-api"
},
datapoints=[
oci.monitoring.models.Datapoint(
timestamp=datetime.now(timezone.utc),
value=float(queue_depth)
)
],
metadata={
"unit": "count",
"displayName": "Request Queue Depth"
}
)
]
)
response = monitoring_client.post_metric_data(
post_metric_data_details=metric_data
)
return response.status

Call this function every 60 seconds from a background thread in your application. Once the metric appears in OCI Monitoring under the custom_orders_api namespace, you can create an autoscaling rule against it.

OCI’s native autoscaling configuration only supports CPU_UTILIZATION and MEMORY_UTILIZATION as built-in metric types. To scale on a custom metric you pair OCI Monitoring alarms with an OCI Functions trigger that calls the Instance Pool resize API directly.

resource "oci_monitoring_alarm" "queue_depth_high" {
compartment_id = var.compartment_id
display_name = "orders-api-queue-depth-high"
is_enabled = true
metric_compartment_id = var.compartment_id
namespace = "custom_orders_api"
query = "RequestQueueDepth[1m]{environment = 'production'}.mean() > 500"
severity = "WARNING"
pending_duration = "PT2M"
destinations = [oci_ons_notification_topic.scaling_topic.id]
body = "Queue depth exceeded 500 for 2 minutes. Scaling out instance pool."
}
resource "oci_monitoring_alarm" "queue_depth_low" {
compartment_id = var.compartment_id
display_name = "orders-api-queue-depth-low"
is_enabled = true
metric_compartment_id = var.compartment_id
namespace = "custom_orders_api"
query = "RequestQueueDepth[5m]{environment = 'production'}.mean() < 100"
severity = "INFO"
pending_duration = "PT10M"
destinations = [oci_ons_notification_topic.scaling_topic.id]
body = "Queue depth below 100 for 10 minutes. Scaling in instance pool."
}
resource "oci_ons_notification_topic" "scaling_topic" {
compartment_id = var.compartment_id
name = "orders-api-scaling-events"
description = "Triggers custom metric scaling function"
}
resource "oci_ons_subscription" "scaling_function_sub" {
compartment_id = var.compartment_id
topic_id = oci_ons_notification_topic.scaling_topic.id
protocol = "ORACLE_FUNCTIONS"
endpoint = oci_functions_function.pool_scaler.id
}

The OCI Function that handles the scaling action:

import io
import json
import oci
import logging
logger = logging.getLogger()
POOL_ID = "ocid1.instancepool.oc1..."
MIN_SIZE = 2
MAX_SIZE = 20
SCALE_OUT_BY = 2
SCALE_IN_BY = 1
def handler(ctx, data: io.BytesIO = None):
try:
body = json.loads(data.getvalue())
alarm_body = body.get("body", "")
logger.info(f"Received alarm notification: {alarm_body}")
except Exception as ex:
logger.error(f"Failed to parse notification: {ex}")
return
signer = oci.auth.signers.get_resource_principals_signer()
compute_mgmt = oci.core.ComputeManagementClient(config={}, signer=signer)
pool = compute_mgmt.get_instance_pool(POOL_ID).data
current_size = pool.size
if "Scaling out" in alarm_body:
new_size = min(current_size + SCALE_OUT_BY, MAX_SIZE)
action = "scale-out"
elif "Scaling in" in alarm_body:
new_size = max(current_size - SCALE_IN_BY, MIN_SIZE)
action = "scale-in"
else:
logger.info("Unrecognized alarm body, no action taken")
return
if new_size == current_size:
logger.info(f"Already at {'max' if action == 'scale-out' else 'min'} size ({current_size}), no action")
return
update_details = oci.core.models.UpdateInstancePoolDetails(size=new_size)
compute_mgmt.update_instance_pool(POOL_ID, update_details)
logger.info(f"Pool resize triggered: {current_size} to {new_size} ({action})")

The function uses get_resource_principals_signer() to authenticate with the Dynamic Group policy. No credentials are stored in the function configuration.

Step 6: Schedule-Based Scaling

For workloads with predictable patterns you can layer a schedule-based policy on top of the metric-based policy. Business-hours applications can scale up before the working day starts and scale down after it ends, reducing idle capacity costs at night.

resource "oci_autoscaling_auto_scaling_configuration" "orders_api_schedule" {
compartment_id = var.compartment_id
display_name = "orders-api-schedule-scaling"
is_enabled = true
cool_down_in_seconds = 300
auto_scaling_resources {
id = oci_core_instance_pool.orders_api_pool.id
type = "instancePool"
}
policies {
display_name = "business-hours-scale-up"
policy_type = "scheduled"
execution_schedule {
expression = "0 7 * * 0-4"
timezone = "Asia/Riyadh"
type = "cron"
}
capacity {
initial = 6
min = 6
max = 20
}
}
policies {
display_name = "after-hours-scale-down"
policy_type = "scheduled"
execution_schedule {
expression = "0 20 * * 0-4"
timezone = "Asia/Riyadh"
type = "cron"
}
capacity {
initial = 2
min = 2
max = 20
}
}
}

The cron expression 0 7 * * 0-4 fires at 07:00 Sunday through Thursday in the Asia/Riyadh timezone, which covers the standard working week in the Gulf region. At 20:00 the pool scales back to the minimum. The max remains at 20 in both schedules so metric-based scaling can still expand beyond the scheduled minimum during peak periods.

Step 7: Rolling Instance Configuration Update

When you need to deploy a new application version, you update the Instance Configuration and then replace pool instances without taking the pool offline.

# Create new instance configuration with updated app version
resource "oci_core_instance_configuration" "app_instance_config_v2" {
compartment_id = var.compartment_id
display_name = "orders-api-instance-config-v${var.new_app_version}"
# Same configuration as v1 with updated user_data referencing new_app_version
instance_details {
instance_type = "compute"
launch_details {
# ... identical to v1 except user_data references new_app_version
}
}
}
# Update the pool to use the new configuration
resource "oci_core_instance_pool" "orders_api_pool" {
instance_configuration_id = oci_core_instance_configuration.app_instance_config_v2.id
# ... rest of pool config unchanged
}

Updating instance_configuration_id on the pool does not immediately replace running instances. Existing instances continue running with the old configuration. New instances added by scaling or manual pool resize use the new configuration. To replace all existing instances with the new version, trigger a rolling replacement using the OCI CLI:

oci compute-management instance-pool-instance attach \
--instance-pool-id <pool-ocid> \
--instance-id <instance-ocid>
# Or use the softreset action to trigger a rolling replace
oci compute-management instance-pool softreset \
--instance-pool-id <pool-ocid>

The softreset action replaces instances one at a time, waiting for each new instance to pass the load balancer health check before terminating the next old instance. Zero downtime rolling deploy without any orchestration tooling.

Validating the Autoscaling Behavior

Generate artificial CPU load to test the scale-out policy:

# SSH into one of the pool instances via OCI Bastion
# Then stress the CPU
stress-ng --cpu 2 --cpu-load 90 --timeout 600

Watch the pool size change in real time:

watch -n 10 'oci compute-management instance-pool get \
--instance-pool-id <pool-ocid> \
--query "data.{size:size, state:\"lifecycle-state\"}" \
--output table'

List all instances currently in the pool with their health status:

oci compute-management instance-pool-instance list \
--instance-pool-id <pool-ocid> \
--query 'data[*].{id:"id", state:"state", ad:"availability-domain", fault-domain:"fault-domain"}' \
--output table

Check the autoscaling activity history to see every scale event with its trigger reason:

oci autoscaling auto-scaling-configuration list \
--compartment-id <compartment-ocid> \
--query 'data[*].{name:"display-name", enabled:"is-enabled"}' \
--output table

Operational Notes

A few things that matter in production but are easy to miss.

The load balancer backend set policy is set to LEAST_CONNECTIONS. This distributes new connections to the instance with the fewest active connections rather than round-robin. For APIs with variable request duration, this prevents a slow request on one instance from causing it to accumulate a backlog while other instances are idle.

Instance Configuration versioning in Terraform requires care. The configuration resource name includes the version number (app_instance_config_v${var.app_version}), which means Terraform creates a new resource rather than modifying the existing one. This preserves the old configuration so you can roll back by pointing the pool back to the previous configuration OCID if the new version has problems.

The minimum pool size of 2 placed across two availability domains means you always have at least one instance in each AD. A complete outage of one availability domain still leaves the pool functional. Set your minimum to at least 2 and spread placement across ADs for any production workload.

Regards,
Osama

OCI DevOps: Building a Production CI/CD Pipeline with Terraform

Most teams running workloads on OCI manage their deployments through a mix of external tools: GitHub Actions pushing to OKE, Jenkins deploying to compute instances, manual Terraform runs triggered from a developer’s laptop. This works until it does not. The audit trail is scattered, secrets flow through CI runners that may not be in your VCN, and there is no native integration between the deployment tooling and the OCI IAM model that controls the infrastructure.

OCI DevOps is Oracle’s native CI/CD service. It covers source code mirroring, build pipelines, artifact management, and deployment pipelines to OKE, compute instances, Functions, and other targets. Everything runs inside your tenancy, authenticates through IAM Dynamic Groups and policies, and integrates natively with OCI Vault for secrets, OCI Container Registry for images, and OCI Artifact Registry for generic artifacts.

In this post I will build a complete pipeline from source code mirror through build, test, image push, and deployment to an OKE cluster, using Terraform for all infrastructure and a real application for the pipeline to deploy.

Service Architecture

OCI DevOps has five main components that work together.

The Project is the top-level container. It groups all related resources: code repositories, build pipelines, deployment pipelines, and environments.

Code Repositories mirror external Git repositories (GitHub, GitLab, Bitbucket) or host code natively inside OCI. Mirroring syncs on a schedule or on webhook trigger.

Build Pipelines execute build stages: managed build (runs your build spec on Oracle-managed runners), deliver artifact (pushes to Container Registry or Artifact Registry), and trigger deployment.

Artifact Registry stores generic versioned artifacts: Helm charts, Terraform modules, JAR files, and deployment manifests.

Deployment Pipelines run the actual deployment to a target environment. They support blue-green, canary, and rolling deployment strategies with built-in approval gates.

Step 1: IAM Setup

OCI DevOps needs a Dynamic Group that matches the build and deployment pipeline resources, and a policy that grants them the permissions to do their work.

resource "oci_identity_dynamic_group" "devops_build_dg" {
compartment_id = var.tenancy_ocid
name = "devops-build-pipelines"
description = "Dynamic group for OCI DevOps build pipeline runners"
matching_rule = "All {resource.type = 'devopsbuildpipeline', resource.compartment.id = '${var.compartment_id}'}"
}
resource "oci_identity_dynamic_group" "devops_deploy_dg" {
compartment_id = var.tenancy_ocid
name = "devops-deploy-pipelines"
description = "Dynamic group for OCI DevOps deployment pipelines"
matching_rule = "All {resource.type = 'devopsdeploypipeline', resource.compartment.id = '${var.compartment_id}'}"
}
resource "oci_identity_policy" "devops_policy" {
compartment_id = var.compartment_id
name = "devops-pipeline-policy"
description = "Permissions for OCI DevOps build and deploy pipelines"
statements = [
# Build pipelines need to read secrets and push to container registry
"Allow dynamic-group devops-build-pipelines to manage repos in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to read secret-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to manage artifacts in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to manage devops-family in compartment id ${var.compartment_id}",
# Deploy pipelines need to manage OKE workloads and read artifacts
"Allow dynamic-group devops-deploy-pipelines to manage cluster-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to use artifacts in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to manage devops-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to read secret-family in compartment id ${var.compartment_id}"
]
}

Step 2: Create the DevOps Project

resource "oci_devops_project" "orders_api_project" {
compartment_id = var.compartment_id
name = "orders-api"
description = "CI/CD pipeline for the orders API service"
notification_config {
topic_id = oci_ons_notification_topic.devops_alerts.id
}
defined_tags = {
"Operations.Environment" = "production"
"Operations.ManagedBy" = "terraform"
}
}
resource "oci_ons_notification_topic" "devops_alerts" {
compartment_id = var.compartment_id
name = "devops-pipeline-alerts"
description = "Notifications for DevOps pipeline events"
}
resource "oci_ons_subscription" "devops_email" {
compartment_id = var.compartment_id
topic_id = oci_ons_notification_topic.devops_alerts.id
protocol = "EMAIL"
endpoint = var.devops_alert_email
}

Step 3: Mirror the GitHub Repository

OCI DevOps can mirror a GitHub repository and trigger a build pipeline on push events. The mirror keeps a copy of the source inside OCI so builds do not depend on external connectivity to GitHub at build time.

resource "oci_devops_repository" "orders_api_repo" {
project_id = oci_devops_project.orders_api_project.id
name = "orders-api"
description = "Mirror of GitHub orders-api repository"
repository_type = "MIRRORED"
default_branch = "main"
mirror_repository_config {
repository_url = "https://github.com/your-org/orders-api.git"
connector_id = oci_devops_connection.github_connection.id
trigger_schedule {
schedule_type = "CUSTOM"
custom_schedule = "0 */6 * * *"
}
}
}
resource "oci_devops_connection" "github_connection" {
project_id = oci_devops_project.orders_api_project.id
display_name = "github-connection"
connection_type = "GITHUB_ACCESS_TOKEN"
description = "Connection to GitHub using PAT stored in OCI Vault"
access_token = oci_vault_secret.github_pat.id
}
resource "oci_vault_secret" "github_pat" {
compartment_id = var.compartment_id
vault_id = var.vault_id
key_id = var.vault_key_id
secret_name = "github-pat-devops"
secret_content {
content_type = "BASE64"
content = base64encode(var.github_personal_access_token)
}
}

The GitHub PAT is stored in OCI Vault, not in a Terraform variable or environment variable on a CI runner. The build pipeline retrieves it at runtime using the Dynamic Group policy.

Step 4: Build Spec

The build spec is a YAML file committed to your repository at build_spec.yaml. It defines the steps the managed build runner executes.

version: 0.1
component: build
timeoutInSeconds: 1800
env:
exportedVariables:
- BUILDRUN_HASH
steps:
- type: Command
name: Set build hash
command: |
export BUILDRUN_HASH=$(echo ${OCI_BUILD_RUN_ID} | tail -c 8)
echo "BUILDRUN_HASH: ${BUILDRUN_HASH}"
- type: Command
name: Install dependencies
command: |
cd orders-api
pip install -r requirements.txt --quiet
- type: Command
name: Run unit tests
command: |
cd orders-api
python -m pytest tests/unit/ -v --tb=short --junitxml=test-results.xml
if [ $? -ne 0 ]; then
echo "Unit tests failed. Aborting build."
exit 1
fi
- type: Command
name: Run security scan
command: |
pip install bandit --quiet
cd orders-api
bandit -r src/ -f json -o bandit-report.json -ll
if [ $? -eq 1 ]; then
echo "High severity security issues found. Aborting build."
exit 1
fi
- type: Command
name: Build container image
command: |
cd orders-api
IMAGE_TAG="${CONTAINER_REGISTRY}/${NAMESPACE}/orders-api:${BUILDRUN_HASH}"
docker build -t orders-api:latest -t ${IMAGE_TAG} .
echo "IMAGE_TAG=${IMAGE_TAG}" >> ${OCI_PRIMARY_SOURCE_DIR}/build_output.env
- type: Command
name: Push image to OCI Container Registry
command: |
docker push ${IMAGE_TAG}
outputArtifacts:
- name: orders-api-image
type: DOCKER_IMAGE
location: ${IMAGE_TAG}
- name: kubernetes-manifests
type: BINARY
location: ${OCI_PRIMARY_SOURCE_DIR}/orders-api/k8s/

The security scan step uses Bandit to flag high-severity Python security issues and fails the build if any are found. This happens before the image is built, not after.

Step 5: Build Pipeline

resource "oci_devops_build_pipeline" "orders_api_build" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-build"
description = "Build, test, scan, and push the orders API container image"
build_pipeline_parameters {
items {
name = "CONTAINER_REGISTRY"
default_value = "${var.oci_region_key}.ocir.io"
description = "OCI Container Registry endpoint"
}
items {
name = "NAMESPACE"
default_value = var.tenancy_namespace
description = "OCI tenancy namespace for Container Registry"
}
}
}
# Stage 1: Managed Build
resource "oci_devops_build_pipeline_stage" "managed_build" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "managed-build"
description = "Execute build spec on managed runner"
build_pipeline_stage_type = "BUILD"
build_spec_file = "build_spec.yaml"
stage_execution_timeout_in_seconds = 1800
image = "OL7_X86_64_STANDARD_10"
build_source_collection {
items {
connection_type = "DEVOPS_CODE_REPOSITORY"
repository_id = oci_devops_repository.orders_api_repo.id
name = "orders-api"
branch = "main"
repository_url = oci_devops_repository.orders_api_repo.http_url
}
}
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline.orders_api_build.id
}
}
}
# Stage 2: Deliver Artifact to Container Registry
resource "oci_devops_build_pipeline_stage" "deliver_artifact" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "deliver-artifact"
description = "Push built image to OCI Container Registry"
build_pipeline_stage_type = "DELIVER_ARTIFACT"
deliver_artifact_collection {
items {
artifact_name = "orders-api-image"
artifact_id = oci_devops_deploy_artifact.orders_api_image.id
}
items {
artifact_name = "kubernetes-manifests"
artifact_id = oci_devops_deploy_artifact.k8s_manifests.id
}
}
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline_stage.managed_build.id
}
}
}
# Stage 3: Trigger Deployment Pipeline
resource "oci_devops_build_pipeline_stage" "trigger_deploy" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "trigger-deployment"
description = "Trigger the deployment pipeline on successful build"
build_pipeline_stage_type = "TRIGGER_DEPLOYMENT_PIPELINE"
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
is_pass_all_parameters_enabled = true
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline_stage.deliver_artifact.id
}
}
}

Step 6: Artifact Registry

resource "oci_artifacts_repository" "k8s_manifests_repo" {
compartment_id = var.compartment_id
display_name = "orders-api-manifests"
description = "Kubernetes deployment manifests for orders API"
is_immutable = false
repository_type = "GENERIC"
}
resource "oci_devops_deploy_artifact" "orders_api_image" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-container-image"
argument_substitution_mode = "SUBSTITUTE_PLACEHOLDERS"
deploy_artifact_type = "DOCKER_IMAGE"
deploy_artifact_source {
deploy_artifact_source_type = "OCIR"
image_uri = "${var.oci_region_key}.ocir.io/${var.tenancy_namespace}/orders-api:$${BUILDRUN_HASH}"
image_digest = " "
}
}
resource "oci_devops_deploy_artifact" "k8s_manifests" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-k8s-manifests"
argument_substitution_mode = "SUBSTITUTE_PLACEHOLDERS"
deploy_artifact_type = "KUBERNETES_MANIFEST"
deploy_artifact_source {
deploy_artifact_source_type = "GENERIC_ARTIFACT"
repository_id = oci_artifacts_repository.k8s_manifests_repo.id
deploy_artifact_path = "k8s/deployment.yaml"
deploy_artifact_version = "$${BUILDRUN_HASH}"
}
}

Step 7: Deployment Environment and Pipeline

The deployment pipeline targets the OKE cluster. Define the environment first, then the pipeline stages.

resource "oci_devops_deploy_environment" "oke_prod" {
project_id = oci_devops_project.orders_api_project.id
display_name = "oke-production"
description = "Production OKE cluster"
deploy_environment_type = "OKE_CLUSTER"
cluster_id = var.oke_cluster_id
}
resource "oci_devops_deploy_pipeline" "orders_api_deploy" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-deploy"
description = "Blue-green deployment of orders API to production OKE"
deploy_pipeline_parameters {
items {
name = "NAMESPACE"
default_value = "orders"
description = "Kubernetes namespace for the deployment"
}
items {
name = "IMAGE_TAG"
default_value = "latest"
description = "Container image tag to deploy"
}
}
}
# Stage 1: Approval gate before production deployment
resource "oci_devops_deploy_stage" "approval_gate" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "production-approval"
description = "Manual approval required before deploying to production"
deploy_stage_type = "MANUAL_APPROVAL"
approval_policy {
approval_policy_type = "COUNT_BASED_APPROVAL"
number_of_approvals_required = 1
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_pipeline.orders_api_deploy.id
}
}
}
# Stage 2: Blue-green deploy to OKE
resource "oci_devops_deploy_stage" "oke_blue_green_deploy" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "oke-blue-green-deploy"
description = "Deploy new version to green environment"
deploy_stage_type = "OKE_BLUE_GREEN_DEPLOYMENT"
oke_blue_green_deploy_stage_details {
kubernetes_manifest_deploy_artifact_ids = [
oci_devops_deploy_artifact.k8s_manifests.id
]
oke_cluster_deploy_environment_id = oci_devops_deploy_environment.oke_prod.id
blue_green_strategy {
strategy_type = "NGINX_INGRESS_STRATEGY"
namespace_a = "orders-blue"
namespace_b = "orders-green"
ingress_name = "orders-api-ingress"
}
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_stage.approval_gate.id
}
}
}
# Stage 3: Traffic shift after successful deployment validation
resource "oci_devops_deploy_stage" "traffic_shift" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "shift-traffic-to-green"
description = "Shift 100% of traffic to the newly deployed green environment"
deploy_stage_type = "OKE_BLUE_GREEN_TRAFFIC_SHIFT"
oke_blue_green_traffic_shift_deploy_stage_details {
oke_blue_green_deployment_deploy_stage_id = oci_devops_deploy_stage.oke_blue_green_deploy.id
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_stage.oke_blue_green_deploy.id
}
}
}

Step 8: Trigger on Code Push

The trigger watches the mirrored repository and fires the build pipeline when a push lands on the main branch.

resource "oci_devops_trigger" "main_branch_push" {
project_id = oci_devops_project.orders_api_project.id
display_name = "main-branch-push-trigger"
description = "Trigger build pipeline on every push to main"
trigger_source = "DEVOPS_CODE_REPOSITORY"
repository_id = oci_devops_repository.orders_api_repo.id
actions {
type = "TRIGGER_BUILD_PIPELINE"
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
filter {
trigger_source = "DEVOPS_CODE_REPOSITORY"
events = ["PUSH"]
include {
head_ref = "main"
}
exclude {
file_filter {
file_paths = ["docs/*", "*.md", ".github/*"]
}
}
}
}
}

The exclude block prevents documentation-only changes from triggering a full build and deploy. Pushing a README update does not kick off the pipeline.

Step 9: Verifying the Pipeline

Once Terraform applies, validate the end-to-end flow.

Check mirror sync status:

oci devops repository get \
--repository-id <your-repo-ocid> \
--query 'data.{name:name, mirror-status:"mirror-repository-config"}' \
--output table

Manually trigger a build to test without waiting for a push:

oci devops build-run create \
--build-pipeline-id <your-build-pipeline-ocid> \
--display-name "manual-validation-run" \
--build-run-arguments '{"items": [{"name": "IMAGE_TAG", "value": "validation-test"}]}'

Watch the build run progress:

oci devops build-run get \
--build-run-id <build-run-ocid> \
--query 'data.{status:"lifecycle-state", phase:"build-run-progress"."build-pipeline-stage-run-progress"}' \
--output table

List deployment history to confirm deployments are being tracked:

oci devops deployment list \
--project-id <project-ocid> \
--sort-by timeCreated \
--sort-order DESC \
--limit 10 \
--query 'data.items[*].{name:"display-name", status:"lifecycle-state", time:\"time-created\"}' \
--output table

Rollback

If a deployment introduces a regression, OCI DevOps blue-green makes rollback immediate. Traffic is still flowing to the old environment until the traffic shift stage completes. If you catch the issue before the shift, simply reject the traffic shift stage from the console or CLI:

oci devops deployment approve \
--deployment-id <deployment-ocid> \
--deploy-stage-id <traffic-shift-stage-ocid> \
--reason "Rolling back: latency regression detected in green environment" \
--action REJECT

The green environment is torn down, the blue environment continues serving traffic, and the deployment is marked as failed with the reason recorded in the audit log.

Where This Fits in a Real Team

The value of OCI DevOps over an external CI/CD tool is not raw feature count. GitHub Actions or GitLab CI have richer marketplace ecosystems. The value is native IAM integration and residency inside your tenancy.

Build runners authenticate to OCI Vault, Container Registry, and Artifact Registry using the Dynamic Group policy with no credentials stored on a third-party platform. Every build and deployment is recorded in OCI Audit with the OCID of the pipeline that ran it. Deployment approvals are logged against the OCI user who approved or rejected them. For regulated environments where you need to prove that every production change was approved by a named human identity and executed by an automated system with least-privilege credentials, OCI DevOps gives you that audit trail natively.

For teams already running everything inside OCI, it is the most operationally coherent choice.

Regards,
Osama

Orchestrating Production Workflows with AWS Step Functions

I want to tell you about a production incident that still bothers me.

We had a payment processing system built on Lambda. Each function did one thing: validate the card, charge the customer, update the order, send the receipt, trigger fulfillment. Clean separation of concerns. Looked great on paper.

Then a Lambda timed out in the middle of the charge step. The card had been charged. The order had not been updated. The receipt never went out. Fulfillment never started. And because there was no central record of what had run, we had no way to resume from where things broke. We ended up with a manual cleanup process, a refund, and an angry customer.

The root problem was not the timeout. The root problem was that we had orchestration logic scattered across function calls, SQS queues, and environment variables. When something went wrong, we had no visibility and no way to recover cleanly.

AWS Step Functions exists to solve exactly this problem. It gives you a managed, visual, stateful orchestration layer that sits above your compute. In this article I will walk you through how Step Functions actually works, the patterns that matter in production, and the mistakes I see teams make when they first adopt it.

What Step Functions Actually Does

Step Functions is a serverless orchestration service. You define a workflow as a state machine using Amazon States Language, a JSON-based specification. Each state in the machine can invoke a Lambda function, call an AWS service directly, wait for a human approval, run a parallel branch, or retry on failure with configurable backoff.

The key thing that separates Step Functions from gluing Lambdas together with SQS is that the state machine itself is the source of truth. Every execution has a complete audit trail. You can look at any execution and see exactly which states ran, what input and output they received, when they ran, and whether they succeeded or failed. When something goes wrong you have a complete picture.

There are two workflow types and the choice matters.

Standard Workflows are designed for long-running, durable processes. They can run for up to a year. Every state transition is recorded in the execution history. You pay per state transition. This is what you want for anything involving payments, order processing, document workflows, or human approvals.

Express Workflows are designed for high-volume, short-duration workloads. They run for up to five minutes, have at-least-once execution semantics, and you pay per execution duration. Use them for event processing pipelines where you need to handle thousands of events per second and idempotency is handled at the application level.

Your First Production State Machine

Let me walk through a real example: an e-commerce order processing workflow. This is a Standard Workflow since order processing is exactly the kind of thing you need full durability and auditability for.

{
"Comment": "Order processing workflow",
"StartAt": "ValidateOrder",
"States": {
"ValidateOrder": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:validate-order",
"Next": "CheckInventory",
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": ["OrderValidationError"],
"Next": "OrderRejected",
"ResultPath": "$.error"
}
]
},
"CheckInventory": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:check-inventory",
"Next": "ProcessPayment",
"Retry": [
{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 5,
"MaxAttempts": 2,
"BackoffRate": 1.5
}
],
"Catch": [
{
"ErrorEquals": ["InsufficientInventoryError"],
"Next": "NotifyOutOfStock",
"ResultPath": "$.error"
}
]
},
"ProcessPayment": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:process-payment",
"Next": "FulfillmentAndNotification",
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": ["PaymentDeclinedError"],
"Next": "NotifyPaymentFailed",
"ResultPath": "$.error"
},
{
"ErrorEquals": ["States.ALL"],
"Next": "OrderProcessingFailed",
"ResultPath": "$.error"
}
]
},
"FulfillmentAndNotification": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "TriggerFulfillment",
"States": {
"TriggerFulfillment": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:trigger-fulfillment",
"End": true
}
}
},
{
"StartAt": "SendConfirmationEmail",
"States": {
"SendConfirmationEmail": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:send-email",
"End": true
}
}
}
],
"Next": "OrderComplete"
},
"OrderComplete": { "Type": "Succeed" },
"OrderRejected": { "Type": "Fail", "Error": "OrderRejected" },
"NotifyOutOfStock": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789:function:notify-out-of-stock", "End": true },
"NotifyPaymentFailed": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789:function:notify-payment-failed", "End": true },
"OrderProcessingFailed": { "Type": "Fail", "Error": "ProcessingFailed" }
}
}

A few things worth pointing out in this definition.

The Retry blocks on each Task state handle transient failures automatically. The configuration above retries on Lambda service exceptions with exponential backoff. You get this behavior for free without writing any retry logic in your Lambda functions themselves.

The Catch blocks handle business-logic failures separately from infrastructure failures. A PaymentDeclinedError routes to a notification state. An unhandled exception routes to a generic failure state. The ResultPath ensures the error detail is written into the execution context alongside the original input, not replacing it.

The Parallel state in FulfillmentAndNotification runs fulfillment and email simultaneously. Both branches must complete before the workflow advances to OrderComplete. If either branch fails, the entire Parallel state fails. This is often exactly the behavior you want: do not mark the order complete until both downstream systems have been notified.


SDK Integrations: Stop Writing Wrapper Lambdas

One of the most common mistakes I see is writing Lambda functions whose only job is to call another AWS service. A Lambda that calls DynamoDB to write a record. A Lambda that sends an SNS message. A Lambda that starts a Glue job.

Step Functions has optimized integrations with over 220 AWS services. You can call these services directly from a state definition without a Lambda in the middle.

Here is a state that writes directly to DynamoDB:

"SaveOrderToDynamo": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "orders",
"Item": {
"orderId": { "S.$": "$.orderId" },
"customerId": { "S.$": "$.customerId" },
"status": { "S": "CONFIRMED" },
"totalAmount":{ "N.$": "States.Format('{}', $.totalAmount)" },
"createdAt": { "S.$": "$$.Execution.StartTime" }
}
},
"Next": "SendToSNS"
}

And a state that publishes to SNS:

"SendToSNS": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"TopicArn": "arn:aws:sns:us-east-1:123456789:order-events",
"Message": {
"orderId.$": "$.orderId",
"customerId.$": "$.customerId",
"status": "CONFIRMED"
}
},
"Next": "OrderComplete"
}

The .$ suffix on a key means “resolve this from the state input.” The $$.Execution.StartTime is a context object reference that gives you metadata about the current execution. These small conveniences add up significantly when building real workflows.

Removing wrapper Lambdas reduces cold starts, lowers your Lambda invocation costs, simplifies your IAM surface, and makes the workflow easier to read because every state’s purpose is self-evident.

The Wait for Callback Pattern

Some workflows cannot move forward until something external happens. A human needs to approve a refund. A third-party payment processor needs to call back. A document needs to pass a review queue.

Step Functions handles this with the waitForTaskToken integration pattern. The state machine pauses, sends a token to an external system, and resumes only when that token is returned.

Here is the state definition:

"WaitForManagerApproval": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789/approval-queue",
"MessageBody": {
"taskToken.$": "$$.Task.Token",
"orderId.$": "$.orderId",
"amount.$": "$.totalAmount",
"requestedBy.$":"$.customerId"
}
},
"HeartbeatSeconds": 3600,
"Next": "ProcessApprovedRefund",
"Catch": [
{
"ErrorEquals": ["ApprovalRejected"],
"Next": "NotifyRejected"
},
{
"ErrorEquals": ["States.HeartbeatTimeout"],
"Next": "EscalateApproval"
}
]
}

The approval service picks up the message, presents it to a manager, and then calls back:

import boto3
sfn = boto3.client("stepfunctions")
def handle_approval_decision(task_token: str, approved: bool, reason: str):
if approved:
sfn.send_task_success(
taskToken=task_token,
output=json.dumps({"approved": True, "approvedBy": "manager@company.com"})
)
else:
sfn.send_task_failure(
taskToken=task_token,
error="ApprovalRejected",
cause=reason
)

The HeartbeatSeconds field is important. If the external system does not send a heartbeat or complete the task within that window, the state fails with a HeartbeatTimeout. In the example above that routes to an escalation state rather than silently hanging forever. Always set a heartbeat on any waitForTaskToken state.

Deploying with Terraform

Defining your state machine in the console is fine for exploration. In production, everything should be in code.

resource "aws_sfn_state_machine" "order_processing" {
name = "order-processing-workflow"
role_arn = aws_iam_role.step_functions_role.arn
type = "STANDARD"
definition = templatefile("${path.module}/state_machine.json", {
validate_order_arn = aws_lambda_function.validate_order.arn
check_inventory_arn = aws_lambda_function.check_inventory.arn
process_payment_arn = aws_lambda_function.process_payment.arn
trigger_fulfillment_arn = aws_lambda_function.trigger_fulfillment.arn
send_email_arn = aws_lambda_function.send_email.arn
})
logging_configuration {
level = "ALL"
include_execution_data = true
log_destination = "${aws_cloudwatch_log_group.sfn_logs.arn}:*"
}
tracing_configuration {
enabled = true
}
}
resource "aws_iam_role" "step_functions_role" {
name = "step-functions-order-processing-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "states.amazonaws.com" }
}]
})
}
resource "aws_iam_role_policy" "sfn_policy" {
name = "sfn-order-processing-policy"
role = aws_iam_role.step_functions_role.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["lambda:InvokeFunction"]
Resource = [
aws_lambda_function.validate_order.arn,
aws_lambda_function.check_inventory.arn,
aws_lambda_function.process_payment.arn,
aws_lambda_function.trigger_fulfillment.arn,
aws_lambda_function.send_email.arn
]
},
{
Effect = "Allow"
Action = ["logs:CreateLogDelivery", "logs:PutLogEvents", "logs:GetLogDelivery"]
Resource = "*"
},
{
Effect = "Allow"
Action = ["xray:PutTraceSegments", "xray:PutTelemetryRecords"]
Resource = "*"
}
]
})
}
resource "aws_cloudwatch_log_group" "sfn_logs" {
name = "/aws/states/order-processing"
retention_in_days = 30
}

Using templatefile to inject Lambda ARNs into the state machine definition keeps your infrastructure code clean and makes it easy to reference the correct function ARN for each environment without hardcoding anything.


Observability in Production

Step Functions gives you three layers of observability out of the box when you configure them properly.

CloudWatch Metrics publishes execution counts, failure rates, and durations for every state machine automatically. Set alarms on ExecutionsFailed and ExecutionsTimedOut. For payment or order workflows, a single failed execution is worth an alert. For high-volume event pipelines, set a threshold based on your acceptable failure rate.

CloudWatch Logs with include_execution_data = true captures the full input and output of every state transition. This is the setting that makes debugging possible. Without it, you know a state failed but not what data it received. With it, you can replay the exact scenario that caused the failure.

X-Ray tracing propagates trace context through Lambda invocations triggered by your state machine. In the AWS console, you get a service map showing exactly where time was spent across each execution. For workflows where latency matters, this is the fastest way to identify the bottleneck.

One practical tip: write a CloudWatch Insights query that you can run immediately when an incident starts.

fields @timestamp, execution_arn, type, details.name, details.status
| filter type in ["ExecutionFailed", "TaskFailed", "TaskStateExited"]
| sort @timestamp desc
| limit 50

Save this query before you need it. Running it during an incident is much faster than clicking through individual executions.


Common Mistakes

Not setting ResultPath on Catch handlers. By default, a Catch block replaces the entire state input with the error object. Your downstream states then receive only the error, not the original order data they need. Always use "ResultPath": "$.error" to merge the error into the existing input.

Using Express Workflows for payment processing. Express Workflows have at-least-once semantics. A state can execute more than once under failure conditions. For anything involving money or external side effects, use Standard Workflows with idempotency keys in your Lambda functions, or use Standard Workflows period.

Ignoring the execution history limit. Standard Workflow execution history is capped at 25,000 events. For very long-running workflows with many state transitions, you can hit this limit. If your workflow runs for days or weeks with thousands of steps, use the Map state with chunking to keep individual execution histories manageable.

Hardcoding ARNs in state machine definitions. Environment-specific ARNs belong in Terraform variables or SSM Parameter Store, not in your state machine JSON. The pattern shown above with templatefile keeps this clean.

Step Functions does not eliminate complexity. What it does is make complexity visible and manageable. Your business logic lives in Lambda. Your orchestration logic lives in the state machine. When something fails, you have a complete, queryable record of exactly what happened and where.

The teams that get the most value from Step Functions are the ones that resist the temptation to build orchestration logic into their Lambda functions. Keep each function focused on a single responsibility. Let the state machine handle sequencing, retries, error routing, and parallelism. The result is a system where debugging takes minutes instead of hours and where new team members can understand the full workflow by reading a single JSON file.

Enjoy the cloud.

Osama

OCI Network Firewall: Building a Centralized Inspection Architecture with Terraform

Security Lists and Network Security Groups handle stateful packet filtering at the subnet and VNIC level. They are the right tool for controlling which ports and protocols reach a resource. What they cannot do is inspect the content of traffic, detect threats based on application-layer signatures, block specific URLs or FQDNs, or apply SSL inspection to decrypt and re-encrypt traffic in flight. That requires a different layer entirely.

OCI Network Firewall is Oracle’s managed next-generation firewall service, built on Palo Alto Networks technology and integrated natively into VCN routing. It supports application-layer inspection, IDPS (Intrusion Detection and Prevention), URL filtering, FQDN-based rules, and TLS inspection. Unlike a third-party firewall appliance you would deploy on a compute instance, OCI Network Firewall is a fully managed service: Oracle handles the underlying infrastructure, HA, and scaling. You manage the policy.

In this post I will walk through designing a hub-and-spoke inspection architecture, deploying the firewall and its policy using Terraform, configuring IDPS and URL filtering rules, and validating traffic flow with OCI Flow Logs.

Architecture: Hub-and-Spoke with Centralized Inspection

The standard pattern for OCI Network Firewall in multi-VCN environments is centralized inspection through a hub VCN. All spoke VCNs route traffic through the hub, and the firewall sits in the hub inspecting both north-south (internet-bound) and east-west (spoke-to-spoke) traffic.

Traffic routing in this architecture uses a combination of DRG route tables and VCN ingress/egress route tables to steer all flows through the firewall subnet before they reach their destination. This is the most important concept to get right: the firewall only inspects traffic that is routed through it. Misconfigured route tables mean packets bypass the firewall entirely with no error or warning.

Step 1: Hub VCN and Firewall Subnet

resource "oci_core_vcn" "hub_vcn" {
compartment_id = var.compartment_id
cidr_blocks = ["192.168.0.0/16"]
display_name = "hub-inspection-vcn"
dns_label = "hubvcn"
}
# Firewall subnet - the firewall VNIC lives here
resource "oci_core_subnet" "firewall_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.hub_vcn.id
cidr_block = "192.168.1.0/24"
display_name = "firewall-subnet"
dns_label = "fwsubnet"
prohibit_public_ip_on_vnic = true
route_table_id = oci_core_route_table.firewall_subnet_rt.id
security_list_ids = [oci_core_security_list.firewall_sl.id]
}
# Internet Gateway for north-south traffic
resource "oci_core_internet_gateway" "hub_igw" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.hub_vcn.id
display_name = "hub-internet-gateway"
enabled = true
}
# DRG for spoke VCN attachment
resource "oci_core_drg" "hub_drg" {
compartment_id = var.compartment_id
display_name = "hub-drg"
}
resource "oci_core_drg_attachment" "hub_vcn_attachment" {
drg_id = oci_core_drg.hub_drg.id
display_name = "hub-vcn-attachment"
network_details {
id = oci_core_vcn.hub_vcn.id
type = "VCN"
}
}

The firewall subnet must not have a public IP on its VNIC. The firewall receives traffic through routing, not through a public endpoint.

Step 2: Firewall Policy

The policy is the heart of the firewall. It contains address lists, URL lists, application lists, and the ordered set of security rules. All of these are defined as child resources of the policy and are applied when the policy is attached to a firewall instance.

resource "oci_network_firewall_network_firewall_policy" "production_policy" {
compartment_id = var.compartment_id
display_name = "production-inspection-policy"
}
# IP address list for trusted internal RFC1918 ranges
resource "oci_network_firewall_network_firewall_policy_address_list" "internal_ranges" {
name = "internal-rfc1918"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
type = "IP"
addresses = [
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16"
]
}
# FQDN list for allowed outbound SaaS destinations
resource "oci_network_firewall_network_firewall_policy_address_list" "allowed_saas" {
name = "allowed-saas-fqdns"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
type = "FQDN"
addresses = [
"*.oracle.com",
"*.oraclecloud.com",
"*.github.com",
"registry-1.docker.io",
"auth.docker.io",
"production.cloudflare.docker.com"
]
}
# URL list for blocked categories
resource "oci_network_firewall_network_firewall_policy_url_list" "blocked_urls" {
name = "blocked-url-categories"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
urls {
pattern = "*.pastebin.com"
type = "SIMPLE"
}
urls {
pattern = "*.ngrok.io"
type = "SIMPLE"
}
urls {
pattern = "*.ngrok-free.app"
type = "SIMPLE"
}
}
# Application list scoping HTTPS traffic
resource "oci_network_firewall_network_firewall_policy_application_group" "web_apps" {
name = "web-traffic"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
apps = ["HTTP", "HTTPS", "SSL"]
}

Step 3: Security Rules

Rules are evaluated in order. The first matching rule wins. Structure your rules from most specific to most general and always end with an explicit deny-all for traffic that does not match any allow rule.

Cross-Cloud Secret Synchronization: AWS Secrets Manager and OCI Vault in a Production Multi-Cloud Setup

One of the most overlooked problems in multi-cloud environments is secrets management across providers. Teams usually solve it badly: they store the same secret in both clouds manually, forget to rotate one of them, and find out during an outage that the credentials have been out of sync for three months.

In this post I will walk through building an automated secrets synchronization pipeline between AWS Secrets Manager and OCI Vault. When a secret rotates in AWS, the pipeline detects the rotation event, retrieves the new value, and pushes it into OCI Vault automatically. Everything is built with Terraform, an AWS Lambda function, and OCI IAM. No manual steps after the initial deployment.

This is a pattern I have used in environments where the database layer runs on OCI (leveraging Oracle Database pricing and performance) while the application layer runs on AWS. Both sides need the same database credentials, and both sides need to stay in sync without human intervention.

Architecture

The flow works like this:

AWS Secrets Manager rotation event fires via EventBridge, which triggers a Lambda function. The Lambda retrieves the new secret value, authenticates to OCI using an API key stored in its own environment (not hardcoded), and calls the OCI Vault API to update the corresponding secret version. OCI Vault stores the new value and makes it available to workloads running in OCI.

Prerequisites

Before starting you need:

  • AWS account with permissions to manage Secrets Manager, Lambda, EventBridge, and IAM
  • OCI tenancy with permissions to manage Vault, Keys, and IAM policies
  • Terraform 1.5 or later
  • Python 3.11 for the Lambda function
  • An existing OCI Vault and master encryption key (or we will create one)

Step 1: OCI Vault and IAM Setup

Start with OCI. We need a Vault, a master key, and an IAM user whose API key the Lambda will use to authenticate.

hcl

# OCI Vault
resource "oci_kms_vault" "app_vault" {
compartment_id = var.compartment_id
display_name = "multi-cloud-secrets-vault"
vault_type = "DEFAULT"
}
# Master Encryption Key inside the Vault
resource "oci_kms_key" "secrets_key" {
compartment_id = var.compartment_id
display_name = "secrets-master-key"
management_endpoint = oci_kms_vault.app_vault.management_endpoint
key_shape {
algorithm = "AES"
length = 32
}
}
# IAM user for cross-cloud access
resource "oci_identity_user" "sync_user" {
compartment_id = var.tenancy_ocid
name = "aws-secrets-sync-user"
description = "Service user for AWS Lambda to push secrets into OCI Vault"
email = "sync-user@internal.example.com"
}
# API key for the sync user (you will generate the actual key pair separately)
resource "oci_identity_api_key" "sync_user_key" {
user_id = oci_identity_user.sync_user.id
key_value = var.oci_sync_user_public_key_pem
}
# IAM group for the sync user
resource "oci_identity_group" "sync_group" {
compartment_id = var.tenancy_ocid
name = "secrets-sync-group"
description = "Group for cross-cloud secrets sync service users"
}
resource "oci_identity_user_group_membership" "sync_membership" {
group_id = oci_identity_group.sync_group.id
user_id = oci_identity_user.sync_user.id
}
# Minimal IAM policy - only what is needed, nothing more
resource "oci_identity_policy" "sync_policy" {
compartment_id = var.compartment_id
name = "secrets-sync-policy"
description = "Allows sync user to manage secrets in the app vault only"
statements = [
"Allow group secrets-sync-group to manage secret-family in compartment id ${var.compartment_id} where target.vault.id = '${oci_kms_vault.app_vault.id}'",
"Allow group secrets-sync-group to use keys in compartment id ${var.compartment_id} where target.key.id = '${oci_kms_key.secrets_key.id}'"
]
}

The policy scope is intentionally narrow. The sync user can only manage secrets inside this specific vault and can only use this specific key. If the AWS Lambda credentials are ever compromised, the blast radius is limited to this vault.

Step 2: Create the Initial Secret in OCI Vault

We need a secret placeholder in OCI Vault that the Lambda will update. The initial value does not matter since it will be overwritten on the first sync.

hcl

resource "oci_vault_secret" "db_password" {
compartment_id = var.compartment_id
vault_id = oci_kms_vault.app_vault.id
key_id = oci_kms_key.secrets_key.id
secret_name = "prod-db-password"
secret_content {
content_type = "BASE64"
content = base64encode("initial-placeholder-value")
name = "v1"
stage = "CURRENT"
}
metadata = {
source = "aws-secrets-manager"
aws_secret = "prod/database/password"
environment = "production"
}
}

Step 3: AWS Secrets Manager and the Source Secret

On the AWS side, create the authoritative secret and enable automatic rotation.

hcl

resource "aws_secretsmanager_secret" "db_password" {
name = "prod/database/password"
description = "Production database password - synced to OCI Vault"
recovery_window_in_days = 7
tags = {
Environment = "production"
SyncTarget = "oci-vault"
OciSecretName = "prod-db-password"
}
}
resource "aws_secretsmanager_secret_version" "db_password_v1" {
secret_id = aws_secretsmanager_secret.db_password.id
secret_string = jsonencode({
username = "db_admin",
password = var.initial_db_password,
host = var.db_host,
port = 1521,
database = "PRODDB"
})
}
# Rotation configuration - rotate every 30 days
resource "aws_secretsmanager_secret_rotation" "db_password_rotation" {
secret_id = aws_secretsmanager_secret.db_password.id
rotation_lambda_arn = aws_lambda_function.db_rotation_lambda.arn
rotation_rules {
automatically_after_days = 30
}
}

Step 4: Store OCI Credentials in AWS Secrets Manager

The Lambda needs OCI API credentials to authenticate. Store them as a secret in AWS Secrets Manager so they never appear in Lambda environment variables in plaintext.

hcl

resource "aws_secretsmanager_secret" "oci_credentials" {
name = "internal/oci-sync-credentials"
description = "OCI API key credentials for secrets sync Lambda"
tags = {
Environment = "production"
Purpose = "cross-cloud-sync"
}
}
resource "aws_secretsmanager_secret_version" "oci_credentials_v1" {
secret_id = aws_secretsmanager_secret.oci_credentials.id
secret_string = jsonencode({
tenancy_ocid = var.oci_tenancy_ocid,
user_ocid = var.oci_sync_user_ocid,
fingerprint = var.oci_api_key_fingerprint,
private_key = var.oci_private_key_pem,
region = var.oci_region
})
}

Step 5: The Lambda Function

This is the core of the pipeline. The Lambda retrieves the rotated secret from AWS Secrets Manager, loads OCI credentials from its own secrets store, and calls the OCI Vault API to create a new secret version.

python

import boto3
import json
import base64
import oci
import logging
import os
from datetime import datetime, timezone
from botocore.exceptions import ClientError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def get_oci_config():
"""Retrieve OCI credentials from AWS Secrets Manager."""
client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
try:
response = client.get_secret_value(
SecretId=os.environ["OCI_CREDENTIALS_SECRET_ARN"]
)
creds = json.loads(response["SecretString"])
return {
"tenancy": creds["tenancy_ocid"],
"user": creds["user_ocid"],
"fingerprint": creds["fingerprint"],
"key_content": creds["private_key"],
"region": creds["region"]
}
except ClientError as e:
logger.error(f"Failed to retrieve OCI credentials: {e}")
raise
def get_aws_secret(secret_arn: str) -> str:
"""Retrieve the current value of an AWS secret."""
client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
try:
response = client.get_secret_value(SecretId=secret_arn)
return response.get("SecretString") or base64.b64decode(
response["SecretBinary"]
).decode("utf-8")
except ClientError as e:
logger.error(f"Failed to retrieve AWS secret {secret_arn}: {e}")
raise
def push_to_oci_vault(
oci_config: dict,
vault_id: str,
key_id: str,
secret_ocid: str,
secret_value: str
):
"""Create a new version of an OCI Vault secret."""
vaults_client = oci.vault.VaultsClient(oci_config)
encoded_value = base64.b64encode(secret_value.encode("utf-8")).decode("utf-8")
update_details = oci.vault.models.UpdateSecretDetails(
secret_content=oci.vault.models.Base64SecretContentDetails(
content_type=oci.vault.models.SecretContentDetails.CONTENT_TYPE_BASE64,
content=encoded_value,
name=f"sync-{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}",
stage="CURRENT"
),
metadata={
"synced_from": "aws-secrets-manager",
"synced_at": datetime.now(timezone.utc).isoformat()
}
)
response = vaults_client.update_secret(
secret_id=secret_ocid,
update_secret_details=update_details
)
logger.info(
f"OCI secret updated. OCID: {secret_ocid}, "
f"New version: {response.data.current_version_number}"
)
return response.data
def handler(event, context):
"""
EventBridge trigger handler.
Expects event detail to contain:
- aws_secret_arn: ARN of the rotated AWS secret
- oci_secret_ocid: OCID of the target OCI Vault secret
- oci_vault_id: OCID of the target OCI Vault
- oci_key_id: OCID of the OCI KMS key
"""
logger.info(f"Received event: {json.dumps(event)}")
detail = event.get("detail", {})
aws_secret_arn = detail.get("aws_secret_arn")
oci_secret_ocid = detail.get("oci_secret_ocid")
oci_vault_id = detail.get("oci_vault_id")
oci_key_id = detail.get("oci_key_id")
if not all([aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id]):
logger.error("Missing required fields in event detail")
raise ValueError("Event detail must include aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id")
logger.info(f"Syncing secret: {aws_secret_arn} to OCI: {oci_secret_ocid}")
# Step 1: Get OCI credentials
oci_config = get_oci_config()
# Step 2: Retrieve the rotated AWS secret
secret_value = get_aws_secret(aws_secret_arn)
# Step 3: Push to OCI Vault
result = push_to_oci_vault(
oci_config=oci_config,
vault_id=oci_vault_id,
key_id=oci_key_id,
secret_ocid=oci_secret_ocid,
secret_value=secret_value
)
return {
"statusCode": 200,
"body": {
"message": "Secret synced successfully",
"oci_secret_ocid": oci_secret_ocid,
"oci_version": result.current_version_number
}
}

Step 6: Lambda IAM Role and Deployment

hcl

data "aws_iam_policy_document" "lambda_assume_role" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["lambda.amazonaws.com"]
}
}
}
data "aws_iam_policy_document" "lambda_permissions" {
statement {
effect = "Allow"
actions = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
]
resources = [
aws_secretsmanager_secret.db_password.arn,
aws_secretsmanager_secret.oci_credentials.arn
]
}
statement {
effect = "Allow"
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
resources = ["arn:aws:logs:*:*:*"]
}
}
resource "aws_iam_role" "sync_lambda_role" {
name = "secrets-sync-lambda-role"
assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}
resource "aws_iam_role_policy" "sync_lambda_policy" {
name = "secrets-sync-lambda-policy"
role = aws_iam_role.sync_lambda_role.id
policy = data.aws_iam_policy_document.lambda_permissions.json
}
resource "aws_lambda_function" "secrets_sync" {
filename = "${path.module}/lambda/secrets_sync.zip"
function_name = "oci-secrets-sync"
role = aws_iam_role.sync_lambda_role.arn
handler = "main.handler"
runtime = "python3.11"
timeout = 60
memory_size = 256
source_code_hash = filebase64sha256("${path.module}/lambda/secrets_sync.zip")
environment {
variables = {
OCI_CREDENTIALS_SECRET_ARN = aws_secretsmanager_secret.oci_credentials.arn
AWS_REGION = var.aws_region
}
}
layers = [aws_lambda_layer_version.oci_sdk_layer.arn]
}

Bundle the OCI Python SDK as a Lambda Layer so the function does not need to package it inline:

bash

mkdir -p lambda_layer/python
pip install oci --target lambda_layer/python
cd lambda_layer && zip -r ../oci_sdk_layer.zip python/

hcl

resource "aws_lambda_layer_version" "oci_sdk_layer" {
filename = "${path.module}/oci_sdk_layer.zip"
layer_name = "oci-python-sdk"
compatible_runtimes = ["python3.11"]
source_code_hash = filebase64sha256("${path.module}/oci_sdk_layer.zip")
}

Step 7: EventBridge Rule to Trigger on Rotation

hcl

resource "aws_cloudwatch_event_rule" "secret_rotation_rule" {
name = "detect-secret-rotation"
description = "Fires when a Secrets Manager secret rotation completes"
event_pattern = jsonencode({
source = ["aws.secretsmanager"],
detail-type = ["AWS API Call via CloudTrail"],
detail = {
eventSource = ["secretsmanager.amazonaws.com"],
eventName = ["RotateSecret", "PutSecretValue"]
}
})
}
resource "aws_cloudwatch_event_target" "sync_lambda_target" {
rule = aws_cloudwatch_event_rule.secret_rotation_rule.name
target_id = "SyncToOCI"
arn = aws_lambda_function.secrets_sync.arn
input_transformer {
input_paths = {
secret_arn = "$.detail.requestParameters.secretId"
}
input_template = <<EOF
{
"detail": {
"aws_secret_arn": "<secret_arn>",
"oci_secret_ocid": "${var.oci_db_password_secret_ocid}",
"oci_vault_id": "${oci_kms_vault.app_vault.id}",
"oci_key_id": "${oci_kms_key.secrets_key.id}"
}
}
EOF
}
}
resource "aws_lambda_permission" "allow_eventbridge" {
statement_id = "AllowEventBridgeInvoke"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.secrets_sync.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.secret_rotation_rule.arn
}

Step 8: Verifying the Pipeline

Manually trigger a rotation to test the full pipeline without waiting 30 days:

bash

# Force a rotation in AWS
aws secretsmanager rotate-secret \
--secret-id prod/database/password \
--region us-east-1
# Check Lambda execution logs
aws logs tail /aws/lambda/oci-secrets-sync --follow
# Verify the new version appeared in OCI Vault
oci vault secret get \
--secret-id <your-oci-secret-ocid> \
--query 'data.{name:secret-name, version:"current-version-number", updated:"time-of-current-version-need-rotation"}' \
--output table

A successful sync produces output similar to this in the Lambda logs:

INFO: Syncing secret: arn:aws:secretsmanager:us-east-1:123456789:secret:prod/database/password to OCI: ocid1.vaultsecret.oc1...
INFO: OCI secret updated. OCID: ocid1.vaultsecret.oc1..., New version: 3

Handling Failures and Drift

The pipeline as built is synchronous and event-driven, which means if the Lambda fails, the OCI secret does not get updated. Add a dead-letter queue and a reconciliation function that runs on a schedule to catch any drift.

hcl

resource "aws_sqs_queue" "sync_dlq" {
name = "secrets-sync-dlq"
message_retention_seconds = 86400
}
resource "aws_lambda_function_event_invoke_config" "sync_retry" {
function_name = aws_lambda_function.secrets_sync.function_name
maximum_retry_attempts = 2
maximum_event_age_in_seconds = 300
destination_config {
on_failure {
destination = aws_sqs_queue.sync_dlq.arn
}
}
}

For reconciliation, a scheduled Lambda that runs every hour compares the LastRotatedDate on the AWS secret against the synced_at metadata tag on the OCI secret. If they differ by more than five minutes, it triggers a forced sync.

Security Considerations

A few things to keep in mind when running this in production.

The OCI private key stored in AWS Secrets Manager should be rotated periodically, just like any other credential. Add it to your rotation schedule.

Enable CloudTrail in AWS and OCI Audit logging so every access to both secrets stores is recorded. If something is off with the sync, the audit logs tell you exactly which principal made the change and when.

Use VPC endpoints for Secrets Manager in AWS so the Lambda traffic never crosses the public internet when retrieving credentials.

On the OCI side, enable Vault audit logging to the OCI Logging service so every secret version write is captured.

Wrapping Up

This pipeline solves a real operational problem without requiring a third-party secrets broker. AWS Secrets Manager stays the authoritative source. OCI Vault stays current automatically. The only manual step is the initial deployment.

The pattern extends to other cross-cloud credential types. Database connection strings, API tokens, TLS certificates — any secret that needs to exist on both clouds can follow the same EventBridge to Lambda to OCI Vault flow. Extend the Lambda to support a mapping table of AWS secret ARNs to OCI secret OCIDs and one function handles your entire secrets estate across both providers.

Regards,
Osama

Building Kubernetes Sentinel: An AI-Powered Cluster Health Dashboard

When you manage Kubernetes clusters at scale, the hardest part is not keeping things running. It is knowing when something is about to break, understanding why it broke, and fixing it before it affects users. Traditional monitoring tools give you metrics and alerts, but they leave the diagnosis entirely up to you. You still have to correlate events, read logs, cross-reference namespaces, and figure out the right kubectl commands to run.

I wanted to change that. So I built Kubernetes Sentinel, an open-source dashboard that not only watches your entire cluster in real time but also uses Claude AI to explain what went wrong and tell you exactly how to fix it.

The Problem with Kubernetes Observability

Anyone who has been on call for a Kubernetes cluster knows the feeling. Your phone goes off at 2am. A pod is crashlooping. You open your terminal, start running kubectl commands, and spend the next twenty minutes piecing together what happened from logs, events, and resource descriptions spread across multiple namespaces.

The tooling has not kept up with the complexity. Prometheus and Grafana are powerful, but they require significant setup and expertise to use effectively. Most teams end up with dashboards full of graphs they never look at and alerts that fire so often they get ignored.

What I wanted was something simpler. A single view of the entire cluster, automatic detection of anything that looks wrong, and an AI that could look at the same data an experienced SRE would look at and tell me what is happening in plain English.

What Kubernetes Sentinel Does

Kubernetes Sentinel is a FastAPI backend that runs either locally or as a pod inside your cluster. It polls the Kubernetes API every 15 seconds across all namespaces, not just one, and stores the current state in memory. A React frontend connects to it over HTTP and receives live updates via Server-Sent Events.

The dashboard gives you four things at once. A health score from 0 to 100 that reflects the overall state of your cluster. A live pod table showing every pod across every namespace with restart counts, phase, and node assignment. An event stream showing everything Kubernetes has logged, filtered and color-coded by severity. And a resources view covering your nodes, deployments, services, and persistent volume claims.

On top of that, the backend runs seven anomaly detection rules continuously. CrashLoopBackOff, OOMKilled, NodeNotReady, FailedMount, BackOff, CPUThrottling, and high restart counts. When any of these fire, an anomaly banner appears at the top of the dashboard immediately.

The AI diagnosis feature is where it gets interesting. When you click Run Diagnosis, the backend assembles the current cluster state into a structured prompt and sends it to Claude. Within seconds you get back a plain-English summary of what is wrong, a root cause explanation, and three kubectl commands you can copy and run immediately to fix it. No more correlating events manually. No more searching Stack Overflow for the right flags.

The Technical Decisions

I made a few deliberate architectural choices that I think are worth explaining.

The backend runs as a single process with one Uvicorn worker. This is intentional. The background polling thread lives inside the same process, so multiple workers would each start their own independent loop and you would end up with redundant API calls and inconsistent state. One process, one source of truth.

Authentication with the Kubernetes API uses the official Python client, which handles both scenarios automatically. When the sentinel runs inside a cluster as a pod, it reads the ServiceAccount token that Kubernetes mounts automatically at a well-known path. When you run it locally for development, it falls back to your kubeconfig. The same code works in both environments without any changes.

The RBAC configuration is strictly read-only. The ClusterRole I wrote grants get, list, and watch on pods, events, nodes, services, persistent volume claims, configmaps, secrets, deployments, statefulsets, daemonsets, and replicasets. Nothing else. The sentinel can observe everything but change nothing. This was a hard requirement for me. A monitoring tool should never have write access to the cluster it is watching.

For the frontend I deliberately chose a single React file with no build step. The dashboard runs as a Claude.ai artifact or drops straight into any React project. There is nothing to compile, no node_modules to install, no webpack config to debug. The entire UI is one file you can read and understand in an afternoon.

I also added a DEV_MODE flag that bypasses the Kubernetes connection entirely and loads realistic mock data instead. This means anyone can clone the repo, set DEV_MODE=true, start the backend, and see the full dashboard working within five minutes even if they have never touched Kubernetes before. It made development much faster and makes the project far more accessible for contributors.

The Stack

The backend is Python 3.12 with FastAPI and the official Kubernetes client library. I used sse-starlette for Server-Sent Events, httpx for calling the Claude API, and Pydantic v2 for data validation. The Docker image is a two-stage build that ends up running as a non-root user.

The frontend is React 18 with no external UI library. All styling is plain inline JavaScript objects, which makes it trivially portable and means there are zero CSS conflicts when you embed it somewhere else.

Kubernetes manifests cover the full production deployment: namespace, ClusterRole, ClusterRoleBinding, ServiceAccount, ConfigMap, Deployment with liveness and readiness probes, and a ClusterIP Service. The Anthropic API key is never stored in any manifest file. It goes into a Kubernetes Secret created directly with kubectl.

What I Learned Building This

The biggest challenge was not the Kubernetes integration or the AI features. It was the import path problem. Claude Code generated all the backend files correctly, but because the server is started from inside the backend directory, every import had to be relative to that directory as the root. Files using from backend.core.x import y worked fine in isolation but crashed immediately when uvicorn tried to load them. Once I understood the issue it was a one-line fix in every file, but it cost me an hour of debugging.

The second thing I learned is that mock data is not optional for a project like this. Without DEV_MODE, you need a running Kubernetes cluster to develop against, which means either paying for cloud infrastructure or running a local cluster with kind. Adding ten lines of mock data to the poller made the development loop dramatically faster and opened the project up to contributors who want to work on the frontend without needing any cluster at all.

The AI diagnosis feature turned out to be far more useful than I expected. I assumed it would be a nice addition but not something I would rely on. After running it against realistic failure scenarios, the quality of the root cause analysis was genuinely impressive. It correctly identified memory limit misconfiguration from OOMKill events, correlated restart back-off with recent image pull failures, and suggested the right sequence of commands to investigate and resolve each issue.

Running It Yourself

The project is open source and available on GitHub. There are three ways to run it.

If you just want to see the dashboard without any cluster setup, clone the repo, copy .env.example to .env, set DEV_MODE=true and your Anthropic API key, then run uvicorn from the backend directory. The whole setup takes under five minutes.

If you have a Kubernetes cluster, set DEV_MODE=false and point it at your kubeconfig. The backend will start polling your real cluster immediately and the dashboard will show live data.

If you want to run it inside your cluster, build the Docker image, push it to your registry, create a Kubernetes Secret with your API key, and apply the manifests with kubectl. The deploy script handles the apply order automatically.

The repository is at https://github.com/OsamaOracle/k8s-sentinel/. Contributions, issues, and feedback are welcome.

Regards
Osama

Kubernetes in the Multi-cloud: Orchestrating Workloads Across AWS and OCI

Why Multicloud Kubernetes Is No Longer Optional

The conversation has shifted. Running Kubernetes on a single cloud provider was once considered best practice simpler networking, unified IAM, one support contract. But modern enterprise reality tells a different story.

Vendor lock-in risk, regional compliance mandates, cost arbitrage opportunities, and resilience requirements are pushing engineering teams to operate Kubernetes clusters across multiple clouds simultaneously. Among the most compelling combinations today is AWS (EKS) paired with Oracle Cloud Infrastructure (OCI/OKE) two providers with fundamentally different strengths that, when combined, can form a genuinely powerful platform.

This post walks through the architectural decisions, tooling choices, and operational patterns for running a production-grade multicloud Kubernetes setup spanning AWS EKS and OCI OKE.

Understanding What Each Cloud Brings

Before designing a multicloud strategy, you need to be honest about why you’re using each provider not just “for redundancy.”

AWS EKS is mature, battle-tested, and has the richest ecosystem of Kubernetes-native tooling. Its managed node groups, Karpenter autoscaler, and deep integration with IAM Roles for Service Accounts (IRSA) make it a natural fit for compute-heavy, stateless microservices. The tradeoff: cost can escalate fast at scale.

OCI OKE (Oracle Container Engine for Kubernetes) is increasingly competitive on price, particularly for compute and egress and has genuine strengths in Oracle Database integrations, bare metal instances, and deterministic network performance via its RDMA fabric. For workloads that touch Oracle DB, Exadata, or need high-throughput interconnects, OKE is not just a fallback, it’s the right tool.

The insight that unlocks a real multicloud strategy: stop treating one cloud as primary and the other as DR. Design for active-active.

The Core Architecture

A production multicloud Kubernetes setup across EKS and OKE requires solving four problems:

  1. Cluster federation or virtual cluster abstraction
  2. Cross-cloud networking
  3. Unified identity and secrets management
  4. Consistent GitOps delivery

Let’s break each down.


1. Cluster Federation: Choosing Your Control Plane Philosophy

There are two schools of thought:

Option A Independent clusters, unified GitOps (recommended) Each cluster (EKS, OKE) is fully autonomous. A GitOps tool typically Flux or Argo CD manages both from a single source of truth. No shared control plane exists between clusters. Workloads are deployed to each cluster independently based on targeting labels or Kustomize overlays.

Option B Virtual Cluster Mesh (Liqo, Admiralty, or Karmada) Tools like Karmada introduce a meta-control plane that federates multiple clusters. You submit workloads to the Karmada API server, and it distributes them across member clusters based on propagation policies.

For most teams, Option A is the right starting point. Karmada adds power but also operational complexity. The GitOps approach keeps blast radius contained a misconfiguration in one cluster doesn’t cascade.


2. Cross-Cloud Networking: The Hard Problem

Kubernetes pods in EKS can’t natively reach pods in OKE, and vice versa. You need a data plane that spans both clouds.

Recommended approach: WireGuard-based mesh with Cilium Cluster Mesh

Cilium’s Cluster Mesh feature allows pods across clusters to communicate using their native pod IPs, with WireGuard encryption in transit. The setup requires:

  • Each cluster runs Cilium as its CNI (replacing the default VPC CNI on EKS and the flannel-based CNI on OKE)
  • A ClusterMesh resource is created linking the two API servers
  • Cross-cluster ServiceExport and ServiceImport resources (via the Kubernetes MCS API) expose services across the mesh

On the infrastructure layer, you need an encrypted tunnel between your AWS VPC and OCI VCN. Options:

  • Site-to-site VPN (quickest to set up, ~1.25 Gbps cap)
  • AWS Direct Connect + OCI FastConnect (for production private, dedicated bandwidth)
  • Overlay via Tailscale or Netbird (great for dev/staging multicloud setups, not production-grade for high-throughput)

yaml

# Example: Cilium ClusterMesh config snippet
apiVersion: cilium.io/v2alpha1
kind: CiliumClusterwideNetworkPolicy
metadata:
name: allow-cross-cluster-services
spec:
endpointSelector: {}
ingress:
- fromEndpoints:
- matchLabels:
io.cilium.k8s.policy.cluster: oci-oke-prod

3. Unified Identity: IRSA on AWS, Workload Identity on OCI

This is where multicloud gets philosophically interesting. Each cloud has its own identity system, and they don’t speak the same language.

On AWS (EKS): Use IRSA (IAM Roles for Service Accounts). Your pod’s service account is annotated with an IAM role ARN. The Pod Identity Webhook injects environment variables that allow the AWS SDK to exchange a projected service account token for temporary AWS credentials.

On OCI (OKE): Use OCI Workload Identity, introduced in recent OKE versions. It works analogously to IRSA a Kubernetes service account is bound to an OCI Dynamic Group and IAM policy, and the pod receives a workload identity token that can be exchanged for OCI API credentials.

The challenge: your application code should not need to know which cloud it’s running on. Use a secrets abstraction layer.

External Secrets Operator (ESO) elegantly solves this. Deploy ESO on both clusters. Point the EKS instance at AWS Secrets Manager; point the OKE instance at OCI Vault. Your application consumes a SecretStore resource with a consistent name. ESO handles the transparent fetching of backend-specific credentials.

# SecretStore on EKS (AWS Secrets Manager backend)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: app-secrets
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
---
# SecretStore on OKE (OCI Vault backend) same name, different spec
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: app-secrets
spec:
provider:
oracle:
vault: ocid1.vault.oc1...
region: us-ashburn-1
auth:
workloadIdentity: {}
```
Your application's `ExternalSecret` resources reference `app-secrets` in both environments the YAML is identical.
### 4. GitOps: One Repository, Multiple Targets
Use **Argo CD ApplicationSets** or **Flux's `Kustomization` with cluster selectors** to manage both clusters from a monorepo.
A typical repo layout:
```
/clusters
/eks-us-east-1
kustomization.yaml # EKS-specific patches
/oke-us-ashburn-1
kustomization.yaml # OKE-specific patches
/base
/apps
deployment.yaml
service.yaml
/infra
external-secrets.yaml
cilium-config.yaml

Flux’s Kustomization resource lets you target specific clusters using the cluster’s kubeconfig context or label selectors. Argo CD’s ApplicationSet with a list generator can enumerate your clusters and deploy the same app with environment-specific values.

The key rule: the base layer must be cloud-agnostic. Patches in cluster-specific overlays handle anything that diverges storage classes, ingress annotations, node selectors.


Observability Across Clouds

A multicloud cluster setup with no unified observability is an incident waiting to happen.

Recommended stack:

  • Prometheus + Thanos for metrics each cluster runs Prometheus; Thanos Sidecar ships blocks to object storage (S3 on AWS, OCI Object Storage on OCI); Thanos Querier federates across both
  • Grafana with both Thanos endpoints as datasources single pane of glass
  • OpenTelemetry Collector deployed as a DaemonSet on each cluster, shipping traces to a common backend (Grafana Tempo, Jaeger, or Honeycomb)
  • Loki for logs, with agents on each cluster shipping to a common Loki instance

Label discipline is critical: ensure every metric, trace, and log carries cluster, cloud_provider, and region labels from the source. Without this, correlation during incidents across clouds becomes extremely difficult.


Cost Management: The Overlooked Dimension

Multicloud adds a new cost vector: egress. Data leaving AWS costs money. Data entering OCI is free. Cross-cloud service calls that seemed free in a single-cloud setup now carry per-GB charges.

Practical rules:

  • Colocate tightly coupled services in the same cluster/cloud don’t split microservices that call each other thousands of times per second across clouds
  • Use Cilium’s network policy to audit cross-cluster traffic volume before enabling services in the mesh
  • Consider OCI’s free egress to the internet for user-facing workloads where latency to OCI regions is acceptable
  • Tag every namespace with cost center labels and use Kubecost or OpenCost deployed on each cluster with a shared object storage backend for unified cost attribution

Operational Runbook Considerations

A few things that will bite you if not planned for:

Clock skew: mTLS certificates and OIDC token validation are sensitive to time drift. Ensure NTP is configured identically on all nodes across both clouds. A 5-minute clock skew will silently break IRSA on EKS and workload identity on OKE.

DNS: Use ExternalDNS on both clusters pointing to a shared DNS provider (Route 53, Cloudflare). Services that need cross-cloud discoverability get DNS entries automatically on deploy.

Cluster upgrades: EKS and OKE release Kubernetes versions on different schedules. Maintain a maximum one-minor-version skew between clusters. Use a canary upgrade pattern: upgrade your OKE cluster first (typically lower blast radius), validate for 48 hours, then upgrade EKS.

Node image parity: Your application containers are cloud-agnostic, but your node OS images are not. Use Bottlerocket on EKS and Oracle Linux 8 on OKE both are minimal, hardened, and have predictable patching cycles.


When NOT to Do This

Multicloud Kubernetes is a force multiplier but only if your team has the operational maturity to support it.

Don’t pursue this architecture if:

  • Your team is still stabilizing single-cluster Kubernetes operations
  • Your workloads have no actual cross-cloud requirement (cost, compliance, or resilience)
  • You lack dedicated platform engineering capacity to maintain the toolchain
  • Your application isn’t designed for network partitioning tolerance

A well-run single-cloud EKS or OKE setup will outperform a poorly-run multicloud one every time. Add complexity only when you’ve exhausted simpler options.


Closing Thoughts

The multicloud Kubernetes story has matured considerably. Tools like Cilium Cluster Mesh, External Secrets Operator, Karmada, and OpenTelemetry have closed most of the operational gaps that made this approach impractical two years ago.

The AWS + OCI combination in particular is underrated. AWS brings ecosystem breadth; OCI brings pricing, Oracle database integration, and a network fabric that punches above its weight. For the right workloads and with the right tooling discipline the combination is genuinely compelling.

The architecture isn’t magic. It’s plumbing. But when it’s done right, it disappears and your developers ship to two clouds the same way they ship to one.


Have questions about multicloud Kubernetes design or EKS/OKE specifics? Reach out or leave a comment below.

Building a Multi-Cloud Secrets Management Strategy with HashiCorp Vault

Let me ask you something. Where are your database passwords right now? Your API keys? Your TLS certificates?

If you’re like most teams I’ve worked with, the honest answer is “scattered everywhere.” Some are in environment variables. Some are in Kubernetes secrets (base64 encoded, which isn’t encryption by the way). A few are probably still hardcoded in configuration files that someone committed to Git three years ago.

I’m not judging. We’ve all been there. But as your infrastructure grows across multiple clouds, this approach becomes a ticking time bomb. One leaked credential can compromise everything.

In this article, I’ll show you how to build a centralized secrets management strategy using HashiCorp Vault. We’ll deploy it properly, integrate it with AWS, Azure, and GCP, and set up dynamic secrets that rotate automatically. No more shared passwords. No more “who has access to what” mysteries.

Why Vault? Why Now?

Before we dive into implementation, let me explain why I recommend Vault over cloud-native solutions like AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager.

Don’t get me wrong. Those services are excellent. If you’re running entirely on one cloud, they might be all you need. But here’s the reality for most organizations:

You have workloads on AWS. Your data team uses GCP for BigQuery. Your enterprise applications run on Azure. Maybe you still have some on-premises systems. And you need a consistent way to manage secrets across all of them.

Vault gives you that single control plane. One audit log. One policy engine. One place to rotate credentials. And it integrates with everything.

Architecture Overview

Here’s what we’re building:

The key principle here is that applications never store long-lived credentials. Instead, they authenticate to Vault and receive short-lived, automatically rotated credentials for the specific resources they need.


Building a Multi-Cloud Secrets Management Strategy with HashiCorp Vault

Let me ask you something. Where are your database passwords right now? Your API keys? Your TLS certificates?

If you’re like most teams I’ve worked with, the honest answer is “scattered everywhere.” Some are in environment variables. Some are in Kubernetes secrets (base64 encoded, which isn’t encryption by the way). A few are probably still hardcoded in configuration files that someone committed to Git three years ago.

I’m not judging. We’ve all been there. But as your infrastructure grows across multiple clouds, this approach becomes a ticking time bomb. One leaked credential can compromise everything.

In this article, I’ll show you how to build a centralized secrets management strategy using HashiCorp Vault. We’ll deploy it properly, integrate it with AWS, Azure, and GCP, and set up dynamic secrets that rotate automatically. No more shared passwords. No more “who has access to what” mysteries.

Why Vault? Why Now?

Before we dive into implementation, let me explain why I recommend Vault over cloud-native solutions like AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager.

Don’t get me wrong. Those services are excellent. If you’re running entirely on one cloud, they might be all you need. But here’s the reality for most organizations:

You have workloads on AWS. Your data team uses GCP for BigQuery. Your enterprise applications run on Azure. Maybe you still have some on-premises systems. And you need a consistent way to manage secrets across all of them.

Vault gives you that single control plane. One audit log. One policy engine. One place to rotate credentials. And it integrates with everything.

Architecture Overview

Here’s what we’re building:

The key principle here is that applications never store long-lived credentials. Instead, they authenticate to Vault and receive short-lived, automatically rotated credentials for the specific resources they need.

Step 1: Deploy Vault on Kubernetes

I prefer running Vault on Kubernetes because it gives you high availability, easy scaling, and integrates beautifully with your existing workloads. We’ll use the official Helm chart.

Prerequisites

You’ll need a Kubernetes cluster. Any managed Kubernetes service works: EKS, AKS, GKE, or even OKE. For this guide, I’ll use commands that work across all of them.

Create the Namespace and Storage

bash

kubectl create namespace vault
# Create storage class for Vault data
# This example uses AWS EBS, adjust for your cloud
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: vault-storage
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF

Configure Vault Helm Values

yaml

# vault-values.yaml
global:
enabled: true
tlsDisable: false
injector:
enabled: true
replicas: 2
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 512Mi
cpu: 500m
server:
enabled: true
# Run 3 replicas for high availability
ha:
enabled: true
replicas: 3
# Use Raft for integrated storage
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = false
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
tls_key_file = "/vault/userconfig/vault-tls/tls.key"
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://vault-1.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
retry_join {
leader_api_addr = "https://vault-2.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
}
}
service_registration "kubernetes" {}
seal "awskms" {
region = "us-east-1"
kms_key_id = "alias/vault-unseal-key"
}
resources:
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 2000m
dataStorage:
enabled: true
size: 20Gi
storageClass: vault-storage
auditStorage:
enabled: true
size: 10Gi
storageClass: vault-storage
# Service account for cloud integrations
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/vault-server-role
ui:
enabled: true
serviceType: LoadBalancer
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Generate TLS Certificates

Vault should always use TLS. Here’s how to create certificates using cert-manager:

yaml

# vault-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: vault-tls
namespace: vault
spec:
secretName: vault-tls
duration: 8760h # 1 year
renewBefore: 720h # 30 days
subject:
organizations:
- YourCompany
commonName: vault.vault.svc.cluster.local
dnsNames:
- vault
- vault.vault
- vault.vault.svc
- vault.vault.svc.cluster.local
- vault-0.vault-internal
- vault-1.vault-internal
- vault-2.vault-internal
- "*.vault-internal"
ipAddresses:
- 127.0.0.1
issuerRef:
name: cluster-issuer
kind: ClusterIssuer

Install Vault

bash

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault \
--namespace vault \
--values vault-values.yaml \
--version 0.27.0

Initialize and Unseal

This is a one-time operation. Keep these keys safe. I mean really safe. Like offline, in multiple secure locations.

bash

# Initialize Vault
kubectl exec -n vault vault-0 -- vault operator init \
-key-shares=5 \
-key-threshold=3 \
-format=json > vault-init.json
# The output contains your unseal keys and root token
# Store these securely!
# If not using auto-unseal, you'd need to unseal manually:
# kubectl exec -n vault vault-0 -- vault operator unseal <key1>
# kubectl exec -n vault vault-0 -- vault operator unseal <key2>
# kubectl exec -n vault vault-0 -- vault operator unseal <key3>
# With AWS KMS auto-unseal configured, Vault unseals automatically

Step 2: Configure Authentication Methods

Now we need to tell Vault how applications will authenticate. This is where it gets interesting.

Kubernetes Authentication

Applications running in Kubernetes can authenticate using their service account tokens. No passwords needed.

bash

# Enable Kubernetes auth
vault auth enable kubernetes
# Configure it to trust our cluster
vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
issuer="https://kubernetes.default.svc.cluster.local"

AWS IAM Authentication

For workloads running on EC2, Lambda, or ECS, they can authenticate using their IAM roles.

bash

# Enable AWS auth
vault auth enable aws
# Configure AWS credentials for Vault to verify requests
vault write auth/aws/config/client \
secret_key=$AWS_SECRET_KEY \
access_key=$AWS_ACCESS_KEY
# Create a role that EC2 instances can use
vault write auth/aws/role/ec2-app-role \
auth_type=iam \
bound_iam_principal_arn="arn:aws:iam::ACCOUNT_ID:role/app-server-role" \
policies=app-policy \
ttl=1h

Azure Authentication

For Azure workloads using Managed Identities:

bash

# Enable Azure auth
vault auth enable azure
# Configure Azure
vault write auth/azure/config \
tenant_id=$AZURE_TENANT_ID \
resource="https://management.azure.com/" \
client_id=$AZURE_CLIENT_ID \
client_secret=$AZURE_CLIENT_SECRET
# Create a role for Azure VMs
vault write auth/azure/role/azure-app-role \
policies=app-policy \
bound_subscription_ids=$AZURE_SUBSCRIPTION_ID \
bound_resource_groups=production-rg \
ttl=1h

GCP Authentication

For GCP workloads using service accounts:

bash

# Enable GCP auth
vault auth enable gcp
# Configure GCP
vault write auth/gcp/config \
credentials=@gcp-credentials.json
# Create a role for GCE instances
vault write auth/gcp/role/gce-app-role \
type="gce" \
policies=app-policy \
bound_projects="my-project-id" \
bound_zones="us-central1-a,us-central1-b" \
ttl=1h

Step 3: Set Up Dynamic Secrets

Here’s where the magic happens. Instead of storing static database passwords, Vault can generate unique credentials on demand and revoke them automatically when they expire.

Dynamic AWS Credentials

bash

# Enable AWS secrets engine
vault secrets enable aws
# Configure root credentials (Vault uses these to create dynamic creds)
vault write aws/config/root \
access_key=$AWS_ACCESS_KEY \
secret_key=$AWS_SECRET_KEY \
region=us-east-1
# Create a role that generates S3 read-only credentials
vault write aws/roles/s3-reader \
credential_type=iam_user \
policy_document=-<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
]
}
EOF
# Now any authenticated client can get temporary AWS credentials
vault read aws/creds/s3-reader
# Returns:
# access_key AKIA...
# secret_key xyz123...
# lease_duration 1h
# These credentials will be automatically revoked after 1 hour

Dynamic Database Credentials

This is probably my favorite feature. Every time an application needs to connect to a database, it gets a unique username and password that only it knows.

bash

# Enable database secrets engine
vault secrets enable database
# Configure PostgreSQL connection
vault write database/config/production-postgres \
plugin_name=postgresql-database-plugin \
allowed_roles="app-readonly,app-readwrite" \
connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/appdb?sslmode=require" \
username="vault_admin" \
password="vault_admin_password"
# Create a read-only role
vault write database/roles/app-readonly \
db_name=production-postgres \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Create a read-write role
vault write database/roles/app-readwrite \
db_name=production-postgres \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"

Now when your application requests credentials:

bash

vault read database/creds/app-readonly
# Returns:
# username v-kubernetes-app-readonly-abc123
# password A1B2C3D4E5F6...
# lease_duration 1h

Every request gets a different username and password. If credentials are compromised, they expire automatically. And you have a complete audit trail of who accessed what, when.

Dynamic Azure Credentials

bash

# Enable Azure secrets engine
vault secrets enable azure
# Configure Azure
vault write azure/config \
subscription_id=$AZURE_SUBSCRIPTION_ID \
tenant_id=$AZURE_TENANT_ID \
client_id=$AZURE_CLIENT_ID \
client_secret=$AZURE_CLIENT_SECRET
# Create a role that generates Azure Service Principals
vault write azure/roles/contributor \
ttl=1h \
azure_roles=-<<EOF
[
{
"role_name": "Contributor",
"scope": "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/production-rg"
}
]
EOF

Step 4: Application Integration

Let’s see how applications actually use Vault. I’ll show you several patterns.

Pattern 1: Vault Agent Sidecar (Kubernetes)

This is my recommended approach for Kubernetes. Vault Agent runs alongside your application and handles authentication and secret retrieval automatically.

yaml

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
template:
metadata:
annotations:
# These annotations tell Vault Agent what to do
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "my-app-role"
vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-readonly"
vault.hashicorp.com/agent-inject-template-db-creds: |
{{- with secret "database/creds/app-readonly" -}}
export DB_USERNAME="{{ .Data.username }}"
export DB_PASSWORD="{{ .Data.password }}"
{{- end }}
spec:
serviceAccountName: my-app
containers:
- name: my-app
image: my-app:latest
command: ["/bin/sh", "-c"]
args:
- source /vault/secrets/db-creds && ./start-app.sh

When this pod starts, Vault Agent automatically:

  1. Authenticates to Vault using the Kubernetes service account
  2. Retrieves database credentials
  3. Writes them to /vault/secrets/db-creds
  4. Renews the credentials before they expire
  5. Updates the file when credentials change

Your application just reads from a file. It doesn’t need to know anything about Vault.

Pattern 2: Direct SDK Integration

For applications that need more control, you can use the Vault SDK directly:

python

# Python example
import hvac
import os
def get_vault_client():
"""Create Vault client using Kubernetes auth."""
client = hvac.Client(url=os.environ['VAULT_ADDR'])
# Read the service account token
with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
jwt = f.read()
# Authenticate to Vault
client.auth.kubernetes.login(
role='my-app-role',
jwt=jwt,
mount_point='kubernetes'
)
return client
def get_database_credentials():
"""Get dynamic database credentials."""
client = get_vault_client()
# Request new database credentials
response = client.secrets.database.generate_credentials(
name='app-readonly',
mount_point='database'
)
return {
'username': response['data']['username'],
'password': response['data']['password'],
'lease_id': response['lease_id'],
'lease_duration': response['lease_duration']
}
def connect_to_database():
"""Connect to database with dynamic credentials."""
creds = get_database_credentials()
connection = psycopg2.connect(
host='db.example.com',
database='appdb',
user=creds['username'],
password=creds['password']
)
return connection

Pattern 3: External Secrets Operator

If you prefer Kubernetes-native secrets, use External Secrets Operator to sync Vault secrets to Kubernetes:

yaml

# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secrets
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: vault-backend
target:
name: app-secrets
creationPolicy: Owner
data:
- secretKey: api-key
remoteRef:
key: secret/data/app/api-key
property: value
- secretKey: db-password
remoteRef:
key: secret/data/app/database
property: password

Step 5: Policies and Access Control

Vault policies determine who can access what. Be specific and follow the principle of least privilege.

hcl

# app-policy.hcl
# Allow reading dynamic database credentials
path "database/creds/app-readonly" {
capabilities = ["read"]
}
# Allow reading application secrets
path "secret/data/app/*" {
capabilities = ["read", "list"]
}
# Deny access to admin paths
path "sys/*" {
capabilities = ["deny"]
}
# Allow the app to renew its own token
path "auth/token/renew-self" {
capabilities = ["update"]
}

Apply the policy:

bash

vault policy write app-policy app-policy.hcl
# Create a Kubernetes auth role that uses this policy
vault write auth/kubernetes/role/my-app-role \
bound_service_account_names=my-app \
bound_service_account_namespaces=production \
policies=app-policy \
ttl=1h

Step 6: Monitoring and Audit

You need visibility into who’s accessing secrets. Enable audit logging:

bash

# Enable file audit device
vault audit enable file file_path=/vault/audit/vault-audit.log
# Enable syslog for centralized logging
vault audit enable syslog tag="vault" facility="AUTH"

For monitoring, Vault exposes Prometheus metrics:

yaml

# ServiceMonitor for Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: vault
namespace: vault
spec:
selector:
matchLabels:
app.kubernetes.io/name: vault
endpoints:
- port: http
path: /v1/sys/metrics
params:
format: ["prometheus"]
scheme: https
tlsConfig:
insecureSkipVerify: true

Key metrics to alert on:

yaml

# Prometheus alerting rules
groups:
- name: vault
rules:
- alert: VaultSealed
expr: vault_core_unsealed == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Vault is sealed"
description: "Vault instance {{ $labels.instance }} is sealed and unable to serve requests"
- alert: VaultTooManyPendingTokens
expr: vault_token_count > 10000
for: 5m
labels:
severity: warning
annotations:
summary: "Too many Vault tokens"
description: "Vault has {{ $value }} active tokens. Consider reducing TTLs."
- alert: VaultLeadershipLost
expr: increase(vault_core_leadership_lost_count[5m]) > 0
labels:
severity: warning
annotations:
summary: "Vault leadership changes detected"

Common Mistakes to Avoid

Let me save you some headaches by sharing mistakes I’ve seen (and made):

Mistake 1: Using the root token for applications

The root token has unlimited access. Create specific policies and tokens for each application.

Mistake 2: Not rotating the root token

After initial setup, generate a new root token and revoke the original:

bash

vault operator generate-root -init
# Follow the process to generate a new root token
vault token revoke <old-root-token>

Mistake 3: Setting TTLs too long

Short TTLs mean compromised credentials are valid for less time. Start with 1 hour and adjust based on your needs.

Mistake 4: Not testing recovery procedures

Practice unsealing Vault. Practice recovering from backup. Do it regularly. The worst time to learn is during an actual incident.

Mistake 5: Storing unseal keys together

Distribute unseal keys to different people in different locations. Use a threshold scheme (3 of 5) so no single person can unseal Vault.

Regards, Enjoy the Cloud
Osama

Building a Multi-Cloud Architecture with OCI and AWS: A Real-World Integration Guide

I’ll tell you something that might sound controversial in cloud circles: the best cloud is often more than one cloud.

I’ve worked with dozens of enterprises over the years, and here’s what I’ve noticed. Some started with AWS years ago and built their entire infrastructure there. Then they realized Oracle Autonomous Database or Exadata could dramatically improve their database performance. Others were Oracle shops that wanted to leverage AWS’s machine learning services or global edge network.

The question isn’t really “which cloud is better?” The question is “how do we get the best of both?”

In this article, I’ll walk you through building a practical multi-cloud architecture connecting OCI and AWS. We’ll cover secure networking, data synchronization, identity federation, and the operational realities of running workloads across both platforms.

Why Multi-Cloud Actually Makes Sense

Let me be clear about something. Multi-cloud for its own sake is a terrible idea. It adds complexity, increases operational burden, and creates more things that can break. But multi-cloud for the right reasons? That’s a different story.

Here are legitimate reasons I’ve seen organizations adopt OCI and AWS together:

Database Performance: Oracle Autonomous Database and Exadata Cloud Service are genuinely difficult to match for Oracle workloads. If you’re running complex OLTP or analytics on Oracle, OCI’s database offerings are purpose-built for that.

AWS Ecosystem: AWS has services that simply don’t exist elsewhere. SageMaker for ML, Lambda’s maturity, CloudFront’s global presence, or specialized services like Rekognition and Comprehend.

Vendor Negotiation: Having workloads on multiple clouds gives you negotiating leverage. I’ve seen organizations save millions in licensing by demonstrating they could move workloads.

Acquisition and Mergers: Company A runs on AWS, Company B runs on OCI. Now they’re one company. Multi-cloud by necessity.

Regulatory Requirements: Some industries require data sovereignty or specific compliance certifications that might be easier to achieve with a particular provider in a particular region.

If none of these apply to you, stick with one cloud. Seriously. But if they do, keep reading.

Architecture Overview

Let’s design a realistic scenario. We have an e-commerce company with:

  • Application tier running on AWS (EKS, Lambda, API Gateway)
  • Core transactional database on OCI (Autonomous Transaction Processing)
  • Data warehouse on OCI (Autonomous Data Warehouse)
  • Machine learning workloads on AWS (SageMaker)
  • Shared data that needs to flow between both clouds


Setting Up Cross-Cloud Networking

The foundation of any multi-cloud architecture is networking. You need a secure, reliable, and performant connection between clouds.

Option 1: IPSec VPN (Good for Starting Out)

IPSec VPN is the quickest way to connect AWS and OCI. It runs over the public internet but encrypts everything. Good for development, testing, or low-bandwidth production workloads.

On OCI Side:

First, create a Dynamic Routing Gateway (DRG) and attach it to your VCN:

bash

# Create DRG
oci network drg create \
--compartment-id $COMPARTMENT_ID \
--display-name "aws-interconnect-drg"
# Attach DRG to VCN
oci network drg-attachment create \
--drg-id $DRG_ID \
--vcn-id $VCN_ID \
--display-name "vcn-attachment"

Create a Customer Premises Equipment (CPE) object representing AWS:

bash

# Create CPE for AWS VPN endpoint
oci network cpe create \
--compartment-id $COMPARTMENT_ID \
--ip-address $AWS_VPN_PUBLIC_IP \
--display-name "aws-vpn-endpoint"

Create the IPSec connection:

bash

# Create IPSec connection
oci network ip-sec-connection create \
--compartment-id $COMPARTMENT_ID \
--cpe-id $CPE_ID \
--drg-id $DRG_ID \
--static-routes '["10.1.0.0/16"]' \
--display-name "oci-to-aws-vpn"

On AWS Side:

Create a Customer Gateway pointing to OCI:

bash

# Create Customer Gateway
aws ec2 create-customer-gateway \
--type ipsec.1 \
--public-ip $OCI_VPN_PUBLIC_IP \
--bgp-asn 65000
# Create VPN Gateway
aws ec2 create-vpn-gateway \
--type ipsec.1
# Attach to VPC
aws ec2 attach-vpn-gateway \
--vpn-gateway-id $VGW_ID \
--vpc-id $VPC_ID
# Create VPN Connection
aws ec2 create-vpn-connection \
--type ipsec.1 \
--customer-gateway-id $CGW_ID \
--vpn-gateway-id $VGW_ID \
--options '{"StaticRoutesOnly": true}'

Update route tables on both sides:

bash

# AWS: Add route to OCI CIDR
aws ec2 create-route \
--route-table-id $ROUTE_TABLE_ID \
--destination-cidr-block 10.2.0.0/16 \
--gateway-id $VGW_ID
# OCI: Add route to AWS CIDR
oci network route-table update \
--rt-id $ROUTE_TABLE_ID \
--route-rules '[{
"destination": "10.1.0.0/16",
"destinationType": "CIDR_BLOCK",
"networkEntityId": "'$DRG_ID'"
}]'

Option 2: Private Connectivity (Production Recommended)

For production workloads, you want dedicated private connectivity. This means OCI FastConnect paired with AWS Direct Connect, meeting at a common colocation facility.

The good news is that Oracle and AWS both have presence in major colocation providers like Equinix. The setup involves:

  1. Establishing FastConnect to your colocation
  2. Establishing Direct Connect to the same colocation
  3. Connecting them via a cross-connect in the facility

hcl

# Terraform for FastConnect virtual circuit
resource "oci_core_virtual_circuit" "aws_interconnect" {
compartment_id = var.compartment_id
display_name = "aws-fastconnect"
type = "PRIVATE"
bandwidth_shape_name = "1 Gbps"
cross_connect_mappings {
customer_bgp_peering_ip = "169.254.100.1/30"
oracle_bgp_peering_ip = "169.254.100.2/30"
}
customer_asn = "65001"
gateway_id = oci_core_drg.main.id
provider_name = "Equinix"
region = "Dubai"
}

hcl

# Terraform for AWS Direct Connect
resource "aws_dx_connection" "oci_interconnect" {
name = "oci-direct-connect"
bandwidth = "1Gbps"
location = "Equinix DX1"
provider_name = "Equinix"
}
resource "aws_dx_private_virtual_interface" "oci" {
connection_id = aws_dx_connection.oci_interconnect.id
name = "oci-vif"
vlan = 4094
address_family = "ipv4"
bgp_asn = 65002
amazon_address = "169.254.100.5/30"
customer_address = "169.254.100.6/30"
dx_gateway_id = aws_dx_gateway.main.id
}

Honestly, setting this up involves coordination with both cloud providers and the colocation facility. Budget 4-8 weeks for the physical connectivity and plan for redundancy from day one.

Database Connectivity from AWS to OCI

Now that we have network connectivity, let’s connect AWS applications to OCI databases.

Configuring Autonomous Database for External Access

First, enable private endpoint access for your Autonomous Database:

bash

# Update ADB to use private endpoint
oci db autonomous-database update \
--autonomous-database-id $ADB_ID \
--is-access-control-enabled true \
--whitelisted-ips '["10.1.0.0/16"]' \ # AWS VPC CIDR
--is-mtls-connection-required false # Allow TLS without mTLS for simplicity

Get the connection string:

bash

oci db autonomous-database get \
--autonomous-database-id $ADB_ID \
--query 'data."connection-strings".profiles[?consumer=="LOW"].value | [0]'

Application Configuration on AWS

Here’s a practical Python example for connecting from AWS Lambda to OCI Autonomous Database:

python

# lambda_function.py
import cx_Oracle
import os
import boto3
from botocore.exceptions import ClientError
def get_db_credentials():
"""Retrieve database credentials from AWS Secrets Manager"""
secret_name = "oci-adb-credentials"
region_name = "us-east-1"
session = boto3.session.Session()
client = session.client(
service_name='secretsmanager',
region_name=region_name
)
try:
response = client.get_secret_value(SecretId=secret_name)
return json.loads(response['SecretString'])
except ClientError as e:
raise e
def handler(event, context):
# Get credentials
creds = get_db_credentials()
# Connection string format for Autonomous DB
dsn = """(description=
(retry_count=20)(retry_delay=3)
(address=(protocol=tcps)(port=1522)
(host=adb.me-dubai-1.oraclecloud.com))
(connect_data=(service_name=xxx_atp_low.adb.oraclecloud.com))
(security=(ssl_server_dn_match=yes)))"""
connection = cx_Oracle.connect(
user=creds['username'],
password=creds['password'],
dsn=dsn,
encoding="UTF-8"
)
cursor = connection.cursor()
cursor.execute("SELECT * FROM orders WHERE order_date = TRUNC(SYSDATE)")
results = []
for row in cursor:
results.append({
'order_id': row[0],
'customer_id': row[1],
'amount': float(row[2])
})
cursor.close()
connection.close()
return {
'statusCode': 200,
'body': json.dumps(results)
}

For containerized applications on EKS, use a connection pool:

python

# db_pool.py
import cx_Oracle
import os
class OCIDatabasePool:
_pool = None
@classmethod
def get_pool(cls):
if cls._pool is None:
cls._pool = cx_Oracle.SessionPool(
user=os.environ['OCI_DB_USER'],
password=os.environ['OCI_DB_PASSWORD'],
dsn=os.environ['OCI_DB_DSN'],
min=2,
max=10,
increment=1,
encoding="UTF-8",
threaded=True,
getmode=cx_Oracle.SPOOL_ATTRVAL_WAIT
)
return cls._pool
@classmethod
def get_connection(cls):
return cls.get_pool().acquire()
@classmethod
def release_connection(cls, connection):
cls.get_pool().release(connection)

Kubernetes deployment for the application:

yaml

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-service
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: order-service
template:
metadata:
labels:
app: order-service
spec:
containers:
- name: order-service
image: 123456789.dkr.ecr.us-east-1.amazonaws.com/order-service:v1.0
ports:
- containerPort: 8080
env:
- name: OCI_DB_USER
valueFrom:
secretKeyRef:
name: oci-db-credentials
key: username
- name: OCI_DB_PASSWORD
valueFrom:
secretKeyRef:
name: oci-db-credentials
key: password
- name: OCI_DB_DSN
valueFrom:
configMapKeyRef:
name: oci-db-config
key: dsn
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10

Data Synchronization Between Clouds

Real multi-cloud architectures need data flowing between clouds. Here are practical patterns:

Pattern 1: Event-Driven Sync with Kafka

Use a managed Kafka service as the bridge:

python

# AWS Lambda producer - sends events to Kafka
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['kafka-broker-1:9092', 'kafka-broker-2:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
security_protocol='SASL_SSL',
sasl_mechanism='PLAIN',
sasl_plain_username=os.environ['KAFKA_USER'],
sasl_plain_password=os.environ['KAFKA_PASSWORD']
)
def handler(event, context):
# Process order and send to Kafka for OCI consumption
order_data = process_order(event)
producer.send(
'orders-topic',
key=str(order_data['order_id']).encode(),
value=order_data
)
producer.flush()
return {'statusCode': 200}

OCI side consumer using OCI Functions:

python

# OCI Function consumer
import io
import json
import logging
import cx_Oracle
from kafka import KafkaConsumer
def handler(ctx, data: io.BytesIO = None):
consumer = KafkaConsumer(
'orders-topic',
bootstrap_servers=['kafka-broker-1:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='oci-order-processor',
value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)
connection = get_adb_connection()
cursor = connection.cursor()
for message in consumer:
order = message.value
cursor.execute("""
MERGE INTO orders o
USING (SELECT :order_id AS order_id FROM dual) src
ON (o.order_id = src.order_id)
WHEN MATCHED THEN
UPDATE SET amount = :amount, status = :status, updated_at = SYSDATE
WHEN NOT MATCHED THEN
INSERT (order_id, customer_id, amount, status, created_at)
VALUES (:order_id, :customer_id, :amount, :status, SYSDATE)
""", order)
connection.commit()
cursor.close()
connection.close()

Pattern 2: Scheduled Batch Sync

For less time-sensitive data, batch synchronization is simpler and more cost-effective:

python

# AWS Step Functions state machine for batch sync
{
"Comment": "Sync data from AWS to OCI",
"StartAt": "ExtractFromAWS",
"States": {
"ExtractFromAWS": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:extract-data",
"Next": "UploadToS3"
},
"UploadToS3": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:upload-to-s3",
"Next": "CopyToOCI"
},
"CopyToOCI": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:copy-to-oci-bucket",
"Next": "LoadToADB"
},
"LoadToADB": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:load-to-adb",
"End": true
}
}
}

The Lambda function to copy data to OCI Object Storage:

python

# copy_to_oci.py
import boto3
import oci
import os
def handler(event, context):
# Get file from S3
s3 = boto3.client('s3')
s3_object = s3.get_object(
Bucket=event['bucket'],
Key=event['key']
)
file_content = s3_object['Body'].read()
# Upload to OCI Object Storage
config = oci.config.from_file()
object_storage = oci.object_storage.ObjectStorageClient(config)
namespace = object_storage.get_namespace().data
object_storage.put_object(
namespace_name=namespace,
bucket_name="data-sync-bucket",
object_name=event['key'],
put_object_body=file_content
)
return {
'oci_bucket': 'data-sync-bucket',
'object_name': event['key']
}

Load into Autonomous Database using DBMS_CLOUD:

sql

-- Create credential for OCI Object Storage access
BEGIN
DBMS_CLOUD.CREATE_CREDENTIAL(
credential_name => 'OCI_CRED',
username => 'your_oci_username',
password => 'your_auth_token'
);
END;
/
-- Load data from Object Storage
BEGIN
DBMS_CLOUD.COPY_DATA(
table_name => 'ORDERS_STAGING',
credential_name => 'OCI_CRED',
file_uri_list => 'https://objectstorage.me-dubai-1.oraclecloud.com/n/namespace/b/data-sync-bucket/o/orders_*.csv',
format => JSON_OBJECT(
'type' VALUE 'CSV',
'skipheaders' VALUE '1',
'dateformat' VALUE 'YYYY-MM-DD'
)
);
END;
/
-- Merge staging into production
MERGE INTO orders o
USING orders_staging s
ON (o.order_id = s.order_id)
WHEN MATCHED THEN
UPDATE SET o.amount = s.amount, o.status = s.status
WHEN NOT MATCHED THEN
INSERT (order_id, customer_id, amount, status)
VALUES (s.order_id, s.customer_id, s.amount, s.status);

Identity Federation

Managing identities across clouds is a headache unless you set up proper federation. Here’s how to enable SSO between AWS and OCI using a common identity provider.

Using Azure AD as Common IdP (Yes, a Third Cloud)

This is actually quite common. Many enterprises use Azure AD for identity even if their workloads run elsewhere.

Configure OCI to Trust Azure AD:

bash

# Create Identity Provider in OCI
oci iam identity-provider create-saml2-identity-provider \
--compartment-id $TENANCY_ID \
--name "AzureAD-Federation" \
--description "Federation with Azure AD" \
--product-type "IDCS" \
--metadata-url "https://login.microsoftonline.com/$TENANT_ID/federationmetadata/2007-06/federationmetadata.xml"

Configure AWS to Trust Azure AD:

bash

# Create SAML provider in AWS
aws iam create-saml-provider \
--saml-metadata-document file://azure-ad-metadata.xml \
--name AzureAD-Federation
# Create role for federated users
aws iam create-role \
--role-name AzureAD-Admins \
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::123456789:saml-provider/AzureAD-Federation"},
"Action": "sts:AssumeRoleWithSAML",
"Condition": {
"StringEquals": {
"SAML:aud": "https://signin.aws.amazon.com/saml"
}
}
}]
}'

Now your team can use the same Azure AD credentials to access both clouds.

Monitoring Across Clouds

You need unified observability. Here’s a practical approach using Grafana as the common dashboard:

yaml

# docker-compose.yml for centralized Grafana
version: '3.8'
services:
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
- ./provisioning:/etc/grafana/provisioning
environment:
- GF_SECURITY_ADMIN_PASSWORD=secure_password
- GF_INSTALL_PLUGINS=oci-metrics-datasource
volumes:
grafana-data:

Configure data sources:

yaml

# provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
- name: AWS-CloudWatch
type: cloudwatch
access: proxy
jsonData:
authType: keys
defaultRegion: us-east-1
secureJsonData:
accessKey: ${AWS_ACCESS_KEY}
secretKey: ${AWS_SECRET_KEY}
- name: OCI-Monitoring
type: oci-metrics-datasource
access: proxy
jsonData:
tenancyOCID: ${OCI_TENANCY_OCID}
userOCID: ${OCI_USER_OCID}
region: me-dubai-1
secureJsonData:
privateKey: ${OCI_PRIVATE_KEY}

Create a unified dashboard that shows both clouds:

json

{
"title": "Multi-Cloud Overview",
"panels": [
{
"title": "AWS EKS CPU Utilization",
"datasource": "AWS-CloudWatch",
"targets": [{
"namespace": "AWS/EKS",
"metricName": "node_cpu_utilization",
"dimensions": {"ClusterName": "production"}
}]
},
{
"title": "OCI Autonomous DB Sessions",
"datasource": "OCI-Monitoring",
"targets": [{
"namespace": "oci_autonomous_database",
"metric": "CurrentOpenSessionCount",
"resourceGroup": "production-adb"
}]
},
{
"title": "Cross-Cloud Latency",
"datasource": "Prometheus",
"targets": [{
"expr": "histogram_quantile(0.95, rate(cross_cloud_request_duration_seconds_bucket[5m]))"
}]
}
]
}

Cost Management

Multi-cloud cost visibility is challenging. Here’s a practical approach:

python

# cost_aggregator.py
import boto3
import oci
from datetime import datetime, timedelta
def get_aws_costs(start_date, end_date):
client = boto3.client('ce')
response = client.get_cost_and_usage(
TimePeriod={
'Start': start_date.strftime('%Y-%m-%d'),
'End': end_date.strftime('%Y-%m-%d')
},
Granularity='DAILY',
Metrics=['UnblendedCost'],
GroupBy=[{'Type': 'DIMENSION', 'Key': 'SERVICE'}]
)
return response['ResultsByTime']
def get_oci_costs(start_date, end_date):
config = oci.config.from_file()
usage_api = oci.usage_api.UsageapiClient(config)
response = usage_api.request_summarized_usages(
request_summarized_usages_details=oci.usage_api.models.RequestSummarizedUsagesDetails(
tenant_id=config['tenancy'],
time_usage_started=start_date,
time_usage_ended=end_date,
granularity="DAILY",
group_by=["service"]
)
)
return response.data.items
def generate_report():
end_date = datetime.now()
start_date = end_date - timedelta(days=30)
aws_costs = get_aws_costs(start_date, end_date)
oci_costs = get_oci_costs(start_date, end_date)
total_aws = sum(float(day['Total']['UnblendedCost']['Amount']) for day in aws_costs)
total_oci = sum(item.computed_amount for item in oci_costs)
print(f"30-Day Multi-Cloud Cost Summary")
print(f"{'='*40}")
print(f"AWS Total: ${total_aws:,.2f}")
print(f"OCI Total: ${total_oci:,.2f}")
print(f"Combined Total: ${total_aws + total_oci:,.2f}")

Lessons Learned

After running multi-cloud architectures for several years, here’s what I’ve learned:

Network is everything. Invest in proper connectivity upfront. The $500/month you save on VPN versus dedicated connectivity will cost you thousands in debugging performance issues.

Pick one cloud for each workload type. Don’t run the same thing in both clouds. Use OCI for Oracle databases, AWS for its unique services. Avoid the temptation to replicate everything everywhere.

Standardize your tooling. Terraform works on both clouds. Use it. Same for monitoring, logging, and CI/CD. The more consistent your tooling, the less your team has to context-switch.

Document your data flows. Know exactly what data goes where and why. This will save you during security audits and incident response.

Test cross-cloud failures. What happens when the VPN goes down? Can your application degrade gracefully? Find out before your customers do.

Conclusion

Multi-cloud between OCI and AWS isn’t simple, but it’s absolutely achievable. The key is having clear reasons for using each cloud, solid networking fundamentals, and consistent operational practices.

Start small. Connect one application to one database across clouds. Get that working reliably before expanding. Build your team’s confidence and expertise incrementally.

The organizations that succeed with multi-cloud are the ones that treat it as an architectural choice, not a checkbox. They know exactly why they need both clouds and have designed their systems accordingly.

Regards,
Osama

Deep Dive into Oracle Kubernetes Engine Security and Networking in Production

Oracle Kubernetes Engine is often introduced as a managed Kubernetes service, but its real strength only becomes clear when you operate it in production. OKE tightly integrates with OCI networking, identity, and security services, which gives you a very different operational model compared to other managed Kubernetes platforms.

This article walks through OKE from a production perspective, focusing on security boundaries, networking design, ingress exposure, private access, and mutual TLS. The goal is not to explain Kubernetes basics, but to explain how OKE behaves when you run regulated, enterprise workloads.

Understanding the OKE Networking Model

OKE does not abstract networking away from you. Every cluster is deeply tied to OCI VCN constructs.

Core Components

An OKE cluster consists of:

  • A managed Kubernetes control plane
  • Worker nodes running in OCI subnets
  • OCI networking primitives controlling traffic flow

Key OCI resources involved:

  • Virtual Cloud Network
  • Subnets for control plane and workers
  • Network Security Groups
  • Route tables
  • OCI Load Balancers

Unlike some platforms, security in OKE is enforced at multiple layers simultaneously.

Worker Node and Pod Networking

OKE uses OCI VCN-native networking. Pods receive IPs from the subnet CIDR through the OCI CNI plugin.

What this means in practice

  • Pods are first-class citizens on the VCN
  • Pod IPs are routable within the VCN
  • Network policies and OCI NSGs both apply

Example subnet design:

VCN: 10.0.0.0/16

Worker Subnet: 10.0.10.0/24
Load Balancer Subnet: 10.0.20.0/24
Private Endpoint Subnet: 10.0.30.0/24

This design allows you to:

  • Keep workers private
  • Expose only ingress through OCI Load Balancer
  • Control east-west traffic using Kubernetes NetworkPolicies and OCI NSGs together

Security Boundaries in OKE

Security in OKE is layered by design.

Layer 1: OCI IAM and Compartments

OKE clusters live inside OCI compartments. IAM policies control:

  • Who can create or modify clusters
  • Who can access worker nodes
  • Who can manage load balancers and subnets

Example IAM policy snippet:

Allow group OKE-Admins to manage cluster-family in compartment OKE-PROD
Allow group OKE-Admins to manage virtual-network-family in compartment OKE-PROD

This separation is critical for regulated environments.

Layer 2: Network Security Groups

Network Security Groups act as virtual firewalls at the VNIC level.

Typical NSG rules:

  • Allow node-to-node communication
  • Allow ingress from load balancer subnet only
  • Block all public inbound traffic

Example inbound NSG rule:

Source: 10.0.20.0/24
Protocol: TCP
Port: 443

This ensures only the OCI Load Balancer can reach your ingress controller.

Layer 3: Kubernetes Network Policies

NetworkPolicies control pod-level traffic.

Example policy allowing traffic only from ingress namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress
  namespace: app-prod
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: ingress

This blocks all lateral movement by default.

Ingress Design in OKE

OKE integrates natively with OCI Load Balancer.

Public vs Private Ingress

You can deploy ingress in two modes:

  • Public Load Balancer
  • Internal Load Balancer

For production workloads, private ingress is strongly recommended.

Example service annotation for private ingress:

service.beta.kubernetes.io/oci-load-balancer-internal: "true"
service.beta.kubernetes.io/oci-load-balancer-subnet1: ocid1.subnet.oc1..

This ensures the load balancer has no public IP.

Private Access to the Cluster Control Plane

OKE supports private API endpoints.

When enabled:

  • The Kubernetes API is accessible only from the VCN
  • No public endpoint exists

This is critical for Zero Trust environments.

Operational impact:

  • kubectl access requires VPN, Bastion, or OCI Cloud Shell inside the VCN
  • CI/CD runners must have private connectivity

This dramatically reduces the attack surface.

Mutual TLS Inside OKE

TLS termination at ingress is not enough for sensitive workloads. Many enterprises require mTLS between services.

Typical mTLS Architecture

  • TLS termination at ingress
  • Internal mTLS between services
  • Certificate management via Vault or cert-manager

Example cert-manager issuer using OCI Vault:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: oci-vault-issuer
spec:
  vault:
    server: https://vault.oci.oraclecloud.com
    path: pki/sign/oke

Each service receives:

  • Its own certificate
  • Short-lived credentials
  • Automatic rotation

Traffic Flow Example

End-to-end request path:

  1. Client connects to OCI Load Balancer
  2. Load Balancer forwards traffic to NGINX Ingress
  3. Ingress enforces TLS and headers
  4. Service-to-service traffic uses mTLS
  5. NetworkPolicy restricts lateral movement
  6. NSGs enforce VCN-level boundaries

Every hop is authenticated and encrypted.


Observability and Security Visibility

OKE integrates with:

  • OCI Logging
  • OCI Flow Logs
  • Kubernetes audit logs

This allows:

  • Tracking ingress traffic
  • Detecting unauthorized access attempts
  • Correlating pod-level events with network flows

Regards
Osama