Cross-Cloud Secret Synchronization: AWS Secrets Manager and OCI Vault in a Production Multi-Cloud Setup

One of the most overlooked problems in multi-cloud environments is secrets management across providers. Teams usually solve it badly: they store the same secret in both clouds manually, forget to rotate one of them, and find out during an outage that the credentials have been out of sync for three months.

In this post I will walk through building an automated secrets synchronization pipeline between AWS Secrets Manager and OCI Vault. When a secret rotates in AWS, the pipeline detects the rotation event, retrieves the new value, and pushes it into OCI Vault automatically. Everything is built with Terraform, an AWS Lambda function, and OCI IAM. No manual steps after the initial deployment.

This is a pattern I have used in environments where the database layer runs on OCI (leveraging Oracle Database pricing and performance) while the application layer runs on AWS. Both sides need the same database credentials, and both sides need to stay in sync without human intervention.

Architecture

The flow works like this:

AWS Secrets Manager rotation event fires via EventBridge, which triggers a Lambda function. The Lambda retrieves the new secret value, authenticates to OCI using an API key stored in its own environment (not hardcoded), and calls the OCI Vault API to update the corresponding secret version. OCI Vault stores the new value and makes it available to workloads running in OCI.

Prerequisites

Before starting you need:

AWS account with permissions to manage Secrets Manager, Lambda, EventBridge, and IAM
OCI tenancy with permissions to manage Vault, Keys, and IAM policies
Terraform 1.5 or later
Python 3.11 for the Lambda function
An existing OCI Vault and master encryption key (or we will create one)

Step 1: OCI Vault and IAM Setup

Start with OCI. We need a Vault, a master key, and an IAM user whose API key the Lambda will use to authenticate.

hcl

			
# OCI Vault
resource "oci_kms_vault" "app_vault" {
  compartment_id = var.compartment_id
  display_name   = "multi-cloud-secrets-vault"
  vault_type     = "DEFAULT"
}
# Master Encryption Key inside the Vault
resource "oci_kms_key" "secrets_key" {
  compartment_id      = var.compartment_id
  display_name        = "secrets-master-key"
  management_endpoint = oci_kms_vault.app_vault.management_endpoint
  key_shape {
    algorithm = "AES"
    length    = 32
  }
}
# IAM user for cross-cloud access
resource "oci_identity_user" "sync_user" {
  compartment_id = var.tenancy_ocid
  name           = "aws-secrets-sync-user"
  description    = "Service user for AWS Lambda to push secrets into OCI Vault"
  email          = "sync-user@internal.example.com"
}
# API key for the sync user (you will generate the actual key pair separately)
resource "oci_identity_api_key" "sync_user_key" {
  user_id   = oci_identity_user.sync_user.id
  key_value = var.oci_sync_user_public_key_pem
}
# IAM group for the sync user
resource "oci_identity_group" "sync_group" {
  compartment_id = var.tenancy_ocid
  name           = "secrets-sync-group"
  description    = "Group for cross-cloud secrets sync service users"
}
resource "oci_identity_user_group_membership" "sync_membership" {
  group_id = oci_identity_group.sync_group.id
  user_id  = oci_identity_user.sync_user.id
}
# Minimal IAM policy - only what is needed, nothing more
resource "oci_identity_policy" "sync_policy" {
  compartment_id = var.compartment_id
  name           = "secrets-sync-policy"
  description    = "Allows sync user to manage secrets in the app vault only"
  statements = [
    "Allow group secrets-sync-group to manage secret-family in compartment id ${var.compartment_id} where target.vault.id = '${oci_kms_vault.app_vault.id}'",
    "Allow group secrets-sync-group to use keys in compartment id ${var.compartment_id} where target.key.id = '${oci_kms_key.secrets_key.id}'"
  ]
}

		

The policy scope is intentionally narrow. The sync user can only manage secrets inside this specific vault and can only use this specific key. If the AWS Lambda credentials are ever compromised, the blast radius is limited to this vault.

Step 2: Create the Initial Secret in OCI Vault

We need a secret placeholder in OCI Vault that the Lambda will update. The initial value does not matter since it will be overwritten on the first sync.

hcl

			
resource "oci_vault_secret" "db_password" {
  compartment_id = var.compartment_id
  vault_id       = oci_kms_vault.app_vault.id
  key_id         = oci_kms_key.secrets_key.id
  secret_name    = "prod-db-password"
  secret_content {
    content_type = "BASE64"
    content      = base64encode("initial-placeholder-value")
    name         = "v1"
    stage        = "CURRENT"
  }
  metadata = {
    source      = "aws-secrets-manager"
    aws_secret  = "prod/database/password"
    environment = "production"
  }
}

		

Step 3: AWS Secrets Manager and the Source Secret

On the AWS side, create the authoritative secret and enable automatic rotation.

hcl

			
resource "aws_secretsmanager_secret" "db_password" {
  name                    = "prod/database/password"
  description             = "Production database password - synced to OCI Vault"
  recovery_window_in_days = 7
  tags = {
    Environment   = "production"
    SyncTarget    = "oci-vault"
    OciSecretName = "prod-db-password"
  }
}
resource "aws_secretsmanager_secret_version" "db_password_v1" {
  secret_id     = aws_secretsmanager_secret.db_password.id
  secret_string = jsonencode({
    username = "db_admin",
    password = var.initial_db_password,
    host     = var.db_host,
    port     = 1521,
    database = "PRODDB"
  })
}
# Rotation configuration - rotate every 30 days
resource "aws_secretsmanager_secret_rotation" "db_password_rotation" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.db_rotation_lambda.arn
  rotation_rules {
    automatically_after_days = 30
  }
}

		

Step 4: Store OCI Credentials in AWS Secrets Manager

The Lambda needs OCI API credentials to authenticate. Store them as a secret in AWS Secrets Manager so they never appear in Lambda environment variables in plaintext.

hcl

			
resource "aws_secretsmanager_secret" "oci_credentials" {
  name        = "internal/oci-sync-credentials"
  description = "OCI API key credentials for secrets sync Lambda"
  tags = {
    Environment = "production"
    Purpose     = "cross-cloud-sync"
  }
}
resource "aws_secretsmanager_secret_version" "oci_credentials_v1" {
  secret_id = aws_secretsmanager_secret.oci_credentials.id
  secret_string = jsonencode({
    tenancy_ocid  = var.oci_tenancy_ocid,
    user_ocid     = var.oci_sync_user_ocid,
    fingerprint   = var.oci_api_key_fingerprint,
    private_key   = var.oci_private_key_pem,
    region        = var.oci_region
  })
}

		

Step 5: The Lambda Function

This is the core of the pipeline. The Lambda retrieves the rotated secret from AWS Secrets Manager, loads OCI credentials from its own secrets store, and calls the OCI Vault API to create a new secret version.

python

			
import boto3
import json
import base64
import oci
import logging
import os
from datetime import datetime, timezone
from botocore.exceptions import ClientError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def get_oci_config():
    """Retrieve OCI credentials from AWS Secrets Manager."""
    client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
    
    try:
        response = client.get_secret_value(
            SecretId=os.environ["OCI_CREDENTIALS_SECRET_ARN"]
        )
        creds = json.loads(response["SecretString"])
        
        return {
            "tenancy": creds["tenancy_ocid"],
            "user": creds["user_ocid"],
            "fingerprint": creds["fingerprint"],
            "key_content": creds["private_key"],
            "region": creds["region"]
        }
    except ClientError as e:
        logger.error(f"Failed to retrieve OCI credentials: {e}")
        raise
def get_aws_secret(secret_arn: str) -> str:
    """Retrieve the current value of an AWS secret."""
    client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
    
    try:
        response = client.get_secret_value(SecretId=secret_arn)
        return response.get("SecretString") or base64.b64decode(
            response["SecretBinary"]
        ).decode("utf-8")
    except ClientError as e:
        logger.error(f"Failed to retrieve AWS secret {secret_arn}: {e}")
        raise
def push_to_oci_vault(
    oci_config: dict,
    vault_id: str,
    key_id: str,
    secret_ocid: str,
    secret_value: str
):
    """Create a new version of an OCI Vault secret."""
    vaults_client = oci.vault.VaultsClient(oci_config)
    
    encoded_value = base64.b64encode(secret_value.encode("utf-8")).decode("utf-8")
    
    update_details = oci.vault.models.UpdateSecretDetails(
        secret_content=oci.vault.models.Base64SecretContentDetails(
            content_type=oci.vault.models.SecretContentDetails.CONTENT_TYPE_BASE64,
            content=encoded_value,
            name=f"sync-{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}",
            stage="CURRENT"
        ),
        metadata={
            "synced_from": "aws-secrets-manager",
            "synced_at": datetime.now(timezone.utc).isoformat()
        }
    )
    
    response = vaults_client.update_secret(
        secret_id=secret_ocid,
        update_secret_details=update_details
    )
    
    logger.info(
        f"OCI secret updated. OCID: {secret_ocid}, "
        f"New version: {response.data.current_version_number}"
    )
    
    return response.data
def handler(event, context):
    """
    EventBridge trigger handler.
    Expects event detail to contain:
      - aws_secret_arn: ARN of the rotated AWS secret
      - oci_secret_ocid: OCID of the target OCI Vault secret
      - oci_vault_id: OCID of the target OCI Vault
      - oci_key_id: OCID of the OCI KMS key
    """
    logger.info(f"Received event: {json.dumps(event)}")
    
    detail = event.get("detail", {})
    aws_secret_arn  = detail.get("aws_secret_arn")
    oci_secret_ocid = detail.get("oci_secret_ocid")
    oci_vault_id    = detail.get("oci_vault_id")
    oci_key_id      = detail.get("oci_key_id")
    
    if not all([aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id]):
        logger.error("Missing required fields in event detail")
        raise ValueError("Event detail must include aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id")
    
    logger.info(f"Syncing secret: {aws_secret_arn} to OCI: {oci_secret_ocid}")
    
    # Step 1: Get OCI credentials
    oci_config = get_oci_config()
    
    # Step 2: Retrieve the rotated AWS secret
    secret_value = get_aws_secret(aws_secret_arn)
    
    # Step 3: Push to OCI Vault
    result = push_to_oci_vault(
        oci_config=oci_config,
        vault_id=oci_vault_id,
        key_id=oci_key_id,
        secret_ocid=oci_secret_ocid,
        secret_value=secret_value
    )
    
    return {
        "statusCode": 200,
        "body": {
            "message": "Secret synced successfully",
            "oci_secret_ocid": oci_secret_ocid,
            "oci_version": result.current_version_number
        }
    }

		

Step 6: Lambda IAM Role and Deployment

hcl

			
data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}
data "aws_iam_policy_document" "lambda_permissions" {
  statement {
    effect = "Allow"
    actions = [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret"
    ]
    resources = [
      aws_secretsmanager_secret.db_password.arn,
      aws_secretsmanager_secret.oci_credentials.arn
    ]
  }
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["arn:aws:logs:*:*:*"]
  }
}
resource "aws_iam_role" "sync_lambda_role" {
  name               = "secrets-sync-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}
resource "aws_iam_role_policy" "sync_lambda_policy" {
  name   = "secrets-sync-lambda-policy"
  role   = aws_iam_role.sync_lambda_role.id
  policy = data.aws_iam_policy_document.lambda_permissions.json
}
resource "aws_lambda_function" "secrets_sync" {
  filename         = "${path.module}/lambda/secrets_sync.zip"
  function_name    = "oci-secrets-sync"
  role             = aws_iam_role.sync_lambda_role.arn
  handler          = "main.handler"
  runtime          = "python3.11"
  timeout          = 60
  memory_size      = 256
  source_code_hash = filebase64sha256("${path.module}/lambda/secrets_sync.zip")
  environment {
    variables = {
      OCI_CREDENTIALS_SECRET_ARN = aws_secretsmanager_secret.oci_credentials.arn
      AWS_REGION                 = var.aws_region
    }
  }
  layers = [aws_lambda_layer_version.oci_sdk_layer.arn]
}

		

Bundle the OCI Python SDK as a Lambda Layer so the function does not need to package it inline:

bash

			
mkdir -p lambda_layer/python
pip install oci --target lambda_layer/python
cd lambda_layer && zip -r ../oci_sdk_layer.zip python/

hcl

			
resource "aws_lambda_layer_version" "oci_sdk_layer" {
  filename            = "${path.module}/oci_sdk_layer.zip"
  layer_name          = "oci-python-sdk"
  compatible_runtimes = ["python3.11"]
  source_code_hash    = filebase64sha256("${path.module}/oci_sdk_layer.zip")
}

		

Step 7: EventBridge Rule to Trigger on Rotation

hcl

			
resource "aws_cloudwatch_event_rule" "secret_rotation_rule" {
  name        = "detect-secret-rotation"
  description = "Fires when a Secrets Manager secret rotation completes"
  event_pattern = jsonencode({
    source      = ["aws.secretsmanager"],
    detail-type = ["AWS API Call via CloudTrail"],
    detail = {
      eventSource = ["secretsmanager.amazonaws.com"],
      eventName   = ["RotateSecret", "PutSecretValue"]
    }
  })
}
resource "aws_cloudwatch_event_target" "sync_lambda_target" {
  rule      = aws_cloudwatch_event_rule.secret_rotation_rule.name
  target_id = "SyncToOCI"
  arn       = aws_lambda_function.secrets_sync.arn
  input_transformer {
    input_paths = {
      secret_arn = "$.detail.requestParameters.secretId"
    }
    input_template = <<EOF
{
  "detail": {
    "aws_secret_arn": "<secret_arn>",
    "oci_secret_ocid": "${var.oci_db_password_secret_ocid}",
    "oci_vault_id": "${oci_kms_vault.app_vault.id}",
    "oci_key_id": "${oci_kms_key.secrets_key.id}"
  }
}
EOF
  }
}
resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.secrets_sync.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.secret_rotation_rule.arn
}

		

Step 8: Verifying the Pipeline

Manually trigger a rotation to test the full pipeline without waiting 30 days:

bash

			
# Force a rotation in AWS
aws secretsmanager rotate-secret \
  --secret-id prod/database/password \
  --region us-east-1
# Check Lambda execution logs
aws logs tail /aws/lambda/oci-secrets-sync --follow
# Verify the new version appeared in OCI Vault
oci vault secret get \
  --secret-id <your-oci-secret-ocid> \
  --query 'data.{name:secret-name, version:"current-version-number", updated:"time-of-current-version-need-rotation"}' \
  --output table

		

A successful sync produces output similar to this in the Lambda logs:

			
INFO: Syncing secret: arn:aws:secretsmanager:us-east-1:123456789:secret:prod/database/password to OCI: ocid1.vaultsecret.oc1...
INFO: OCI secret updated. OCID: ocid1.vaultsecret.oc1..., New version: 3

Handling Failures and Drift

The pipeline as built is synchronous and event-driven, which means if the Lambda fails, the OCI secret does not get updated. Add a dead-letter queue and a reconciliation function that runs on a schedule to catch any drift.

hcl

			
resource "aws_sqs_queue" "sync_dlq" {
  name                      = "secrets-sync-dlq"
  message_retention_seconds = 86400
}
resource "aws_lambda_function_event_invoke_config" "sync_retry" {
  function_name                = aws_lambda_function.secrets_sync.function_name
  maximum_retry_attempts       = 2
  maximum_event_age_in_seconds = 300
  destination_config {
    on_failure {
      destination = aws_sqs_queue.sync_dlq.arn
    }
  }
}

		

For reconciliation, a scheduled Lambda that runs every hour compares the LastRotatedDate on the AWS secret against the synced_at metadata tag on the OCI secret. If they differ by more than five minutes, it triggers a forced sync.

Security Considerations

A few things to keep in mind when running this in production.

The OCI private key stored in AWS Secrets Manager should be rotated periodically, just like any other credential. Add it to your rotation schedule.

Enable CloudTrail in AWS and OCI Audit logging so every access to both secrets stores is recorded. If something is off with the sync, the audit logs tell you exactly which principal made the change and when.

Use VPC endpoints for Secrets Manager in AWS so the Lambda traffic never crosses the public internet when retrieving credentials.

On the OCI side, enable Vault audit logging to the OCI Logging service so every secret version write is captured.

Wrapping Up

This pipeline solves a real operational problem without requiring a third-party secrets broker. AWS Secrets Manager stays the authoritative source. OCI Vault stays current automatically. The only manual step is the initial deployment.

The pattern extends to other cross-cloud credential types. Database connection strings, API tokens, TLS certificates — any secret that needs to exist on both clouds can follow the same EventBridge to Lambda to OCI Vault flow. Extend the Lambda to support a mapping table of AWS secret ARNs to OCI secret OCIDs and one function handles your entire secrets estate across both providers.

Regards,
Osama

Share this:

Related

Published by Osama Mustafa

Leave a comment Cancel reply