OCI DevOps: Building a Production CI/CD Pipeline with Terraform

Most teams running workloads on OCI manage their deployments through a mix of external tools: GitHub Actions pushing to OKE, Jenkins deploying to compute instances, manual Terraform runs triggered from a developer’s laptop. This works until it does not. The audit trail is scattered, secrets flow through CI runners that may not be in your VCN, and there is no native integration between the deployment tooling and the OCI IAM model that controls the infrastructure.

OCI DevOps is Oracle’s native CI/CD service. It covers source code mirroring, build pipelines, artifact management, and deployment pipelines to OKE, compute instances, Functions, and other targets. Everything runs inside your tenancy, authenticates through IAM Dynamic Groups and policies, and integrates natively with OCI Vault for secrets, OCI Container Registry for images, and OCI Artifact Registry for generic artifacts.

In this post I will build a complete pipeline from source code mirror through build, test, image push, and deployment to an OKE cluster, using Terraform for all infrastructure and a real application for the pipeline to deploy.

Service Architecture

OCI DevOps has five main components that work together.

The Project is the top-level container. It groups all related resources: code repositories, build pipelines, deployment pipelines, and environments.

Code Repositories mirror external Git repositories (GitHub, GitLab, Bitbucket) or host code natively inside OCI. Mirroring syncs on a schedule or on webhook trigger.

Build Pipelines execute build stages: managed build (runs your build spec on Oracle-managed runners), deliver artifact (pushes to Container Registry or Artifact Registry), and trigger deployment.

Artifact Registry stores generic versioned artifacts: Helm charts, Terraform modules, JAR files, and deployment manifests.

Deployment Pipelines run the actual deployment to a target environment. They support blue-green, canary, and rolling deployment strategies with built-in approval gates.

Step 1: IAM Setup

OCI DevOps needs a Dynamic Group that matches the build and deployment pipeline resources, and a policy that grants them the permissions to do their work.

resource "oci_identity_dynamic_group" "devops_build_dg" {
compartment_id = var.tenancy_ocid
name = "devops-build-pipelines"
description = "Dynamic group for OCI DevOps build pipeline runners"
matching_rule = "All {resource.type = 'devopsbuildpipeline', resource.compartment.id = '${var.compartment_id}'}"
}
resource "oci_identity_dynamic_group" "devops_deploy_dg" {
compartment_id = var.tenancy_ocid
name = "devops-deploy-pipelines"
description = "Dynamic group for OCI DevOps deployment pipelines"
matching_rule = "All {resource.type = 'devopsdeploypipeline', resource.compartment.id = '${var.compartment_id}'}"
}
resource "oci_identity_policy" "devops_policy" {
compartment_id = var.compartment_id
name = "devops-pipeline-policy"
description = "Permissions for OCI DevOps build and deploy pipelines"
statements = [
# Build pipelines need to read secrets and push to container registry
"Allow dynamic-group devops-build-pipelines to manage repos in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to read secret-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to manage artifacts in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-build-pipelines to manage devops-family in compartment id ${var.compartment_id}",
# Deploy pipelines need to manage OKE workloads and read artifacts
"Allow dynamic-group devops-deploy-pipelines to manage cluster-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to use artifacts in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to manage devops-family in compartment id ${var.compartment_id}",
"Allow dynamic-group devops-deploy-pipelines to read secret-family in compartment id ${var.compartment_id}"
]
}

Step 2: Create the DevOps Project

resource "oci_devops_project" "orders_api_project" {
compartment_id = var.compartment_id
name = "orders-api"
description = "CI/CD pipeline for the orders API service"
notification_config {
topic_id = oci_ons_notification_topic.devops_alerts.id
}
defined_tags = {
"Operations.Environment" = "production"
"Operations.ManagedBy" = "terraform"
}
}
resource "oci_ons_notification_topic" "devops_alerts" {
compartment_id = var.compartment_id
name = "devops-pipeline-alerts"
description = "Notifications for DevOps pipeline events"
}
resource "oci_ons_subscription" "devops_email" {
compartment_id = var.compartment_id
topic_id = oci_ons_notification_topic.devops_alerts.id
protocol = "EMAIL"
endpoint = var.devops_alert_email
}

Step 3: Mirror the GitHub Repository

OCI DevOps can mirror a GitHub repository and trigger a build pipeline on push events. The mirror keeps a copy of the source inside OCI so builds do not depend on external connectivity to GitHub at build time.

resource "oci_devops_repository" "orders_api_repo" {
project_id = oci_devops_project.orders_api_project.id
name = "orders-api"
description = "Mirror of GitHub orders-api repository"
repository_type = "MIRRORED"
default_branch = "main"
mirror_repository_config {
repository_url = "https://github.com/your-org/orders-api.git"
connector_id = oci_devops_connection.github_connection.id
trigger_schedule {
schedule_type = "CUSTOM"
custom_schedule = "0 */6 * * *"
}
}
}
resource "oci_devops_connection" "github_connection" {
project_id = oci_devops_project.orders_api_project.id
display_name = "github-connection"
connection_type = "GITHUB_ACCESS_TOKEN"
description = "Connection to GitHub using PAT stored in OCI Vault"
access_token = oci_vault_secret.github_pat.id
}
resource "oci_vault_secret" "github_pat" {
compartment_id = var.compartment_id
vault_id = var.vault_id
key_id = var.vault_key_id
secret_name = "github-pat-devops"
secret_content {
content_type = "BASE64"
content = base64encode(var.github_personal_access_token)
}
}

The GitHub PAT is stored in OCI Vault, not in a Terraform variable or environment variable on a CI runner. The build pipeline retrieves it at runtime using the Dynamic Group policy.

Step 4: Build Spec

The build spec is a YAML file committed to your repository at build_spec.yaml. It defines the steps the managed build runner executes.

version: 0.1
component: build
timeoutInSeconds: 1800
env:
exportedVariables:
- BUILDRUN_HASH
steps:
- type: Command
name: Set build hash
command: |
export BUILDRUN_HASH=$(echo ${OCI_BUILD_RUN_ID} | tail -c 8)
echo "BUILDRUN_HASH: ${BUILDRUN_HASH}"
- type: Command
name: Install dependencies
command: |
cd orders-api
pip install -r requirements.txt --quiet
- type: Command
name: Run unit tests
command: |
cd orders-api
python -m pytest tests/unit/ -v --tb=short --junitxml=test-results.xml
if [ $? -ne 0 ]; then
echo "Unit tests failed. Aborting build."
exit 1
fi
- type: Command
name: Run security scan
command: |
pip install bandit --quiet
cd orders-api
bandit -r src/ -f json -o bandit-report.json -ll
if [ $? -eq 1 ]; then
echo "High severity security issues found. Aborting build."
exit 1
fi
- type: Command
name: Build container image
command: |
cd orders-api
IMAGE_TAG="${CONTAINER_REGISTRY}/${NAMESPACE}/orders-api:${BUILDRUN_HASH}"
docker build -t orders-api:latest -t ${IMAGE_TAG} .
echo "IMAGE_TAG=${IMAGE_TAG}" >> ${OCI_PRIMARY_SOURCE_DIR}/build_output.env
- type: Command
name: Push image to OCI Container Registry
command: |
docker push ${IMAGE_TAG}
outputArtifacts:
- name: orders-api-image
type: DOCKER_IMAGE
location: ${IMAGE_TAG}
- name: kubernetes-manifests
type: BINARY
location: ${OCI_PRIMARY_SOURCE_DIR}/orders-api/k8s/

The security scan step uses Bandit to flag high-severity Python security issues and fails the build if any are found. This happens before the image is built, not after.

Step 5: Build Pipeline

resource "oci_devops_build_pipeline" "orders_api_build" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-build"
description = "Build, test, scan, and push the orders API container image"
build_pipeline_parameters {
items {
name = "CONTAINER_REGISTRY"
default_value = "${var.oci_region_key}.ocir.io"
description = "OCI Container Registry endpoint"
}
items {
name = "NAMESPACE"
default_value = var.tenancy_namespace
description = "OCI tenancy namespace for Container Registry"
}
}
}
# Stage 1: Managed Build
resource "oci_devops_build_pipeline_stage" "managed_build" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "managed-build"
description = "Execute build spec on managed runner"
build_pipeline_stage_type = "BUILD"
build_spec_file = "build_spec.yaml"
stage_execution_timeout_in_seconds = 1800
image = "OL7_X86_64_STANDARD_10"
build_source_collection {
items {
connection_type = "DEVOPS_CODE_REPOSITORY"
repository_id = oci_devops_repository.orders_api_repo.id
name = "orders-api"
branch = "main"
repository_url = oci_devops_repository.orders_api_repo.http_url
}
}
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline.orders_api_build.id
}
}
}
# Stage 2: Deliver Artifact to Container Registry
resource "oci_devops_build_pipeline_stage" "deliver_artifact" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "deliver-artifact"
description = "Push built image to OCI Container Registry"
build_pipeline_stage_type = "DELIVER_ARTIFACT"
deliver_artifact_collection {
items {
artifact_name = "orders-api-image"
artifact_id = oci_devops_deploy_artifact.orders_api_image.id
}
items {
artifact_name = "kubernetes-manifests"
artifact_id = oci_devops_deploy_artifact.k8s_manifests.id
}
}
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline_stage.managed_build.id
}
}
}
# Stage 3: Trigger Deployment Pipeline
resource "oci_devops_build_pipeline_stage" "trigger_deploy" {
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
display_name = "trigger-deployment"
description = "Trigger the deployment pipeline on successful build"
build_pipeline_stage_type = "TRIGGER_DEPLOYMENT_PIPELINE"
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
is_pass_all_parameters_enabled = true
build_pipeline_stage_predecessor_collection {
items {
id = oci_devops_build_pipeline_stage.deliver_artifact.id
}
}
}

Step 6: Artifact Registry

resource "oci_artifacts_repository" "k8s_manifests_repo" {
compartment_id = var.compartment_id
display_name = "orders-api-manifests"
description = "Kubernetes deployment manifests for orders API"
is_immutable = false
repository_type = "GENERIC"
}
resource "oci_devops_deploy_artifact" "orders_api_image" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-container-image"
argument_substitution_mode = "SUBSTITUTE_PLACEHOLDERS"
deploy_artifact_type = "DOCKER_IMAGE"
deploy_artifact_source {
deploy_artifact_source_type = "OCIR"
image_uri = "${var.oci_region_key}.ocir.io/${var.tenancy_namespace}/orders-api:$${BUILDRUN_HASH}"
image_digest = " "
}
}
resource "oci_devops_deploy_artifact" "k8s_manifests" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-k8s-manifests"
argument_substitution_mode = "SUBSTITUTE_PLACEHOLDERS"
deploy_artifact_type = "KUBERNETES_MANIFEST"
deploy_artifact_source {
deploy_artifact_source_type = "GENERIC_ARTIFACT"
repository_id = oci_artifacts_repository.k8s_manifests_repo.id
deploy_artifact_path = "k8s/deployment.yaml"
deploy_artifact_version = "$${BUILDRUN_HASH}"
}
}

Step 7: Deployment Environment and Pipeline

The deployment pipeline targets the OKE cluster. Define the environment first, then the pipeline stages.

resource "oci_devops_deploy_environment" "oke_prod" {
project_id = oci_devops_project.orders_api_project.id
display_name = "oke-production"
description = "Production OKE cluster"
deploy_environment_type = "OKE_CLUSTER"
cluster_id = var.oke_cluster_id
}
resource "oci_devops_deploy_pipeline" "orders_api_deploy" {
project_id = oci_devops_project.orders_api_project.id
display_name = "orders-api-deploy"
description = "Blue-green deployment of orders API to production OKE"
deploy_pipeline_parameters {
items {
name = "NAMESPACE"
default_value = "orders"
description = "Kubernetes namespace for the deployment"
}
items {
name = "IMAGE_TAG"
default_value = "latest"
description = "Container image tag to deploy"
}
}
}
# Stage 1: Approval gate before production deployment
resource "oci_devops_deploy_stage" "approval_gate" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "production-approval"
description = "Manual approval required before deploying to production"
deploy_stage_type = "MANUAL_APPROVAL"
approval_policy {
approval_policy_type = "COUNT_BASED_APPROVAL"
number_of_approvals_required = 1
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_pipeline.orders_api_deploy.id
}
}
}
# Stage 2: Blue-green deploy to OKE
resource "oci_devops_deploy_stage" "oke_blue_green_deploy" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "oke-blue-green-deploy"
description = "Deploy new version to green environment"
deploy_stage_type = "OKE_BLUE_GREEN_DEPLOYMENT"
oke_blue_green_deploy_stage_details {
kubernetes_manifest_deploy_artifact_ids = [
oci_devops_deploy_artifact.k8s_manifests.id
]
oke_cluster_deploy_environment_id = oci_devops_deploy_environment.oke_prod.id
blue_green_strategy {
strategy_type = "NGINX_INGRESS_STRATEGY"
namespace_a = "orders-blue"
namespace_b = "orders-green"
ingress_name = "orders-api-ingress"
}
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_stage.approval_gate.id
}
}
}
# Stage 3: Traffic shift after successful deployment validation
resource "oci_devops_deploy_stage" "traffic_shift" {
deploy_pipeline_id = oci_devops_deploy_pipeline.orders_api_deploy.id
display_name = "shift-traffic-to-green"
description = "Shift 100% of traffic to the newly deployed green environment"
deploy_stage_type = "OKE_BLUE_GREEN_TRAFFIC_SHIFT"
oke_blue_green_traffic_shift_deploy_stage_details {
oke_blue_green_deployment_deploy_stage_id = oci_devops_deploy_stage.oke_blue_green_deploy.id
}
deploy_stage_predecessor_collection {
items {
id = oci_devops_deploy_stage.oke_blue_green_deploy.id
}
}
}

Step 8: Trigger on Code Push

The trigger watches the mirrored repository and fires the build pipeline when a push lands on the main branch.

resource "oci_devops_trigger" "main_branch_push" {
project_id = oci_devops_project.orders_api_project.id
display_name = "main-branch-push-trigger"
description = "Trigger build pipeline on every push to main"
trigger_source = "DEVOPS_CODE_REPOSITORY"
repository_id = oci_devops_repository.orders_api_repo.id
actions {
type = "TRIGGER_BUILD_PIPELINE"
build_pipeline_id = oci_devops_build_pipeline.orders_api_build.id
filter {
trigger_source = "DEVOPS_CODE_REPOSITORY"
events = ["PUSH"]
include {
head_ref = "main"
}
exclude {
file_filter {
file_paths = ["docs/*", "*.md", ".github/*"]
}
}
}
}
}

The exclude block prevents documentation-only changes from triggering a full build and deploy. Pushing a README update does not kick off the pipeline.

Step 9: Verifying the Pipeline

Once Terraform applies, validate the end-to-end flow.

Check mirror sync status:

oci devops repository get \
--repository-id <your-repo-ocid> \
--query 'data.{name:name, mirror-status:"mirror-repository-config"}' \
--output table

Manually trigger a build to test without waiting for a push:

oci devops build-run create \
--build-pipeline-id <your-build-pipeline-ocid> \
--display-name "manual-validation-run" \
--build-run-arguments '{"items": [{"name": "IMAGE_TAG", "value": "validation-test"}]}'

Watch the build run progress:

oci devops build-run get \
--build-run-id <build-run-ocid> \
--query 'data.{status:"lifecycle-state", phase:"build-run-progress"."build-pipeline-stage-run-progress"}' \
--output table

List deployment history to confirm deployments are being tracked:

oci devops deployment list \
--project-id <project-ocid> \
--sort-by timeCreated \
--sort-order DESC \
--limit 10 \
--query 'data.items[*].{name:"display-name", status:"lifecycle-state", time:\"time-created\"}' \
--output table

Rollback

If a deployment introduces a regression, OCI DevOps blue-green makes rollback immediate. Traffic is still flowing to the old environment until the traffic shift stage completes. If you catch the issue before the shift, simply reject the traffic shift stage from the console or CLI:

oci devops deployment approve \
--deployment-id <deployment-ocid> \
--deploy-stage-id <traffic-shift-stage-ocid> \
--reason "Rolling back: latency regression detected in green environment" \
--action REJECT

The green environment is torn down, the blue environment continues serving traffic, and the deployment is marked as failed with the reason recorded in the audit log.

Where This Fits in a Real Team

The value of OCI DevOps over an external CI/CD tool is not raw feature count. GitHub Actions or GitLab CI have richer marketplace ecosystems. The value is native IAM integration and residency inside your tenancy.

Build runners authenticate to OCI Vault, Container Registry, and Artifact Registry using the Dynamic Group policy with no credentials stored on a third-party platform. Every build and deployment is recorded in OCI Audit with the OCID of the pipeline that ran it. Deployment approvals are logged against the OCI user who approved or rejected them. For regulated environments where you need to prove that every production change was approved by a named human identity and executed by an automated system with least-privilege credentials, OCI DevOps gives you that audit trail natively.

For teams already running everything inside OCI, it is the most operationally coherent choice.

Regards,
Osama

Orchestrating Production Workflows with AWS Step Functions

I want to tell you about a production incident that still bothers me.

We had a payment processing system built on Lambda. Each function did one thing: validate the card, charge the customer, update the order, send the receipt, trigger fulfillment. Clean separation of concerns. Looked great on paper.

Then a Lambda timed out in the middle of the charge step. The card had been charged. The order had not been updated. The receipt never went out. Fulfillment never started. And because there was no central record of what had run, we had no way to resume from where things broke. We ended up with a manual cleanup process, a refund, and an angry customer.

The root problem was not the timeout. The root problem was that we had orchestration logic scattered across function calls, SQS queues, and environment variables. When something went wrong, we had no visibility and no way to recover cleanly.

AWS Step Functions exists to solve exactly this problem. It gives you a managed, visual, stateful orchestration layer that sits above your compute. In this article I will walk you through how Step Functions actually works, the patterns that matter in production, and the mistakes I see teams make when they first adopt it.

What Step Functions Actually Does

Step Functions is a serverless orchestration service. You define a workflow as a state machine using Amazon States Language, a JSON-based specification. Each state in the machine can invoke a Lambda function, call an AWS service directly, wait for a human approval, run a parallel branch, or retry on failure with configurable backoff.

The key thing that separates Step Functions from gluing Lambdas together with SQS is that the state machine itself is the source of truth. Every execution has a complete audit trail. You can look at any execution and see exactly which states ran, what input and output they received, when they ran, and whether they succeeded or failed. When something goes wrong you have a complete picture.

There are two workflow types and the choice matters.

Standard Workflows are designed for long-running, durable processes. They can run for up to a year. Every state transition is recorded in the execution history. You pay per state transition. This is what you want for anything involving payments, order processing, document workflows, or human approvals.

Express Workflows are designed for high-volume, short-duration workloads. They run for up to five minutes, have at-least-once execution semantics, and you pay per execution duration. Use them for event processing pipelines where you need to handle thousands of events per second and idempotency is handled at the application level.

Your First Production State Machine

Let me walk through a real example: an e-commerce order processing workflow. This is a Standard Workflow since order processing is exactly the kind of thing you need full durability and auditability for.

{
"Comment": "Order processing workflow",
"StartAt": "ValidateOrder",
"States": {
"ValidateOrder": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:validate-order",
"Next": "CheckInventory",
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": ["OrderValidationError"],
"Next": "OrderRejected",
"ResultPath": "$.error"
}
]
},
"CheckInventory": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:check-inventory",
"Next": "ProcessPayment",
"Retry": [
{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 5,
"MaxAttempts": 2,
"BackoffRate": 1.5
}
],
"Catch": [
{
"ErrorEquals": ["InsufficientInventoryError"],
"Next": "NotifyOutOfStock",
"ResultPath": "$.error"
}
]
},
"ProcessPayment": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:process-payment",
"Next": "FulfillmentAndNotification",
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2
}
],
"Catch": [
{
"ErrorEquals": ["PaymentDeclinedError"],
"Next": "NotifyPaymentFailed",
"ResultPath": "$.error"
},
{
"ErrorEquals": ["States.ALL"],
"Next": "OrderProcessingFailed",
"ResultPath": "$.error"
}
]
},
"FulfillmentAndNotification": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "TriggerFulfillment",
"States": {
"TriggerFulfillment": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:trigger-fulfillment",
"End": true
}
}
},
{
"StartAt": "SendConfirmationEmail",
"States": {
"SendConfirmationEmail": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789:function:send-email",
"End": true
}
}
}
],
"Next": "OrderComplete"
},
"OrderComplete": { "Type": "Succeed" },
"OrderRejected": { "Type": "Fail", "Error": "OrderRejected" },
"NotifyOutOfStock": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789:function:notify-out-of-stock", "End": true },
"NotifyPaymentFailed": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789:function:notify-payment-failed", "End": true },
"OrderProcessingFailed": { "Type": "Fail", "Error": "ProcessingFailed" }
}
}

A few things worth pointing out in this definition.

The Retry blocks on each Task state handle transient failures automatically. The configuration above retries on Lambda service exceptions with exponential backoff. You get this behavior for free without writing any retry logic in your Lambda functions themselves.

The Catch blocks handle business-logic failures separately from infrastructure failures. A PaymentDeclinedError routes to a notification state. An unhandled exception routes to a generic failure state. The ResultPath ensures the error detail is written into the execution context alongside the original input, not replacing it.

The Parallel state in FulfillmentAndNotification runs fulfillment and email simultaneously. Both branches must complete before the workflow advances to OrderComplete. If either branch fails, the entire Parallel state fails. This is often exactly the behavior you want: do not mark the order complete until both downstream systems have been notified.


SDK Integrations: Stop Writing Wrapper Lambdas

One of the most common mistakes I see is writing Lambda functions whose only job is to call another AWS service. A Lambda that calls DynamoDB to write a record. A Lambda that sends an SNS message. A Lambda that starts a Glue job.

Step Functions has optimized integrations with over 220 AWS services. You can call these services directly from a state definition without a Lambda in the middle.

Here is a state that writes directly to DynamoDB:

"SaveOrderToDynamo": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "orders",
"Item": {
"orderId": { "S.$": "$.orderId" },
"customerId": { "S.$": "$.customerId" },
"status": { "S": "CONFIRMED" },
"totalAmount":{ "N.$": "States.Format('{}', $.totalAmount)" },
"createdAt": { "S.$": "$$.Execution.StartTime" }
}
},
"Next": "SendToSNS"
}

And a state that publishes to SNS:

"SendToSNS": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"TopicArn": "arn:aws:sns:us-east-1:123456789:order-events",
"Message": {
"orderId.$": "$.orderId",
"customerId.$": "$.customerId",
"status": "CONFIRMED"
}
},
"Next": "OrderComplete"
}

The .$ suffix on a key means “resolve this from the state input.” The $$.Execution.StartTime is a context object reference that gives you metadata about the current execution. These small conveniences add up significantly when building real workflows.

Removing wrapper Lambdas reduces cold starts, lowers your Lambda invocation costs, simplifies your IAM surface, and makes the workflow easier to read because every state’s purpose is self-evident.

The Wait for Callback Pattern

Some workflows cannot move forward until something external happens. A human needs to approve a refund. A third-party payment processor needs to call back. A document needs to pass a review queue.

Step Functions handles this with the waitForTaskToken integration pattern. The state machine pauses, sends a token to an external system, and resumes only when that token is returned.

Here is the state definition:

"WaitForManagerApproval": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789/approval-queue",
"MessageBody": {
"taskToken.$": "$$.Task.Token",
"orderId.$": "$.orderId",
"amount.$": "$.totalAmount",
"requestedBy.$":"$.customerId"
}
},
"HeartbeatSeconds": 3600,
"Next": "ProcessApprovedRefund",
"Catch": [
{
"ErrorEquals": ["ApprovalRejected"],
"Next": "NotifyRejected"
},
{
"ErrorEquals": ["States.HeartbeatTimeout"],
"Next": "EscalateApproval"
}
]
}

The approval service picks up the message, presents it to a manager, and then calls back:

import boto3
sfn = boto3.client("stepfunctions")
def handle_approval_decision(task_token: str, approved: bool, reason: str):
if approved:
sfn.send_task_success(
taskToken=task_token,
output=json.dumps({"approved": True, "approvedBy": "manager@company.com"})
)
else:
sfn.send_task_failure(
taskToken=task_token,
error="ApprovalRejected",
cause=reason
)

The HeartbeatSeconds field is important. If the external system does not send a heartbeat or complete the task within that window, the state fails with a HeartbeatTimeout. In the example above that routes to an escalation state rather than silently hanging forever. Always set a heartbeat on any waitForTaskToken state.

Deploying with Terraform

Defining your state machine in the console is fine for exploration. In production, everything should be in code.

resource "aws_sfn_state_machine" "order_processing" {
name = "order-processing-workflow"
role_arn = aws_iam_role.step_functions_role.arn
type = "STANDARD"
definition = templatefile("${path.module}/state_machine.json", {
validate_order_arn = aws_lambda_function.validate_order.arn
check_inventory_arn = aws_lambda_function.check_inventory.arn
process_payment_arn = aws_lambda_function.process_payment.arn
trigger_fulfillment_arn = aws_lambda_function.trigger_fulfillment.arn
send_email_arn = aws_lambda_function.send_email.arn
})
logging_configuration {
level = "ALL"
include_execution_data = true
log_destination = "${aws_cloudwatch_log_group.sfn_logs.arn}:*"
}
tracing_configuration {
enabled = true
}
}
resource "aws_iam_role" "step_functions_role" {
name = "step-functions-order-processing-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "states.amazonaws.com" }
}]
})
}
resource "aws_iam_role_policy" "sfn_policy" {
name = "sfn-order-processing-policy"
role = aws_iam_role.step_functions_role.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["lambda:InvokeFunction"]
Resource = [
aws_lambda_function.validate_order.arn,
aws_lambda_function.check_inventory.arn,
aws_lambda_function.process_payment.arn,
aws_lambda_function.trigger_fulfillment.arn,
aws_lambda_function.send_email.arn
]
},
{
Effect = "Allow"
Action = ["logs:CreateLogDelivery", "logs:PutLogEvents", "logs:GetLogDelivery"]
Resource = "*"
},
{
Effect = "Allow"
Action = ["xray:PutTraceSegments", "xray:PutTelemetryRecords"]
Resource = "*"
}
]
})
}
resource "aws_cloudwatch_log_group" "sfn_logs" {
name = "/aws/states/order-processing"
retention_in_days = 30
}

Using templatefile to inject Lambda ARNs into the state machine definition keeps your infrastructure code clean and makes it easy to reference the correct function ARN for each environment without hardcoding anything.


Observability in Production

Step Functions gives you three layers of observability out of the box when you configure them properly.

CloudWatch Metrics publishes execution counts, failure rates, and durations for every state machine automatically. Set alarms on ExecutionsFailed and ExecutionsTimedOut. For payment or order workflows, a single failed execution is worth an alert. For high-volume event pipelines, set a threshold based on your acceptable failure rate.

CloudWatch Logs with include_execution_data = true captures the full input and output of every state transition. This is the setting that makes debugging possible. Without it, you know a state failed but not what data it received. With it, you can replay the exact scenario that caused the failure.

X-Ray tracing propagates trace context through Lambda invocations triggered by your state machine. In the AWS console, you get a service map showing exactly where time was spent across each execution. For workflows where latency matters, this is the fastest way to identify the bottleneck.

One practical tip: write a CloudWatch Insights query that you can run immediately when an incident starts.

fields @timestamp, execution_arn, type, details.name, details.status
| filter type in ["ExecutionFailed", "TaskFailed", "TaskStateExited"]
| sort @timestamp desc
| limit 50

Save this query before you need it. Running it during an incident is much faster than clicking through individual executions.


Common Mistakes

Not setting ResultPath on Catch handlers. By default, a Catch block replaces the entire state input with the error object. Your downstream states then receive only the error, not the original order data they need. Always use "ResultPath": "$.error" to merge the error into the existing input.

Using Express Workflows for payment processing. Express Workflows have at-least-once semantics. A state can execute more than once under failure conditions. For anything involving money or external side effects, use Standard Workflows with idempotency keys in your Lambda functions, or use Standard Workflows period.

Ignoring the execution history limit. Standard Workflow execution history is capped at 25,000 events. For very long-running workflows with many state transitions, you can hit this limit. If your workflow runs for days or weeks with thousands of steps, use the Map state with chunking to keep individual execution histories manageable.

Hardcoding ARNs in state machine definitions. Environment-specific ARNs belong in Terraform variables or SSM Parameter Store, not in your state machine JSON. The pattern shown above with templatefile keeps this clean.

Step Functions does not eliminate complexity. What it does is make complexity visible and manageable. Your business logic lives in Lambda. Your orchestration logic lives in the state machine. When something fails, you have a complete, queryable record of exactly what happened and where.

The teams that get the most value from Step Functions are the ones that resist the temptation to build orchestration logic into their Lambda functions. Keep each function focused on a single responsibility. Let the state machine handle sequencing, retries, error routing, and parallelism. The result is a system where debugging takes minutes instead of hours and where new team members can understand the full workflow by reading a single JSON file.

Enjoy the cloud.

Osama

OCI Network Firewall: Building a Centralized Inspection Architecture with Terraform

Security Lists and Network Security Groups handle stateful packet filtering at the subnet and VNIC level. They are the right tool for controlling which ports and protocols reach a resource. What they cannot do is inspect the content of traffic, detect threats based on application-layer signatures, block specific URLs or FQDNs, or apply SSL inspection to decrypt and re-encrypt traffic in flight. That requires a different layer entirely.

OCI Network Firewall is Oracle’s managed next-generation firewall service, built on Palo Alto Networks technology and integrated natively into VCN routing. It supports application-layer inspection, IDPS (Intrusion Detection and Prevention), URL filtering, FQDN-based rules, and TLS inspection. Unlike a third-party firewall appliance you would deploy on a compute instance, OCI Network Firewall is a fully managed service: Oracle handles the underlying infrastructure, HA, and scaling. You manage the policy.

In this post I will walk through designing a hub-and-spoke inspection architecture, deploying the firewall and its policy using Terraform, configuring IDPS and URL filtering rules, and validating traffic flow with OCI Flow Logs.

Architecture: Hub-and-Spoke with Centralized Inspection

The standard pattern for OCI Network Firewall in multi-VCN environments is centralized inspection through a hub VCN. All spoke VCNs route traffic through the hub, and the firewall sits in the hub inspecting both north-south (internet-bound) and east-west (spoke-to-spoke) traffic.

Traffic routing in this architecture uses a combination of DRG route tables and VCN ingress/egress route tables to steer all flows through the firewall subnet before they reach their destination. This is the most important concept to get right: the firewall only inspects traffic that is routed through it. Misconfigured route tables mean packets bypass the firewall entirely with no error or warning.

Step 1: Hub VCN and Firewall Subnet

resource "oci_core_vcn" "hub_vcn" {
compartment_id = var.compartment_id
cidr_blocks = ["192.168.0.0/16"]
display_name = "hub-inspection-vcn"
dns_label = "hubvcn"
}
# Firewall subnet - the firewall VNIC lives here
resource "oci_core_subnet" "firewall_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.hub_vcn.id
cidr_block = "192.168.1.0/24"
display_name = "firewall-subnet"
dns_label = "fwsubnet"
prohibit_public_ip_on_vnic = true
route_table_id = oci_core_route_table.firewall_subnet_rt.id
security_list_ids = [oci_core_security_list.firewall_sl.id]
}
# Internet Gateway for north-south traffic
resource "oci_core_internet_gateway" "hub_igw" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.hub_vcn.id
display_name = "hub-internet-gateway"
enabled = true
}
# DRG for spoke VCN attachment
resource "oci_core_drg" "hub_drg" {
compartment_id = var.compartment_id
display_name = "hub-drg"
}
resource "oci_core_drg_attachment" "hub_vcn_attachment" {
drg_id = oci_core_drg.hub_drg.id
display_name = "hub-vcn-attachment"
network_details {
id = oci_core_vcn.hub_vcn.id
type = "VCN"
}
}

The firewall subnet must not have a public IP on its VNIC. The firewall receives traffic through routing, not through a public endpoint.

Step 2: Firewall Policy

The policy is the heart of the firewall. It contains address lists, URL lists, application lists, and the ordered set of security rules. All of these are defined as child resources of the policy and are applied when the policy is attached to a firewall instance.

resource "oci_network_firewall_network_firewall_policy" "production_policy" {
compartment_id = var.compartment_id
display_name = "production-inspection-policy"
}
# IP address list for trusted internal RFC1918 ranges
resource "oci_network_firewall_network_firewall_policy_address_list" "internal_ranges" {
name = "internal-rfc1918"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
type = "IP"
addresses = [
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16"
]
}
# FQDN list for allowed outbound SaaS destinations
resource "oci_network_firewall_network_firewall_policy_address_list" "allowed_saas" {
name = "allowed-saas-fqdns"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
type = "FQDN"
addresses = [
"*.oracle.com",
"*.oraclecloud.com",
"*.github.com",
"registry-1.docker.io",
"auth.docker.io",
"production.cloudflare.docker.com"
]
}
# URL list for blocked categories
resource "oci_network_firewall_network_firewall_policy_url_list" "blocked_urls" {
name = "blocked-url-categories"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
urls {
pattern = "*.pastebin.com"
type = "SIMPLE"
}
urls {
pattern = "*.ngrok.io"
type = "SIMPLE"
}
urls {
pattern = "*.ngrok-free.app"
type = "SIMPLE"
}
}
# Application list scoping HTTPS traffic
resource "oci_network_firewall_network_firewall_policy_application_group" "web_apps" {
name = "web-traffic"
network_firewall_policy_id = oci_network_firewall_network_firewall_policy.production_policy.id
apps = ["HTTP", "HTTPS", "SSL"]
}

Step 3: Security Rules

Rules are evaluated in order. The first matching rule wins. Structure your rules from most specific to most general and always end with an explicit deny-all for traffic that does not match any allow rule.

OCI Dedicated Region Cloud@Customer: Architecture, Deployment Patterns, and Terraform Automation

Most cloud conversations start with a simple assumption: your workloads go to the cloud provider’s data center. For a large number of organizations, particularly in government, financial services, healthcare, and defense, that assumption is the exact problem. Data sovereignty laws, regulatory requirements, and security classification levels mean that certain workloads cannot leave a specific physical location, full stop.

OCI Dedicated Region Cloud@Customer, commonly referred to as DRCC, solves this without forcing a compromise. Oracle deploys a full OCI region — not a subset of services, not a gateway appliance, but a complete cloud region with the same hardware, software stack, APIs, and SLAs — inside your own data center. You get every OCI service you would use in a public region, with the control plane managed by Oracle and the physical infrastructure sitting on your floor.

In this post I will cover how DRCC is architected, how it differs from OCI Exadata Cloud@Customer and Roving Edge, the networking requirements, IAM federation considerations, and how to automate workload deployment using Terraform once the region is live.

What DRCC Actually Delivers

The distinction between DRCC and other on-premises cloud appliances matters technically. Most cloud-at-customer offerings give you a subset of services through a dedicated appliance: a handful of compute shapes, object storage, and maybe a managed database. DRCC is architecturally different.

Oracle physically ships and installs the same rack infrastructure used in public OCI regions into your facility. The region runs the same OCI control plane software, exposes the same REST APIs, and integrates with OCI IAM and Oracle Cloud Console using the same tooling. When you run a Terraform plan against a DRCC region, the provider configuration is identical to a public region. You change the region identifier in your config and the code works without modification.

The full service catalog available in DRCC includes Compute (including bare metal and GPU shapes), OKE (Oracle Kubernetes Engine), Autonomous Database, Exadata Database Service, Object Storage, Block Volumes, File Storage, VCN, Load Balancer, API Gateway, Functions, Streaming, OCI Vault, Identity and Access Management, Monitoring, Logging, Events, and Notifications. This is not a stripped-down subset — it is the complete stack.

Hardware minimum footprint starts at a base rack configuration that supports a production workload. Oracle handles all hardware maintenance, software patching, and control plane operations. Your team manages what runs on top: compartments, IAM policies, networking, and workloads.

How DRCC Differs from Related Oracle Offerings

Before going further it is worth clarifying where DRCC sits relative to two commonly confused offerings.

OCI Exadata Cloud@Customer deploys Exadata Database Service hardware into your data center. It is a database-specific offering. You get Autonomous Database and Exadata Database Service on-premises, but not the broader OCI service catalog. If you need compute, containers, serverless, and object storage alongside the database layer, Exadata Cloud@Customer alone does not cover it.

OCI Roving Edge Infrastructure is a ruggedized portable device designed for disconnected or intermittently connected environments: ships, remote field operations, military forward deployments. It runs a subset of OCI services and is designed to operate without a persistent connection to the OCI control plane. DRCC requires a reliable network connection back to Oracle for control plane operations and is designed for fixed, well-connected facilities.

DRCC is the right choice when you need the full OCI service catalog, the workloads must stay on-premises for regulatory or sovereignty reasons, and you have a proper data center with the power, cooling, and network capacity to host the infrastructure.

Network Architecture Requirements

DRCC has specific network requirements that you need to understand before the hardware arrives. Getting these wrong means the region cannot operate.

The DRCC racks need connectivity on three planes: the management network, the customer data network, and the Oracle back-channel.

The management network connects Oracle’s control plane software running inside your facility to Oracle’s global control plane over the internet or a dedicated circuit. Oracle uses this path for software updates, monitoring, and operational management of the region. This connection is outbound-initiated from the DRCC hardware, encrypted with TLS, and authenticated with certificates. Oracle publishes the specific IP ranges that need to be permitted through your firewall. You do not control what flows over this channel, but Oracle’s contractual commitments define exactly what does.

The customer data network connects your existing on-premises infrastructure to the DRCC region. This is a standard 25G or 100G ethernet connection depending on the rack configuration. You configure VCN peering or FastConnect-equivalent local connections to bridge your existing network into the DRCC VCN.

Here is how you configure a VCN in DRCC using Terraform, which is identical to a public region:

terraform {
required_providers {
oci = {
source = "oracle/oci"
version = ">= 5.0.0"
}
}
}
provider "oci" {
tenancy_ocid = var.tenancy_ocid
user_ocid = var.user_ocid
fingerprint = var.fingerprint
private_key_path = var.private_key_path
# This is your DRCC region identifier
# Oracle assigns this during provisioning, format: us-yourdatacenter-1
region = var.drcc_region
}
resource "oci_core_vcn" "drcc_primary_vcn" {
compartment_id = var.compartment_id
cidr_blocks = ["10.100.0.0/16"]
display_name = "drcc-primary-vcn"
dns_label = "drccprimary"
}
# Application tier subnet - private
resource "oci_core_subnet" "app_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.drcc_primary_vcn.id
cidr_block = "10.100.1.0/24"
display_name = "app-private-subnet"
dns_label = "apppriv"
prohibit_public_ip_on_vnic = true
route_table_id = oci_core_route_table.private_rt.id
security_list_ids = [oci_core_security_list.app_sl.id]
}
# Database tier subnet - private
resource "oci_core_subnet" "db_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.drcc_primary_vcn.id
cidr_block = "10.100.2.0/24"
display_name = "db-private-subnet"
dns_label = "dbpriv"
prohibit_public_ip_on_vnic = true
route_table_id = oci_core_route_table.private_rt.id
security_list_ids = [oci_core_security_list.db_sl.id]
}
# Local Peering Gateway to connect DRCC VCN to your on-premises network
resource "oci_core_local_peering_gateway" "onprem_lpg" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.drcc_primary_vcn.id
display_name = "onprem-peering-gateway"
}

The Local Peering Gateway in DRCC context connects the DRCC VCN to your on-premises routed network via the physical data network. This gives your existing on-premises workloads direct, low-latency access to everything running in the DRCC region without traffic ever leaving your facility.

IAM Federation in a DRCC Deployment

DRCC shares the OCI IAM control plane with the public region associated with your tenancy. This has important implications for how you manage identities.

Your DRCC region is part of your existing OCI tenancy. Users, groups, and dynamic groups created in OCI IAM apply to DRCC resources the same way they apply to public region resources. If you already federate OCI IAM with your corporate identity provider (Active Directory, Okta, Azure AD), those federated identities work in DRCC without additional configuration.

Here is the IAM federation configuration for Active Directory using SAML:

# Identity Provider configuration for AD FS
resource "oci_identity_identity_provider" "ad_federation" {
compartment_id = var.tenancy_ocid
name = "corporate-adfs"
description = "Corporate Active Directory Federation Services"
product_type = "ADFS"
protocol = "SAML2"
metadata = file("${path.module}/adfs-metadata.xml")
freeform_tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
# Map AD group to OCI group for DRCC operations team
resource "oci_identity_idp_group_mapping" "drcc_admins_mapping" {
idp_id = oci_identity_identity_provider.ad_federation.id
idp_group_name = "CN=DRCC-Admins,OU=CloudTeams,DC=corp,DC=example,DC=com"
group_id = oci_identity_group.drcc_admins.id
}
resource "oci_identity_group" "drcc_admins" {
compartment_id = var.tenancy_ocid
name = "drcc-platform-admins"
description = "DRCC platform administration team"
}
# Compartment structure for DRCC workload isolation
resource "oci_identity_compartment" "drcc_root" {
compartment_id = var.tenancy_ocid
name = "drcc-production"
description = "Root compartment for all DRCC production workloads"
}
resource "oci_identity_compartment" "drcc_networking" {
compartment_id = oci_identity_compartment.drcc_root.id
name = "drcc-networking"
description = "Networking resources for DRCC region"
}
resource "oci_identity_compartment" "drcc_workloads" {
compartment_id = oci_identity_compartment.drcc_root.id
name = "drcc-workloads"
description = "Application workloads running in DRCC"
}
# Least-privilege policy for DRCC admins
resource "oci_identity_policy" "drcc_admin_policy" {
compartment_id = oci_identity_compartment.drcc_root.id
name = "drcc-admin-policy"
description = "Platform admin permissions scoped to DRCC compartment"
statements = [
"Allow group drcc-platform-admins to manage all-resources in compartment drcc-production",
"Allow group drcc-platform-admins to read all-resources in tenancy where request.region = '${var.drcc_region}'",
"Allow group drcc-platform-admins to manage virtual-network-family in compartment drcc-production:drcc-networking",
"Allow group drcc-platform-admins to manage instance-family in compartment drcc-production:drcc-workloads",
"Allow group drcc-platform-admins to manage autonomous-database-family in compartment drcc-production:drcc-workloads"
]
}

One critical IAM behavior specific to DRCC: you can write IAM policies that restrict actions to your DRCC region using the request.region condition. This means a group can have full admin rights in DRCC but zero access to your public OCI regions, or vice versa. For organizations with strict separation between on-premises and cloud teams, this is an important control.

Deploying OKE on DRCC

OKE on DRCC runs the same as OKE in a public region. The control plane components run inside the DRCC rack. The API server endpoint is reachable from within your data center network without any traffic leaving the facility.

resource "oci_containerengine_cluster" "drcc_cluster" {
compartment_id = oci_identity_compartment.drcc_workloads.id
kubernetes_version = "v1.29.1"
name = "drcc-production-cluster"
vcn_id = oci_core_vcn.drcc_primary_vcn.id
endpoint_config {
is_public_ip_enabled = false
subnet_id = oci_core_subnet.app_subnet.id
}
options {
service_lb_subnet_ids = [oci_core_subnet.app_subnet.id]
kubernetes_network_config {
pods_cidr = "10.244.0.0/16"
services_cidr = "10.96.0.0/16"
}
add_ons {
is_kubernetes_dashboard_enabled = false
is_tiller_enabled = false
}
}
}
resource "oci_containerengine_node_pool" "drcc_workers" {
cluster_id = oci_containerengine_cluster.drcc_cluster.id
compartment_id = oci_identity_compartment.drcc_workloads.id
kubernetes_version = "v1.29.1"
name = "drcc-worker-pool"
node_config_details {
size = 3
placement_configs {
availability_domain = data.oci_identity_availability_domains.drcc_ads.availability_domains[0].name
subnet_id = oci_core_subnet.app_subnet.id
}
}
node_shape = "VM.Standard3.Flex"
node_shape_config {
memory_in_gbs = 64
ocpus = 8
}
node_source_details {
image_id = data.oci_core_images.ol8_image.images[0].id
source_type = "IMAGE"
boot_volume_size_in_gbs = 100
}
initial_node_labels {
key = "workload-tier"
value = "application"
}
}

The is_public_ip_enabled = false on the endpoint config is non-negotiable in a DRCC context. The API server should only be reachable from within your data center network. Any tooling that manages the cluster (Argo CD, Flux, CI pipelines) connects to the internal endpoint directly.

Deploying Autonomous Database on DRCC

Autonomous Database on DRCC is identical in API and behavior to the public region version. The database runs entirely within your facility.

resource "oci_database_autonomous_database" "drcc_adb" {
compartment_id = oci_identity_compartment.drcc_workloads.id
db_name = "DRCCPROD"
display_name = "drcc-production-adb"
db_workload = "OLTP"
cpu_core_count = 4
data_storage_size_in_tbs = 2
admin_password = var.adb_admin_password
is_auto_scaling_enabled = true
is_dedicated = false
# Private endpoint configuration - no public access
subnet_id = oci_core_subnet.db_subnet.id
private_endpoint_label = "drccprodadb"
is_access_control_enabled = true
whitelisted_ips = [
oci_core_subnet.app_subnet.id
]
defined_tags = {
"Operations.Environment" = "production"
"Operations.Region" = "drcc"
"Operations.ManagedBy" = "terraform"
}
}

The subnet_id and private_endpoint_label fields configure the database with a private endpoint inside the db subnet. Only resources in the whitelisted subnets can connect. No public endpoint is created.

Security Baseline for DRCC Deployments

DRCC gives you physical control over the hardware, but that does not mean you can skip the standard OCI security baseline. The software layer still requires proper configuration.

Enable Cloud Guard at the tenancy level scoped to your DRCC compartments:

resource "oci_cloud_guard_cloud_guard_configuration" "drcc_cloud_guard" {
compartment_id = var.tenancy_ocid
reporting_region = var.drcc_region
status = "ENABLED"
}
resource "oci_cloud_guard_target" "drcc_target" {
compartment_id = oci_identity_compartment.drcc_root.id
display_name = "drcc-production-target"
target_resource_id = oci_identity_compartment.drcc_root.id
target_resource_type = "COMPARTMENT"
target_detector_recipes {
detector_recipe_id = data.oci_cloud_guard_detector_recipes.config_recipe.detector_recipe_collection[0].items[0].id
}
target_responder_recipes {
responder_recipe_id = data.oci_cloud_guard_responder_recipes.oci_responder.responder_recipe_collection[0].items[0].id
}
}

Enable Vault for all secrets, keys, and credentials used by workloads running in DRCC. Because the Vault service runs inside the rack, key material never leaves your facility:

resource "oci_kms_vault" "drcc_vault" {
compartment_id = oci_identity_compartment.drcc_workloads.id
display_name = "drcc-workloads-vault"
vault_type = "VIRTUAL_PRIVATE"
}
resource "oci_kms_key" "drcc_master_key" {
compartment_id = oci_identity_compartment.drcc_workloads.id
display_name = "drcc-master-encryption-key"
management_endpoint = oci_kms_vault.drcc_vault.management_endpoint
key_shape {
algorithm = "AES"
length = 32
}
protection_mode = "HSM"
}

The VIRTUAL_PRIVATE vault type and HSM protection mode ensure the key material is stored in the hardware security module inside the DRCC rack. Combined with the fact that the rack is physically in your data center, you have full chain-of-custody over the cryptographic material protecting your data.

Operational Considerations

A few things that are specific to operating DRCC that do not come up when working with public regions.

Oracle is responsible for hardware maintenance and software patching of the control plane. You receive advance notification of maintenance windows. During a control plane maintenance window, the management APIs may be briefly unavailable, but running workloads continue without interruption. Plan your deployment pipelines to account for these windows.

Capacity planning is different from the public cloud. In a public region, you scale up by requesting more resources and the cloud absorbs the demand. In DRCC, you have a fixed hardware footprint. If you need to scale beyond the initial rack configuration, you work with Oracle to add capacity. Build capacity planning reviews into your quarterly operations cycle and monitor resource utilization with OCI Monitoring the same way you would in a public region.

The Oracle back-channel for management operations needs to be permanently open. If your network team applies a firewall rule that blocks this traffic, the control plane loses contact with Oracle and becomes degraded. Work with Oracle to get the exact IP ranges and port requirements before go-live and document them clearly in your firewall change management process.

When DRCC Is the Right Choice

DRCC makes sense when at least one of these conditions is true: your regulatory framework requires data residency within a specific physical location you control, your security classification means workloads cannot traverse public internet infrastructure at any point, your latency requirements for database and application tiers demand co-location in your own facility, or you have existing on-premises infrastructure that needs tight integration with cloud services without egress cost or latency overhead.

It is not the right choice for organizations that want cloud economics without data center investment, for workloads with highly variable capacity requirements that would benefit from elastic public cloud scaling, or for teams that want to avoid the operational overhead of maintaining physical infrastructure.

For those who do meet the criteria, DRCC is one of the more complete sovereign cloud offerings on the market. The fact that the APIs and tooling are identical to the public cloud means your engineers do not need to learn a second system, your Terraform code travels unchanged, and your OKE workloads run without modification.

Regards,

Osama

Cross-Cloud Secret Synchronization: AWS Secrets Manager and OCI Vault in a Production Multi-Cloud Setup

One of the most overlooked problems in multi-cloud environments is secrets management across providers. Teams usually solve it badly: they store the same secret in both clouds manually, forget to rotate one of them, and find out during an outage that the credentials have been out of sync for three months.

In this post I will walk through building an automated secrets synchronization pipeline between AWS Secrets Manager and OCI Vault. When a secret rotates in AWS, the pipeline detects the rotation event, retrieves the new value, and pushes it into OCI Vault automatically. Everything is built with Terraform, an AWS Lambda function, and OCI IAM. No manual steps after the initial deployment.

This is a pattern I have used in environments where the database layer runs on OCI (leveraging Oracle Database pricing and performance) while the application layer runs on AWS. Both sides need the same database credentials, and both sides need to stay in sync without human intervention.

Architecture

The flow works like this:

AWS Secrets Manager rotation event fires via EventBridge, which triggers a Lambda function. The Lambda retrieves the new secret value, authenticates to OCI using an API key stored in its own environment (not hardcoded), and calls the OCI Vault API to update the corresponding secret version. OCI Vault stores the new value and makes it available to workloads running in OCI.

Prerequisites

Before starting you need:

  • AWS account with permissions to manage Secrets Manager, Lambda, EventBridge, and IAM
  • OCI tenancy with permissions to manage Vault, Keys, and IAM policies
  • Terraform 1.5 or later
  • Python 3.11 for the Lambda function
  • An existing OCI Vault and master encryption key (or we will create one)

Step 1: OCI Vault and IAM Setup

Start with OCI. We need a Vault, a master key, and an IAM user whose API key the Lambda will use to authenticate.

hcl

# OCI Vault
resource "oci_kms_vault" "app_vault" {
compartment_id = var.compartment_id
display_name = "multi-cloud-secrets-vault"
vault_type = "DEFAULT"
}
# Master Encryption Key inside the Vault
resource "oci_kms_key" "secrets_key" {
compartment_id = var.compartment_id
display_name = "secrets-master-key"
management_endpoint = oci_kms_vault.app_vault.management_endpoint
key_shape {
algorithm = "AES"
length = 32
}
}
# IAM user for cross-cloud access
resource "oci_identity_user" "sync_user" {
compartment_id = var.tenancy_ocid
name = "aws-secrets-sync-user"
description = "Service user for AWS Lambda to push secrets into OCI Vault"
email = "sync-user@internal.example.com"
}
# API key for the sync user (you will generate the actual key pair separately)
resource "oci_identity_api_key" "sync_user_key" {
user_id = oci_identity_user.sync_user.id
key_value = var.oci_sync_user_public_key_pem
}
# IAM group for the sync user
resource "oci_identity_group" "sync_group" {
compartment_id = var.tenancy_ocid
name = "secrets-sync-group"
description = "Group for cross-cloud secrets sync service users"
}
resource "oci_identity_user_group_membership" "sync_membership" {
group_id = oci_identity_group.sync_group.id
user_id = oci_identity_user.sync_user.id
}
# Minimal IAM policy - only what is needed, nothing more
resource "oci_identity_policy" "sync_policy" {
compartment_id = var.compartment_id
name = "secrets-sync-policy"
description = "Allows sync user to manage secrets in the app vault only"
statements = [
"Allow group secrets-sync-group to manage secret-family in compartment id ${var.compartment_id} where target.vault.id = '${oci_kms_vault.app_vault.id}'",
"Allow group secrets-sync-group to use keys in compartment id ${var.compartment_id} where target.key.id = '${oci_kms_key.secrets_key.id}'"
]
}

The policy scope is intentionally narrow. The sync user can only manage secrets inside this specific vault and can only use this specific key. If the AWS Lambda credentials are ever compromised, the blast radius is limited to this vault.

Step 2: Create the Initial Secret in OCI Vault

We need a secret placeholder in OCI Vault that the Lambda will update. The initial value does not matter since it will be overwritten on the first sync.

hcl

resource "oci_vault_secret" "db_password" {
compartment_id = var.compartment_id
vault_id = oci_kms_vault.app_vault.id
key_id = oci_kms_key.secrets_key.id
secret_name = "prod-db-password"
secret_content {
content_type = "BASE64"
content = base64encode("initial-placeholder-value")
name = "v1"
stage = "CURRENT"
}
metadata = {
source = "aws-secrets-manager"
aws_secret = "prod/database/password"
environment = "production"
}
}

Step 3: AWS Secrets Manager and the Source Secret

On the AWS side, create the authoritative secret and enable automatic rotation.

hcl

resource "aws_secretsmanager_secret" "db_password" {
name = "prod/database/password"
description = "Production database password - synced to OCI Vault"
recovery_window_in_days = 7
tags = {
Environment = "production"
SyncTarget = "oci-vault"
OciSecretName = "prod-db-password"
}
}
resource "aws_secretsmanager_secret_version" "db_password_v1" {
secret_id = aws_secretsmanager_secret.db_password.id
secret_string = jsonencode({
username = "db_admin",
password = var.initial_db_password,
host = var.db_host,
port = 1521,
database = "PRODDB"
})
}
# Rotation configuration - rotate every 30 days
resource "aws_secretsmanager_secret_rotation" "db_password_rotation" {
secret_id = aws_secretsmanager_secret.db_password.id
rotation_lambda_arn = aws_lambda_function.db_rotation_lambda.arn
rotation_rules {
automatically_after_days = 30
}
}

Step 4: Store OCI Credentials in AWS Secrets Manager

The Lambda needs OCI API credentials to authenticate. Store them as a secret in AWS Secrets Manager so they never appear in Lambda environment variables in plaintext.

hcl

resource "aws_secretsmanager_secret" "oci_credentials" {
name = "internal/oci-sync-credentials"
description = "OCI API key credentials for secrets sync Lambda"
tags = {
Environment = "production"
Purpose = "cross-cloud-sync"
}
}
resource "aws_secretsmanager_secret_version" "oci_credentials_v1" {
secret_id = aws_secretsmanager_secret.oci_credentials.id
secret_string = jsonencode({
tenancy_ocid = var.oci_tenancy_ocid,
user_ocid = var.oci_sync_user_ocid,
fingerprint = var.oci_api_key_fingerprint,
private_key = var.oci_private_key_pem,
region = var.oci_region
})
}

Step 5: The Lambda Function

This is the core of the pipeline. The Lambda retrieves the rotated secret from AWS Secrets Manager, loads OCI credentials from its own secrets store, and calls the OCI Vault API to create a new secret version.

python

import boto3
import json
import base64
import oci
import logging
import os
from datetime import datetime, timezone
from botocore.exceptions import ClientError
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def get_oci_config():
"""Retrieve OCI credentials from AWS Secrets Manager."""
client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
try:
response = client.get_secret_value(
SecretId=os.environ["OCI_CREDENTIALS_SECRET_ARN"]
)
creds = json.loads(response["SecretString"])
return {
"tenancy": creds["tenancy_ocid"],
"user": creds["user_ocid"],
"fingerprint": creds["fingerprint"],
"key_content": creds["private_key"],
"region": creds["region"]
}
except ClientError as e:
logger.error(f"Failed to retrieve OCI credentials: {e}")
raise
def get_aws_secret(secret_arn: str) -> str:
"""Retrieve the current value of an AWS secret."""
client = boto3.client("secretsmanager", region_name=os.environ["AWS_REGION"])
try:
response = client.get_secret_value(SecretId=secret_arn)
return response.get("SecretString") or base64.b64decode(
response["SecretBinary"]
).decode("utf-8")
except ClientError as e:
logger.error(f"Failed to retrieve AWS secret {secret_arn}: {e}")
raise
def push_to_oci_vault(
oci_config: dict,
vault_id: str,
key_id: str,
secret_ocid: str,
secret_value: str
):
"""Create a new version of an OCI Vault secret."""
vaults_client = oci.vault.VaultsClient(oci_config)
encoded_value = base64.b64encode(secret_value.encode("utf-8")).decode("utf-8")
update_details = oci.vault.models.UpdateSecretDetails(
secret_content=oci.vault.models.Base64SecretContentDetails(
content_type=oci.vault.models.SecretContentDetails.CONTENT_TYPE_BASE64,
content=encoded_value,
name=f"sync-{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}",
stage="CURRENT"
),
metadata={
"synced_from": "aws-secrets-manager",
"synced_at": datetime.now(timezone.utc).isoformat()
}
)
response = vaults_client.update_secret(
secret_id=secret_ocid,
update_secret_details=update_details
)
logger.info(
f"OCI secret updated. OCID: {secret_ocid}, "
f"New version: {response.data.current_version_number}"
)
return response.data
def handler(event, context):
"""
EventBridge trigger handler.
Expects event detail to contain:
- aws_secret_arn: ARN of the rotated AWS secret
- oci_secret_ocid: OCID of the target OCI Vault secret
- oci_vault_id: OCID of the target OCI Vault
- oci_key_id: OCID of the OCI KMS key
"""
logger.info(f"Received event: {json.dumps(event)}")
detail = event.get("detail", {})
aws_secret_arn = detail.get("aws_secret_arn")
oci_secret_ocid = detail.get("oci_secret_ocid")
oci_vault_id = detail.get("oci_vault_id")
oci_key_id = detail.get("oci_key_id")
if not all([aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id]):
logger.error("Missing required fields in event detail")
raise ValueError("Event detail must include aws_secret_arn, oci_secret_ocid, oci_vault_id, oci_key_id")
logger.info(f"Syncing secret: {aws_secret_arn} to OCI: {oci_secret_ocid}")
# Step 1: Get OCI credentials
oci_config = get_oci_config()
# Step 2: Retrieve the rotated AWS secret
secret_value = get_aws_secret(aws_secret_arn)
# Step 3: Push to OCI Vault
result = push_to_oci_vault(
oci_config=oci_config,
vault_id=oci_vault_id,
key_id=oci_key_id,
secret_ocid=oci_secret_ocid,
secret_value=secret_value
)
return {
"statusCode": 200,
"body": {
"message": "Secret synced successfully",
"oci_secret_ocid": oci_secret_ocid,
"oci_version": result.current_version_number
}
}

Step 6: Lambda IAM Role and Deployment

hcl

data "aws_iam_policy_document" "lambda_assume_role" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["lambda.amazonaws.com"]
}
}
}
data "aws_iam_policy_document" "lambda_permissions" {
statement {
effect = "Allow"
actions = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
]
resources = [
aws_secretsmanager_secret.db_password.arn,
aws_secretsmanager_secret.oci_credentials.arn
]
}
statement {
effect = "Allow"
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
resources = ["arn:aws:logs:*:*:*"]
}
}
resource "aws_iam_role" "sync_lambda_role" {
name = "secrets-sync-lambda-role"
assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}
resource "aws_iam_role_policy" "sync_lambda_policy" {
name = "secrets-sync-lambda-policy"
role = aws_iam_role.sync_lambda_role.id
policy = data.aws_iam_policy_document.lambda_permissions.json
}
resource "aws_lambda_function" "secrets_sync" {
filename = "${path.module}/lambda/secrets_sync.zip"
function_name = "oci-secrets-sync"
role = aws_iam_role.sync_lambda_role.arn
handler = "main.handler"
runtime = "python3.11"
timeout = 60
memory_size = 256
source_code_hash = filebase64sha256("${path.module}/lambda/secrets_sync.zip")
environment {
variables = {
OCI_CREDENTIALS_SECRET_ARN = aws_secretsmanager_secret.oci_credentials.arn
AWS_REGION = var.aws_region
}
}
layers = [aws_lambda_layer_version.oci_sdk_layer.arn]
}

Bundle the OCI Python SDK as a Lambda Layer so the function does not need to package it inline:

bash

mkdir -p lambda_layer/python
pip install oci --target lambda_layer/python
cd lambda_layer && zip -r ../oci_sdk_layer.zip python/

hcl

resource "aws_lambda_layer_version" "oci_sdk_layer" {
filename = "${path.module}/oci_sdk_layer.zip"
layer_name = "oci-python-sdk"
compatible_runtimes = ["python3.11"]
source_code_hash = filebase64sha256("${path.module}/oci_sdk_layer.zip")
}

Step 7: EventBridge Rule to Trigger on Rotation

hcl

resource "aws_cloudwatch_event_rule" "secret_rotation_rule" {
name = "detect-secret-rotation"
description = "Fires when a Secrets Manager secret rotation completes"
event_pattern = jsonencode({
source = ["aws.secretsmanager"],
detail-type = ["AWS API Call via CloudTrail"],
detail = {
eventSource = ["secretsmanager.amazonaws.com"],
eventName = ["RotateSecret", "PutSecretValue"]
}
})
}
resource "aws_cloudwatch_event_target" "sync_lambda_target" {
rule = aws_cloudwatch_event_rule.secret_rotation_rule.name
target_id = "SyncToOCI"
arn = aws_lambda_function.secrets_sync.arn
input_transformer {
input_paths = {
secret_arn = "$.detail.requestParameters.secretId"
}
input_template = <<EOF
{
"detail": {
"aws_secret_arn": "<secret_arn>",
"oci_secret_ocid": "${var.oci_db_password_secret_ocid}",
"oci_vault_id": "${oci_kms_vault.app_vault.id}",
"oci_key_id": "${oci_kms_key.secrets_key.id}"
}
}
EOF
}
}
resource "aws_lambda_permission" "allow_eventbridge" {
statement_id = "AllowEventBridgeInvoke"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.secrets_sync.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.secret_rotation_rule.arn
}

Step 8: Verifying the Pipeline

Manually trigger a rotation to test the full pipeline without waiting 30 days:

bash

# Force a rotation in AWS
aws secretsmanager rotate-secret \
--secret-id prod/database/password \
--region us-east-1
# Check Lambda execution logs
aws logs tail /aws/lambda/oci-secrets-sync --follow
# Verify the new version appeared in OCI Vault
oci vault secret get \
--secret-id <your-oci-secret-ocid> \
--query 'data.{name:secret-name, version:"current-version-number", updated:"time-of-current-version-need-rotation"}' \
--output table

A successful sync produces output similar to this in the Lambda logs:

INFO: Syncing secret: arn:aws:secretsmanager:us-east-1:123456789:secret:prod/database/password to OCI: ocid1.vaultsecret.oc1...
INFO: OCI secret updated. OCID: ocid1.vaultsecret.oc1..., New version: 3

Handling Failures and Drift

The pipeline as built is synchronous and event-driven, which means if the Lambda fails, the OCI secret does not get updated. Add a dead-letter queue and a reconciliation function that runs on a schedule to catch any drift.

hcl

resource "aws_sqs_queue" "sync_dlq" {
name = "secrets-sync-dlq"
message_retention_seconds = 86400
}
resource "aws_lambda_function_event_invoke_config" "sync_retry" {
function_name = aws_lambda_function.secrets_sync.function_name
maximum_retry_attempts = 2
maximum_event_age_in_seconds = 300
destination_config {
on_failure {
destination = aws_sqs_queue.sync_dlq.arn
}
}
}

For reconciliation, a scheduled Lambda that runs every hour compares the LastRotatedDate on the AWS secret against the synced_at metadata tag on the OCI secret. If they differ by more than five minutes, it triggers a forced sync.

Security Considerations

A few things to keep in mind when running this in production.

The OCI private key stored in AWS Secrets Manager should be rotated periodically, just like any other credential. Add it to your rotation schedule.

Enable CloudTrail in AWS and OCI Audit logging so every access to both secrets stores is recorded. If something is off with the sync, the audit logs tell you exactly which principal made the change and when.

Use VPC endpoints for Secrets Manager in AWS so the Lambda traffic never crosses the public internet when retrieving credentials.

On the OCI side, enable Vault audit logging to the OCI Logging service so every secret version write is captured.

Wrapping Up

This pipeline solves a real operational problem without requiring a third-party secrets broker. AWS Secrets Manager stays the authoritative source. OCI Vault stays current automatically. The only manual step is the initial deployment.

The pattern extends to other cross-cloud credential types. Database connection strings, API tokens, TLS certificates — any secret that needs to exist on both clouds can follow the same EventBridge to Lambda to OCI Vault flow. Extend the Lambda to support a mapping table of AWS secret ARNs to OCI secret OCIDs and one function handles your entire secrets estate across both providers.

Regards,
Osama

Building a Production Serverless API with OCI API Gateway and OCI Functions

I have seen many teams deploy OCI Functions and call it done. The function works, the test passes, and then they realize there is no authentication, no rate limiting, no proper routing, and no way to version the API. The function URL is just floating there, exposed.

OCI API Gateway is what turns a collection of serverless functions into an actual production API. In this post I will walk through building a complete serverless API stack from scratch using OCI API Gateway, OCI Functions, and Terraform. Everything here is production-oriented: proper IAM, CORS, authentication via JWT, rate limiting, and a deployment pipeline.

Architecture Overview

The stack we are building looks like this:

Client Request → OCI API Gateway → Route Policy (auth + rate limit) → OCI Function → Response

The API Gateway sits in a public subnet and handles all the cross-cutting concerns: TLS termination, JWT validation, CORS headers, and usage plans. The Functions sit in a private subnet with no public exposure. The Gateway invokes them over OCI’s internal network.

Prerequisites

You need the following before starting:

  • OCI CLI configured with a valid profile
  • Terraform 1.5 or later
  • Docker installed locally (for building function images)
  • An OCI tenancy with permissions to manage API Gateway, Functions, IAM, and Networking

Setting Up the Network

Functions should never run in a public subnet. We need a VCN with a public subnet for the Gateway and a private subnet for Functions.

resource "oci_core_vcn" "api_vcn" {
compartment_id = var.compartment_id
cidr_block = "10.0.0.0/16"
display_name = "api-gateway-vcn"
dns_label = "apivcn"
}
resource "oci_core_subnet" "public_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.api_vcn.id
cidr_block = "10.0.1.0/24"
display_name = "gateway-public-subnet"
dns_label = "gatewaypub"
route_table_id = oci_core_route_table.public_rt.id
security_list_ids = [oci_core_security_list.gateway_sl.id]
}
resource "oci_core_subnet" "private_subnet" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.api_vcn.id
cidr_block = "10.0.2.0/24"
display_name = "functions-private-subnet"
dns_label = "funcpriv"
prohibit_public_ip_on_vnic = true
route_table_id = oci_core_route_table.private_rt.id
security_list_ids = [oci_core_security_list.functions_sl.id]
}

The private subnet has prohibit_public_ip_on_vnic = true. This is not optional — it ensures no Function instance can accidentally get a public IP assigned.

For the private subnet to reach OCI services (like Container Registry to pull images), add a Service Gateway:

resource "oci_core_service_gateway" "sgw" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.api_vcn.id
display_name = "functions-service-gateway"
services {
service_id = data.oci_core_services.all_services.services[0].id
}
}
resource "oci_core_route_table" "private_rt" {
compartment_id = var.compartment_id
vcn_id = oci_core_vcn.api_vcn.id
display_name = "private-route-table"
route_rules {
network_entity_id = oci_core_service_gateway.sgw.id
destination = "all-iad-services-in-oracle-services-network"
destination_type = "SERVICE_CIDR_BLOCK"
}
}

IAM: Dynamic Groups and Policies

OCI Functions need explicit permission to be invoked by API Gateway. This is done through Dynamic Groups and policies not hardcoded credentials.

resource "oci_identity_dynamic_group" "api_gateway_dg" {
compartment_id = var.tenancy_ocid
name = "api-gateway-dynamic-group"
description = "Allows API Gateway to invoke Functions"
matching_rule = "ALL {resource.type = 'ApiGateway', resource.compartment.id = '${var.compartment_id}'}"
}
resource "oci_identity_policy" "gateway_invoke_policy" {
compartment_id = var.compartment_id
name = "gateway-invoke-functions-policy"
description = "Grants API Gateway permission to invoke Functions"
statements = [
"Allow dynamic-group api-gateway-dynamic-group to use functions-family in compartment id ${var.compartment_id}"
]
}

Without this policy, the Gateway will return a 500 when it tries to invoke your function, and the error message is not always obvious about the cause.

Building and Pushing the Function

We will write a simple order validation function in Python. Create this directory structure:

order-validator/
├── func.py
├── func.yaml
└── requirements.txt

schema_version: 20180708
name: order-validator
version: 0.0.1
runtime: python3.11
build_image: fnproject/python:3.11-dev
run_image: fnproject/python:3.11
entrypoint: /python/bin/fdk /function/func.py handler
memory: 256

requirements.txt:

fdk>=0.1.57

func.py:

import io
import json
import logging
from fdk import response
def handler(ctx, data: io.BytesIO = None):
logger = logging.getLogger()
try:
body = json.loads(data.getvalue())
except (Exception, ValueError) as ex:
logger.error("Failed to parse request body: " + str(ex))
return response.Response(
ctx,
status_code=400,
response_data=json.dumps({"error": "Invalid JSON in request body"}),
headers={"Content-Type": "application/json"}
)
required_fields = ["order_id", "customer_id", "items", "total_amount"]
missing = [f for f in required_fields if f not in body]
if missing:
return response.Response(
ctx,
status_code=422,
response_data=json.dumps({
"error": "Missing required fields",
"fields": missing
}),
headers={"Content-Type": "application/json"}
)
if not isinstance(body.get("items"), list) or len(body["items"]) == 0:
return response.Response(
ctx,
status_code=422,
response_data=json.dumps({"error": "Order must contain at least one item"}),
headers={"Content-Type": "application/json"}
)
if body["total_amount"] <= 0:
return response.Response(
ctx,
status_code=422,
response_data=json.dumps({"error": "total_amount must be greater than zero"}),
headers={"Content-Type": "application/json"}
)
return response.Response(
ctx,
status_code=200,
response_data=json.dumps({
"status": "valid",
"order_id": body["order_id"],
"item_count": len(body["items"]),
"validated_at": ctx.RequestID()
}),
headers={"Content-Type": "application/json"}
)

Build and push the function image to OCI Container Registry:

# Log in to OCI Container Registry
docker login <region-key>.ocir.io -u '<tenancy-namespace>/<username>'
# Build the function image
fn build --verbose
# Tag and push
docker tag order-validator:0.0.1 <region-key>.ocir.io/<tenancy-namespace>/functions/order-validator:0.0.1
docker push <region-key>.ocir.io/<tenancy-namespace>/functions/order-validator:0.0.1

Deploying the Function with Terraform

resource "oci_functions_application" "orders_app" {
compartment_id = var.compartment_id
display_name = "orders-api"
subnet_ids = [oci_core_subnet.private_subnet.id]
config = {
LOG_LEVEL = "INFO"
ENV = "production"
}
trace_config {
is_enabled = true
domain_id = oci_apm_apm_domain.tracing.id
}
}
resource "oci_functions_function" "order_validator" {
application_id = oci_functions_application.orders_app.id
display_name = "order-validator"
image = "<region-key>.ocir.io/<tenancy-namespace>/functions/order-validator:0.0.1"
memory_in_mbs = 256
timeout_in_seconds = 30
provisioned_concurrency_config {
strategy = "CONSTANT"
count = 2
}
}

The provisioned_concurrency_config block with count = 2 keeps two warm instances running at all times. This eliminates cold starts for your two most frequent concurrent requests — critical for an API that needs consistent latency.

Creating the API Gateway and Deployment

This is where everything comes together. The Gateway deployment defines your routes, authentication, and rate limiting in a single resource:

resource "oci_apigateway_gateway" "orders_gateway" {
compartment_id = var.compartment_id
display_name = "orders-api-gateway"
endpoint_type = "PUBLIC"
subnet_id = oci_core_subnet.public_subnet.id
certificate_id = var.tls_certificate_ocid
}
resource "oci_apigateway_deployment" "orders_deployment" {
compartment_id = var.compartment_id
display_name = "orders-api-v1"
gateway_id = oci_apigateway_gateway.orders_gateway.id
path_prefix = "/v1"
specification {
request_policies {
authentication {
type = "JWT_AUTHENTICATION"
token_header = "Authorization"
token_auth_scheme = "Bearer"
is_anonymous_access_allowed = false
public_keys {
type = "REMOTE_JWKS"
uri = "https://your-identity-provider.com/.well-known/jwks.json"
max_cache_duration_in_hours = 1
}
verify_claims {
key = "iss"
values = ["https://your-identity-provider.com"]
is_required = true
}
verify_claims {
key = "aud"
values = ["orders-api"]
is_required = true
}
}
rate_limiting {
rate_in_requests_per_second = 100
rate_key = "CLIENT_IP"
}
cors {
allowed_origins = ["https://yourdomain.com"]
allowed_methods = ["GET", "POST", "OPTIONS"]
allowed_headers = ["Authorization", "Content-Type"]
max_age_in_seconds = 3600
is_allow_credentials_enabled = true
}
}
routes {
path = "/orders/validate"
methods = ["POST"]
backend {
type = "ORACLE_FUNCTIONS_BACKEND"
function_id = oci_functions_function.order_validator.id
}
request_policies {
body_validation {
required = true
content {
media_type = "application/json"
validation_type = "NONE"
}
}
}
response_policies {
header_transformations {
set_headers {
items {
name = "X-Request-ID"
values = ["${request.headers[x-request-id]}"]
}
items {
name = "Strict-Transport-Security"
values = ["max-age=31536000; includeSubDomains"]
}
}
}
}
}
routes {
path = "/orders/validate"
methods = ["OPTIONS"]
backend {
type = "STOCK_RESPONSE_BACKEND"
status = 204
headers {
name = "Access-Control-Allow-Origin"
value = "https://yourdomain.com"
}
}
}
}
}

A few things worth highlighting in this configuration.

The JWT authentication block uses REMOTE_JWKS, which means the Gateway fetches your identity provider’s public keys and caches them for one hour. It validates the signature, the issuer, and the audience on every request before your Function ever sees the traffic. Your function code does not need to do any token validation at all.

The rate_limiting block uses CLIENT_IP as the rate key, which applies the 100 req/sec limit per caller rather than globally across all callers. Switch this to TOTAL if you want a single shared limit for the entire API.

The OPTIONS route returns a 204 with no backend function invoked. This handles preflight CORS requests without consuming Function compute time.

Testing the Deployment

Once Terraform applies successfully, get your Gateway endpoint:

terraform output gateway_endpoint

Test without a token (should get 401):

curl -X POST https://<gateway-id>.apigateway.<region>.oci.customer-oci.com/v1/orders/validate \
-H "Content-Type: application/json" \
-d '{"order_id": "ORD-001"}'

Test with a valid JWT:

TOKEN=$(curl -s -X POST https://your-idp.com/oauth/token \
-d "grant_type=client_credentials&client_id=...&client_secret=..." | jq -r .access_token)
curl -X POST https://<gateway-id>.apigateway.<region>.oci.customer-oci.com/v1/orders/validate \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"order_id": "ORD-001",
"customer_id": "CUST-999",
"items": [{"sku": "ITEM-A", "qty": 2}],
"total_amount": 49.99
}'

Expected response:

{
"status": "valid",
"order_id": "ORD-001",
"item_count": 1,
"validated_at": "ocid1.apigateway.request..."
}

Enabling Execution Logs

Without logs, debugging a Gateway issue is nearly impossible. Enable execution logs on both the Gateway and the Function application:

resource "oci_logging_log" "gateway_execution_log" {
display_name = "gateway-execution-log"
log_group_id = oci_logging_log_group.api_logs.id
log_type = "SERVICE"
configuration {
source {
category = "execution"
resource = oci_apigateway_deployment.orders_deployment.id
service = "apigateway"
source_type = "OCISERVICE"
}
compartment_id = var.compartment_id
}
retention_duration = 30
is_enabled = true
}

Gateway execution logs include the full request path, JWT claim values, matched route, backend invocation time, and response status for every request. These logs are the first place to look when a request is failing.

Final Thoughts

The combination of OCI API Gateway and OCI Functions gives you a production API stack with almost no infrastructure to manage. The Gateway handles authentication, rate limiting, CORS, and TLS. The Functions handle business logic. Terraform manages the entire configuration as code, so every change is reviewed, versioned, and repeatable.

The pieces that trip most people up are the Dynamic Group IAM policy (the Gateway cannot invoke Functions without it), provisioned concurrency (without it your p99 latency will be terrible on cold paths), and execution logging (without it you are debugging blind).

Get those three right and the rest follows naturally.

Regards, Osama

Building Kubernetes Sentinel: An AI-Powered Cluster Health Dashboard

When you manage Kubernetes clusters at scale, the hardest part is not keeping things running. It is knowing when something is about to break, understanding why it broke, and fixing it before it affects users. Traditional monitoring tools give you metrics and alerts, but they leave the diagnosis entirely up to you. You still have to correlate events, read logs, cross-reference namespaces, and figure out the right kubectl commands to run.

I wanted to change that. So I built Kubernetes Sentinel, an open-source dashboard that not only watches your entire cluster in real time but also uses Claude AI to explain what went wrong and tell you exactly how to fix it.

The Problem with Kubernetes Observability

Anyone who has been on call for a Kubernetes cluster knows the feeling. Your phone goes off at 2am. A pod is crashlooping. You open your terminal, start running kubectl commands, and spend the next twenty minutes piecing together what happened from logs, events, and resource descriptions spread across multiple namespaces.

The tooling has not kept up with the complexity. Prometheus and Grafana are powerful, but they require significant setup and expertise to use effectively. Most teams end up with dashboards full of graphs they never look at and alerts that fire so often they get ignored.

What I wanted was something simpler. A single view of the entire cluster, automatic detection of anything that looks wrong, and an AI that could look at the same data an experienced SRE would look at and tell me what is happening in plain English.

What Kubernetes Sentinel Does

Kubernetes Sentinel is a FastAPI backend that runs either locally or as a pod inside your cluster. It polls the Kubernetes API every 15 seconds across all namespaces, not just one, and stores the current state in memory. A React frontend connects to it over HTTP and receives live updates via Server-Sent Events.

The dashboard gives you four things at once. A health score from 0 to 100 that reflects the overall state of your cluster. A live pod table showing every pod across every namespace with restart counts, phase, and node assignment. An event stream showing everything Kubernetes has logged, filtered and color-coded by severity. And a resources view covering your nodes, deployments, services, and persistent volume claims.

On top of that, the backend runs seven anomaly detection rules continuously. CrashLoopBackOff, OOMKilled, NodeNotReady, FailedMount, BackOff, CPUThrottling, and high restart counts. When any of these fire, an anomaly banner appears at the top of the dashboard immediately.

The AI diagnosis feature is where it gets interesting. When you click Run Diagnosis, the backend assembles the current cluster state into a structured prompt and sends it to Claude. Within seconds you get back a plain-English summary of what is wrong, a root cause explanation, and three kubectl commands you can copy and run immediately to fix it. No more correlating events manually. No more searching Stack Overflow for the right flags.

The Technical Decisions

I made a few deliberate architectural choices that I think are worth explaining.

The backend runs as a single process with one Uvicorn worker. This is intentional. The background polling thread lives inside the same process, so multiple workers would each start their own independent loop and you would end up with redundant API calls and inconsistent state. One process, one source of truth.

Authentication with the Kubernetes API uses the official Python client, which handles both scenarios automatically. When the sentinel runs inside a cluster as a pod, it reads the ServiceAccount token that Kubernetes mounts automatically at a well-known path. When you run it locally for development, it falls back to your kubeconfig. The same code works in both environments without any changes.

The RBAC configuration is strictly read-only. The ClusterRole I wrote grants get, list, and watch on pods, events, nodes, services, persistent volume claims, configmaps, secrets, deployments, statefulsets, daemonsets, and replicasets. Nothing else. The sentinel can observe everything but change nothing. This was a hard requirement for me. A monitoring tool should never have write access to the cluster it is watching.

For the frontend I deliberately chose a single React file with no build step. The dashboard runs as a Claude.ai artifact or drops straight into any React project. There is nothing to compile, no node_modules to install, no webpack config to debug. The entire UI is one file you can read and understand in an afternoon.

I also added a DEV_MODE flag that bypasses the Kubernetes connection entirely and loads realistic mock data instead. This means anyone can clone the repo, set DEV_MODE=true, start the backend, and see the full dashboard working within five minutes even if they have never touched Kubernetes before. It made development much faster and makes the project far more accessible for contributors.

The Stack

The backend is Python 3.12 with FastAPI and the official Kubernetes client library. I used sse-starlette for Server-Sent Events, httpx for calling the Claude API, and Pydantic v2 for data validation. The Docker image is a two-stage build that ends up running as a non-root user.

The frontend is React 18 with no external UI library. All styling is plain inline JavaScript objects, which makes it trivially portable and means there are zero CSS conflicts when you embed it somewhere else.

Kubernetes manifests cover the full production deployment: namespace, ClusterRole, ClusterRoleBinding, ServiceAccount, ConfigMap, Deployment with liveness and readiness probes, and a ClusterIP Service. The Anthropic API key is never stored in any manifest file. It goes into a Kubernetes Secret created directly with kubectl.

What I Learned Building This

The biggest challenge was not the Kubernetes integration or the AI features. It was the import path problem. Claude Code generated all the backend files correctly, but because the server is started from inside the backend directory, every import had to be relative to that directory as the root. Files using from backend.core.x import y worked fine in isolation but crashed immediately when uvicorn tried to load them. Once I understood the issue it was a one-line fix in every file, but it cost me an hour of debugging.

The second thing I learned is that mock data is not optional for a project like this. Without DEV_MODE, you need a running Kubernetes cluster to develop against, which means either paying for cloud infrastructure or running a local cluster with kind. Adding ten lines of mock data to the poller made the development loop dramatically faster and opened the project up to contributors who want to work on the frontend without needing any cluster at all.

The AI diagnosis feature turned out to be far more useful than I expected. I assumed it would be a nice addition but not something I would rely on. After running it against realistic failure scenarios, the quality of the root cause analysis was genuinely impressive. It correctly identified memory limit misconfiguration from OOMKill events, correlated restart back-off with recent image pull failures, and suggested the right sequence of commands to investigate and resolve each issue.

Running It Yourself

The project is open source and available on GitHub. There are three ways to run it.

If you just want to see the dashboard without any cluster setup, clone the repo, copy .env.example to .env, set DEV_MODE=true and your Anthropic API key, then run uvicorn from the backend directory. The whole setup takes under five minutes.

If you have a Kubernetes cluster, set DEV_MODE=false and point it at your kubeconfig. The backend will start polling your real cluster immediately and the dashboard will show live data.

If you want to run it inside your cluster, build the Docker image, push it to your registry, create a Kubernetes Secret with your API key, and apply the manifests with kubectl. The deploy script handles the apply order automatically.

The repository is at https://github.com/OsamaOracle/k8s-sentinel/. Contributions, issues, and feedback are welcome.

Regards
Osama

Building Generative AI Applications with Vector Databases on AWS

A few months ago, I was helping a team that had just integrated an LLM into their product. The use case was straightforward: users ask questions, the LLM answers. They had it running. The demos looked great. Then they went to production.

The model kept confidently making things up. It had no idea about the company’s internal documentation, the latest product specs, or anything that happened after its training cutoff. The team was frustrated. They had the right model, the right infrastructure, but the wrong architecture.

The fix was not fine-tuning. Fine-tuning is expensive, slow, and you have to redo it every time your data changes. The fix was Retrieval Augmented Generation, or RAG. And at the heart of RAG is something called a vector database.

In this article, I will walk you through building a production-grade RAG architecture on AWS. We will cover what vector databases actually are, when to use Aurora pgvector versus OpenSearch versus Amazon Bedrock Knowledge Bases, and how to wire everything together with real code.

What Is a Vector Database and Why Does It Matter

Before writing any infrastructure code, let me explain what problem we are actually solving.

When you work with text, images, or audio in AI systems, the raw data is not what gets compared. Instead, you pass the data through an embedding model, which converts it into a list of numbers called a vector. That vector captures the semantic meaning of the content.

Two sentences that mean the same thing will have vectors that are close to each other in vector space, even if they use completely different words. “The server is down” and “the system is not responding” will be closer to each other than “the server is down” and “I had pasta for lunch.”

A vector database is optimized for one specific operation: given a query vector, find me the N closest vectors in the collection. This is called approximate nearest neighbor search, and it is fundamentally different from SQL WHERE clauses or text search.

In a RAG architecture, the flow looks like this:

  1. You chunk your documents and generate embeddings for each chunk
  2. You store those embeddings in a vector database
  3. When a user asks a question, you generate an embedding for the question
  4. You query the vector database to retrieve the most semantically similar chunks
  5. You pass the question plus those chunks to your LLM as context
  6. The LLM answers based on actual, grounded information

The result is a model that knows your data, stays current as your data changes, and does not hallucinate facts from your knowledge base because the facts are right there in the prompt.

Options on AWS

AWS gives you three serious paths for vector storage, and choosing the wrong one will cost you performance and money.

Amazon Aurora PostgreSQL with pgvector

pgvector is an open source PostgreSQL extension that adds native vector storage and similarity search. If you already run Aurora PostgreSQL, this is often the right starting point.

The extension supports three distance metrics: L2 (Euclidean), inner product, and cosine similarity. For most text embedding use cases, cosine similarity is what you want.

Here is a minimal setup to get you started:

-- Enable the extension on your Aurora instance
CREATE EXTENSION vector;
-- Create a table for your document chunks
CREATE TABLE document_chunks (
id BIGSERIAL PRIMARY KEY,
doc_id TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_url TEXT,
embedding vector(1536), -- 1536 dims for text-embedding-3-small
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- IVFFlat index for approximate nearest neighbor search
-- lists = sqrt(number of rows) is a good starting point
CREATE INDEX ON document_chunks
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
SELECT
chunk_text,
source_url,
1 - (embedding <=> $1::vector) AS similarity_score
FROM document_chunks
ORDER BY embedding <=> $1::vector
LIMIT 5;

The <=> operator computes cosine distance. One minus that gives you similarity.

For production, tune the ivfflat.probes parameter at query time. Higher probes means more accuracy but slower queries. For most use cases, setting it between 10 and 20 is a reasonable balance:

Aurora pgvector is the right choice when your team already knows PostgreSQL, you want to join vector search results with relational data in the same query, or you have an existing Aurora cluster and want to avoid managing another service.

The limitation is scale. Once you push past 10 to 20 million vectors, or you need sub-10ms latency at high concurrency, you will start to feel the ceiling.

Amazon OpenSearch Service with Vector Engine

OpenSearch’s vector engine is built for scale. It uses the HNSW (Hierarchical Navigable Small World) algorithm, which delivers excellent recall and latency even at hundreds of millions of vectors.

Setting up an index for vector search:

PUT /documents
{
"settings": {
"index": {
"knn": true,
"knn.algo_param.ef_search": 512
}
},
"mappings": {
"properties": {
"doc_id": { "type": "keyword" },
"chunk_text": { "type": "text" },
"source_url": { "type": "keyword" },
"embedding": {
"type": "knn_vector",
"dimension": 1536,
"method": {
"name": "hnsw",
"space_type": "cosinesimil",
"engine": "nmslib",
"parameters": {
"ef_construction": 512,
"m": 16
}
}
}
}
}
}

The ef_construction and m parameters control the index build quality. Higher values give better recall but increase memory usage and indexing time. For most production workloads, m=16 and ef_construction=512 is a solid baseline.

Indexing a document:

import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
region = "us-east-1"
service = "es"
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,
region, service, session_token=credentials.token)
client = OpenSearch(
hosts=[{"host": your_opensearch_endpoint, "port": 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
document = {
"doc_id": "product-manual-v3-page-42",
"chunk_text": "The power button is located on the right side of the device...",
"source_url": "s3://your-bucket/manuals/product-v3.pdf",
"embedding": generate_embedding("The power button is located...")
}
client.index(index="documents", body=document)

Querying for semantic similarity:

query = {
"size": 5,
"query": {
"knn": {
"embedding": {
"vector": generate_embedding(user_question),
"k": 5
}
}
},
"_source": ["chunk_text", "source_url"]
}
response = client.search(index="documents", body=query)

OpenSearch also lets you combine vector search with traditional filters, which is something pgvector struggles with at scale:

hybrid_query = {
"size": 5,
"query": {
"bool": {
"must": [
{
"knn": {
"embedding": {
"vector": generate_embedding(user_question),
"k": 20
}
}
}
],
"filter": [
{ "term": { "product_line": "enterprise" } },
{ "range": { "doc_date": { "gte": "2024-01-01" } } }
]
}
}
}

Retrieving 20 candidates via vector search, then filtering down with metadata, is called pre-filtering, and it is critical when your knowledge base spans multiple products, teams, or access tiers.

Amazon Bedrock Knowledge Bases

If you want the fastest path to production and do not want to manage chunking, embedding, or indexing yourself, Bedrock Knowledge Bases handles all of it.

You point it at an S3 bucket. It crawls your documents, chunks them, generates embeddings using your chosen model, and stores them in an OpenSearch Serverless collection. When you query it, it handles the retrieval and optionally the generation too.

resource "aws_bedrockagent_knowledge_base" "product_docs" {
name = "product-documentation-kb"
role_arn = aws_iam_role.bedrock_kb_role.arn
knowledge_base_configuration {
type = "VECTOR"
vector_knowledge_base_configuration {
embedding_model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0"
}
}
storage_configuration {
type = "OPENSEARCH_SERVERLESS"
opensearch_serverless_configuration {
collection_arn = aws_opensearchserverless_collection.kb_vectors.arn
vector_index_name = "bedrock-knowledge-base-default-index"
field_mapping {
vector_field = "bedrock-knowledge-base-default-vector"
text_field = "AMAZON_BEDROCK_TEXT_CHUNK"
metadata_field = "AMAZON_BEDROCK_METADATA"
}
}
}
}
resource "aws_bedrockagent_data_source" "s3_docs" {
knowledge_base_id = aws_bedrockagent_knowledge_base.product_docs.id
name = "s3-product-documentation"
data_source_configuration {
type = "S3"
s3_configuration {
bucket_arn = aws_s3_bucket.documentation.arn
}
}
vector_ingestion_configuration {
chunking_configuration {
chunking_strategy = "SEMANTIC"
semantic_chunking_configuration {
max_token = 300
buffer_size = 0
breakpoint_percentile_threshold = 95
}
}
}
}

Querying it from your application:

import boto3
bedrock_agent = boto3.client("bedrock-agent-runtime", region_name="us-east-1")
response = bedrock_agent.retrieve_and_generate(
input={
"text": user_question
},
retrieveAndGenerateConfiguration={
"type": "KNOWLEDGE_BASE",
"knowledgeBaseConfiguration": {
"knowledgeBaseId": "YOUR_KB_ID",
"modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0",
"retrievalConfiguration": {
"vectorSearchConfiguration": {
"numberOfResults": 5,
"overrideSearchType": "HYBRID"
}
}
}
}
)
answer = response["output"]["text"]
citations = response["citations"]

The HYBRID search type combines vector similarity with keyword search under the hood, which improves recall for queries that contain specific product names, version numbers, or technical terms that embeddings alone sometimes miss.

Chunking Strategy: The Part Everyone Gets Wrong

The quality of your RAG system depends more on how you chunk your documents than on which vector database you choose. I have seen teams spend weeks optimizing their similarity search while their chunking strategy was destroying recall.

A few rules that hold up in practice:

Chunk size matters. Too small and you lose context. Too large and you dilute the semantic signal. For most document types, 300 to 500 tokens with a 50-token overlap between chunks is a reasonable starting point. The overlap ensures that sentences that fall on chunk boundaries are still retrievable.

Chunk by structure when you can. If your documents have headers, sections, or natural breaks, use those as chunk boundaries rather than fixed token counts. A section about “Troubleshooting Network Errors” should stay together rather than getting split at 400 tokens.

Store metadata with every chunk. The chunk text alone is not enough. You need the source document, the section title, the creation date, the product version. This metadata enables the filtering patterns we covered in OpenSearch and prevents your model from citing a three-year-old document when a current one exists.

Test with real queries. The only way to validate your chunking strategy is to run the queries your users will actually ask and check whether the right chunks are being retrieved. Build a small evaluation set early, before you optimize anything else.

Embedding Model Selection

For AWS workloads, you have two main options through Bedrock:

Amazon Titan Text Embeddings V2 produces 1024-dimensional vectors. It is fast, cheap, and fine for general English text. If you are building an internal knowledge base over English documents, this is the right default.

Cohere Embed v3 supports multilingual embeddings and produces 1024-dimensional vectors with better performance on technical and domain-specific text. If your documents cover specialized subject matter legal, medical, engineering Cohere will typically outperform Titan on retrieval quality.

A critical point that is easy to overlook: you must use the same embedding model at indexing time and query time. If you indexed your documents with Titan and query with Cohere, the vectors live in different spaces and your similarity scores will be meaningless. Build this constraint into your infrastructure from day one.

Architecture Summary

For a production RAG system on AWS, here is the architecture that has worked well for teams I have worked with.

Document ingestion: an S3 bucket triggers a Lambda function, or Step Functions for large files. The function chunks the document, generates embeddings via Bedrock, and writes to your vector store with metadata.

Vector storage: Aurora pgvector for under 5 million vectors with heavy relational joins. OpenSearch for everything larger, or when you need metadata filtering at scale. Bedrock Knowledge Bases when you want fully managed infrastructure and your team does not want to own the pipeline.

Query path: API Gateway triggers a Lambda function that embeds the user query, retrieves top-k chunks from the vector store, builds a context-enriched prompt, and calls Claude or another Bedrock model for the final response.

Observability: CloudWatch captures embedding latency, retrieval similarity scores, and end-to-end response time. Set alerts if retrieval quality drops since that is usually a signal that something changed in your document pipeline.

Regards
Osama

Enforcing SLA Compliance with SQL Assertions in Oracle 23ai: A Real-World Use Case

One of the most frustrating things I’ve dealt with as a DBA is cleaning up data that should never have existed in the first place. Orphaned records, overlapping date ranges, business rules violated because some batch job skipped a validation step. We’ve all been there.

The traditional solution was triggers. And if you’ve written cross-table validation triggers in Oracle, you know the pain: mutating table errors (ORA-04091), complex exception handling, scattered logic across multiple trigger bodies, and debugging sessions that make you question your career choices.

Starting with Oracle Database 23ai (release 23.26.1), Oracle introduced SQL Assertions, and they change everything about how we enforce cross-table business rules.

What Are SQL Assertions?

An assertion is a schema-level integrity constraint defined by a boolean expression. If that expression evaluates to false during a transaction, the transaction fails. That’s it. The concept has been part of the SQL standard since SQL-92, but no major database vendor actually implemented it until Oracle did it in 23.26.1.

There are two types of assertion expressions:

Existential expressions use [NOT] EXISTS with a subquery. If the condition is true, the transaction proceeds.

Universal expressions use the new ALL ... SATISFY syntax. This lets you say “for every row matching this query, this condition must hold.” It’s Oracle’s elegant alternative to the awkward double-negation pattern (NOT EXISTS ... WHERE NOT EXISTS ...) that SQL traditionally requires for universal quantification.

The Scenario: SLA Compliance for a Ticketing System

Let me show you a real-world use case that goes beyond toy examples. Imagine you run a support ticketing system for an enterprise. You have service level agreements (SLAs) with your customers, and the database needs to enforce these rules:

  1. Every customer must have an active SLA before they can submit a ticket. No SLA, no support.
  2. Tickets can only be created while the customer’s SLA is active (between start and end dates).
  3. High-priority tickets must be assigned to a senior engineer. You can’t assign a critical production issue to a junior team member.
  4. Every SLA must cover at least one service category. An SLA with no covered services is meaningless.

In a traditional Oracle setup, enforcing these rules would require at least four separate triggers across three tables, careful handling of mutating table errors, and a lot of testing to make sure they don’t interfere with each other.

With assertions, each rule is a single declarative statement.

Building the Schema

sql

DROP TABLE IF EXISTS tickets CASCADE CONSTRAINTS PURGE;
DROP TABLE IF EXISTS sla_services CASCADE CONSTRAINTS PURGE;
DROP TABLE IF EXISTS slas CASCADE CONSTRAINTS PURGE;
DROP TABLE IF EXISTS engineers CASCADE CONSTRAINTS PURGE;
DROP TABLE IF EXISTS customers CASCADE CONSTRAINTS PURGE;
CREATE TABLE customers (
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
name VARCHAR2(200) NOT NULL,
company VARCHAR2(200),
created_at TIMESTAMP DEFAULT SYSTIMESTAMP
);
CREATE TABLE engineers (
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
name VARCHAR2(200) NOT NULL,
seniority VARCHAR2(20) CHECK (
seniority IN ('junior','mid','senior','lead')
),
specialization VARCHAR2(100)
);
CREATE TABLE slas (
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
customer_id NUMBER NOT NULL REFERENCES customers(id),
sla_tier VARCHAR2(20) CHECK (
sla_tier IN ('bronze','silver','gold','platinum')
),
start_date DATE NOT NULL,
end_date DATE NOT NULL,
CONSTRAINT sla_dates_valid CHECK (end_date > start_date)
);
CREATE TABLE sla_services (
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
sla_id NUMBER NOT NULL REFERENCES slas(id),
service_name VARCHAR2(100) NOT NULL
);
CREATE TABLE tickets (
id NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
customer_id NUMBER NOT NULL REFERENCES customers(id),
engineer_id NUMBER REFERENCES engineers(id),
priority VARCHAR2(20) CHECK (
priority IN ('low','medium','high','critical')
),
subject VARCHAR2(500) NOT NULL,
created_at TIMESTAMP DEFAULT SYSTIMESTAMP,
status VARCHAR2(20) DEFAULT 'open' CHECK (
status IN ('open','in_progress','resolved','closed')
)
);

Assertion 1: Customers Need an Active SLA to Submit Tickets

This is the core business rule. No active SLA, no ticket creation.

sql

CREATE ASSERTION ticket_requires_active_sla
CHECK (
ALL (SELECT customer_id, created_at FROM tickets) SATISFY
EXISTS (
SELECT 1 FROM slas
WHERE slas.customer_id = tickets.customer_id
AND tickets.created_at
BETWEEN slas.start_date AND slas.end_date
)
);

Read that in plain English: “For all tickets, there must exist an SLA for that customer where the ticket creation date falls within the SLA period.”

If someone tries to insert a ticket for a customer whose SLA has expired, the database will reject the transaction. No application code needed. No trigger needed. The rule is declarative and self-documenting.

Assertion 2: High-Priority Tickets Need Senior Engineers

This is a cross-table constraint that would be especially painful with triggers because it spans tickets and engineers.

sql

CREATE ASSERTION critical_tickets_need_senior_engineer
CHECK (
NOT EXISTS (
SELECT 1
FROM tickets t
JOIN engineers e ON t.engineer_id = e.id
WHERE t.priority IN ('high', 'critical')
AND e.seniority IN ('junior', 'mid')
)
);

This uses the existential pattern. It looks for any high-priority ticket assigned to a junior or mid-level engineer. If it finds one, the transaction fails. Simple, clear, and impossible to bypass from any application that touches this database.

Assertion 3: Every SLA Must Cover at Least One Service

An SLA without any covered services is a data integrity problem waiting to happen.

sql

CREATE ASSERTION sla_must_have_services
CHECK (
ALL (SELECT id FROM slas) SATISFY
EXISTS (
SELECT 1 FROM sla_services
WHERE sla_services.sla_id = slas.id
)
)
DEFERRABLE INITIALLY DEFERRED;

This one uses DEFERRABLE INITIALLY DEFERRED because of the chicken-and-egg problem: the foreign key on sla_services requires the SLA to exist first, but this assertion requires services to exist when an SLA exists. By deferring validation to commit time, you can insert both the SLA and its services in a single transaction.

Testing It Out

Let’s load some data and see the assertions in action:

sql

-- Insert customers
INSERT INTO customers (name, company)
VALUES ('Ahmad Hassan', 'TechCorp Jordan');
INSERT INTO customers (name, company)
VALUES ('Sara Ali', 'DataFlow ME');
-- Insert engineers
INSERT INTO engineers (name, seniority, specialization)
VALUES ('Omar Khalid', 'senior', 'Database');
INSERT INTO engineers (name, seniority, specialization)
VALUES ('Lina Nasser', 'junior', 'Networking');
-- Insert SLA with services (in one transaction
-- because of deferred assertion)
INSERT INTO slas (customer_id, sla_tier, start_date, end_date)
VALUES (1, 'gold', DATE '2025-01-01', DATE '2026-12-31');
INSERT INTO sla_services (sla_id, service_name)
VALUES (1, 'Database Support');
INSERT INTO sla_services (sla_id, service_name)
VALUES (1, '24/7 Monitoring');
COMMIT; -- Assertion validates here: SLA has services, OK
-- This should succeed: customer has active SLA,
-- senior engineer assigned
INSERT INTO tickets
(customer_id, engineer_id, priority, subject)
VALUES
(1, 1, 'critical', 'Production database performance issue');
COMMIT;

Now let’s try violating the rules:

sql

-- This should FAIL: assigning critical ticket
-- to junior engineer
INSERT INTO tickets
(customer_id, engineer_id, priority, subject)
VALUES
(1, 2, 'critical', 'Server outage');
COMMIT;
-- ERROR: assertion CRITICAL_TICKETS_NEED_SENIOR_ENGINEER violated
-- This should FAIL: customer 2 has no SLA
INSERT INTO tickets
(customer_id, engineer_id, priority, subject)
VALUES
(2, 1, 'low', 'General question');
COMMIT;
-- ERROR: assertion TICKET_REQUIRES_ACTIVE_SLA violated

The database enforces the rules. Every time. Regardless of which application, API, or batch job is inserting the data.

Why This Matters

The traditional approach to these rules would involve:

  • Four or more BEFORE INSERT triggers across multiple tables
  • Careful handling of ORA-04091 mutating table errors (probably using compound triggers or package variables)
  • Testing every combination of insert/update/delete across all tables
  • Documentation that explains what each trigger does and how they interact
  • A maintenance burden that grows with every new business rule

With assertions, each rule is one statement. They live in the data dictionary alongside your other constraints. You can query USER_CONSTRAINTS to see them. They are self-documenting. And Oracle’s internal incremental checking mechanism ensures they perform well because the database only validates the data that actually changed, not the entire table.

Practical Notes

Grant the privilege. CREATE ASSERTION is not included in RESOURCE. Use GRANT DB_DEVELOPER_ROLE TO your_user; or grant it explicitly.

Assertions share the constraint namespace. You cannot have an assertion and a constraint with the same name in the same schema.

Cross-schema assertions need ASSERTION REFERENCES. If your assertion references tables in another schema, you need this object privilege on those tables, and you must use fully qualified table names (synonyms are not supported).

Start with ENABLE NOVALIDATE on existing systems. This lets you add an assertion without checking existing data, which is essential when adding rules to a database that might already contain violations.

Subqueries can nest up to three levels. For most business rules, this is more than enough.

Resources

Thank you

Osama

Building a Customer Management System with Oracle 23ai: Domains, Duality Views, and Annotations

I’ve been exploring the new features in Oracle Database 23ai, and I have to say, the combination of SQL Domains, JSON Relational Duality Views, and Annotations completely changes how I think about schema design. In this post, I’ll walk through building a small customer and order management system that uses all three features together. And the best part? You can run every single example right here on FreeSQL without installing anything.

The Problem

Let’s say we’re building a simple e-commerce backend. We need customer records with validated email addresses and credit card numbers, and we need order records tied to those customers. On the application side, our frontend team wants to consume the data as JSON documents. On the database side, we want clean, normalized relational tables with proper constraints.

In older Oracle versions, you would have to:

  • Repeat CHECK constraints for email validation on every table that stores emails
  • Build complex application-layer ORM logic to convert between relational rows and JSON objects
  • Keep documentation about your schema in external wikis or README files that nobody updates

Oracle 23ai solves all three problems with native features. Let me show you how.

Setting Up the Foundation: SQL Domains

SQL Domains are reusable column-type definitions. Think of them as named templates that bundle a data type, constraints, display formatting, ordering behavior, and documentation into a single schema object. Once you create a domain, any column can reference it and automatically inherit everything.

Here’s what that looks like for email addresses and credit card numbers:

sql

PURGE RECYCLEBIN;
DROP DOMAIN IF EXISTS emails;
DROP DOMAIN IF EXISTS cc;
CREATE DOMAIN emails AS VARCHAR2(100)
CONSTRAINT email_chk CHECK (
REGEXP_LIKE(emails, '^(\S+)\@(\S+)\.(\S+)$')
)
DISPLAY LOWER(emails)
ORDER LOWER(emails)
ANNOTATIONS (
Description 'An email address with a check constraint
for name @ domain dot (.) something'
);
CREATE DOMAIN cc AS VARCHAR2(19)
CONSTRAINT cc_chk CHECK (
REGEXP_LIKE(cc, '^\d+(\d+)*$')
)
ANNOTATIONS (
Description 'Credit card number with a check constraint
no dashes, no spaces!'
);

Notice a few things here. The DISPLAY clause means that whenever someone queries an email column, it will automatically be shown in lowercase. The ORDER clause ensures sorting is also case-insensitive. And the ANNOTATIONS clause embeds documentation directly in the data dictionary. No external docs needed.

Try inserting an invalid email like not-an-email into any column using the emails domain, and the database will reject it automatically. The validation lives in the schema, not in your application code.

Creating the Tables

Now let’s create our customers and orders tables. Notice how the email column simply references the emails domain, and the credit_card column references the cc domain. No need to repeat the CHECK constraints.

sql

DROP TABLE IF EXISTS orders CASCADE CONSTRAINTS PURGE;
DROP TABLE IF EXISTS customers CASCADE CONSTRAINTS PURGE;
CREATE TABLE IF NOT EXISTS orders (
id NUMBER,
product_id NUMBER,
order_date TIMESTAMP,
customer_id NUMBER,
total_value NUMBER(6,2),
order_shipped BOOLEAN,
warranty INTERVAL YEAR TO MONTH
);
CREATE TABLE IF NOT EXISTS customers (
id NUMBER,
first_name VARCHAR2(100),
last_name VARCHAR2(100),
dob DATE,
email emails,
address VARCHAR2(200),
zip VARCHAR2(10),
phone_number VARCHAR2(20),
credit_card cc,
joined_date TIMESTAMP DEFAULT SYSTIMESTAMP,
gold_customer BOOLEAN DEFAULT FALSE,
CONSTRAINT new_customers_pk PRIMARY KEY (id)
);
ALTER TABLE orders ADD (CONSTRAINT orders_pk PRIMARY KEY (id));
ALTER TABLE orders ADD (
CONSTRAINT orders_fk FOREIGN KEY (customer_id)
REFERENCES customers (id)
);

Also worth noting: BOOLEAN is now a native SQL data type in 23ai. No more NUMBER(1) or CHAR(1) workarounds. And INTERVAL YEAR TO MONTH gives us clean warranty period tracking without date math.

Loading Sample Data

Let’s insert a handful of customers and a couple of orders:

sql

INSERT INTO customers
(id, first_name, last_name, dob, email, address,
zip, phone_number, credit_card)
VALUES
(1, 'Alice', 'Brown', DATE '1990-01-01',
'alice.brown@example.com', '123 Maple Street',
'12345', '555-1234', '4111111111110000'),
(3, 'Bob', 'Brown', DATE '1990-01-01',
'email1@example.com', '333 Maple Street',
'12345', '555-5678', '4111111111111111'),
(4, 'Clarice', 'Jones', DATE '1990-01-01',
'email8888@example.com', '222 Bourbon Street',
'12345', '555-7856', '4111111111111110'),
(5, 'David', 'Smith', DATE '1990-01-01',
'email375@example.com', '111 Walnut Street',
'12345', '555-3221', '4111111111111112');
INSERT INTO orders
(id, customer_id, product_id, order_date,
total_value, order_shipped, warranty)
VALUES
(100, 1, 101, SYSTIMESTAMP, 300.00, NULL, NULL),
(101, 4, 101, SYSTIMESTAMP - 30, 129.99, TRUE,
INTERVAL '5' YEAR);
COMMIT;

The Magic Part: JSON Relational Duality Views

Here’s where it gets really interesting. JSON Relational Duality Views let you expose your normalized relational tables as JSON documents. The data stays in the relational tables (normalized, efficient, properly constrained), but applications can read and write it as JSON. Both representations stay perfectly in sync, automatically.

First, a simple duality view for just the customers table:

sql

CREATE OR REPLACE FORCE JSON RELATIONAL DUALITY VIEW
customers_dv AS
customers @insert @update @delete
{
_id : id,
FirstName : first_name,
LastName : last_name,
DateOfBirth : dob,
Email : email,
Address : address,
Zip : zip,
phoneNumber : phone_number,
creditCard : credit_card,
joinedDate : joined_date,
goldStatus : gold_customer
};

Now you can insert data as JSON:

sql

INSERT INTO customers_dv VALUES (
'{"_id": 2, "FirstName": "Jim", "LastName": "Brown",
"Email": "jim.brown@example.com",
"Address": "456 Maple Street", "Zip": 12345}'
);
COMMIT;

That JSON insert automatically populates the underlying relational customers table. The domain validation still applies, so if you try to insert a bad email through the JSON interface, Oracle will reject it.

Nested Duality Views: Customers with Their Orders

Now for the real power. Let’s create a duality view that nests orders inside customer documents:

sql

CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW
customer_orders_dv
ANNOTATIONS (
Description 'JSON Relational Duality View
sourced from CUSTOMERS and ORDERS'
)
AS SELECT JSON {
'_id' : c.ID,
'FirstName' : c.FIRST_NAME,
'LastName' : c.LAST_NAME,
'Address' : c.ADDRESS,
'Zip' : c.ZIP,
'orders' :
[ SELECT JSON {
'OrderID' : o.ID WITH NOUPDATE,
'ProductID' : o.PRODUCT_ID,
'OrderDate' : o.ORDER_DATE,
'TotalValue' : o.TOTAL_VALUE,
'OrderShipped' : o.ORDER_SHIPPED
}
FROM ORDERS o WITH INSERT UPDATE DELETE
WHERE o.CUSTOMER_ID = c.ID
]
}
FROM CUSTOMERS c;

Query it, and you get clean JSON with nested orders:

sql

SELECT * FROM customer_orders_dv o
WHERE o.data."_id" = 1;

You can even add a new order by updating the JSON document directly using JSON_TRANSFORM:

sql

UPDATE customer_orders_dv c
SET c.data = json_transform(
data,
APPEND '$.orders' = JSON {
'OrderID': 123,
'ProductID': 202,
'OrderDate': SYSTIMESTAMP,
'TotalValue': 150.00
}
)
WHERE c.data."_id" = 1;
COMMIT;
SELECT * FROM customer_orders_dv o
WHERE o.data."_id" = 1;

That single JSON update automatically inserted a new row into the relational ORDERS table with the correct foreign key. No ORM. No application-layer mapping. The database handles the translation.

Try It Yourself on FreeSQL

The complete script is available to run on FreeSQL. Click the button below, and you’ll have everything set up: domains, tables, sample data, and both duality views. You can modify the queries, try inserting invalid emails to see domain validation in action, and experiment with the JSON interface.

https://freesql.com/embedded/?layout=vertical&compressed_code=H4sIAAAAAAAAE61YbW%252FiuBb%252Bnl9xLlopsBO4JLy0pRqpKXGnzKbAJmlHnTt7kZu4xdMQIye00x31v1%252FZDpAAYbTSzQdIfI7tx%252Bc8x36S6a33CYGHhvdDF12OxueaFnG2hIgtME2APgL5QdMsBbLANE7PK6xheK5pISc4I2uj6gA4hRfMwznmVt1stxtayJI045gmmXKZhfNnCOckfIY6J0%252Fkx3IW02cCdTWAAfp%252F69%252F8D41vF%252BqvJf9%252B0xsNLaLpMsZvELNXwnP%252FhsZ4RDjATitOEpbhjLIkhbpD0pDTpXgC3c6xAo4iTtIUXmk2B5xjKsB9ZBwSvCBwsV5jxDKotxqQsgXJ5jR50ht7cQjDcgzOSiEIw8r1h6FY%252B7foQ%252F1b9KHxu1xx9SqGnEQ0gxDzCGrJavFAeK1yJQmDCKdzkhriNl3ikKT%252FktgdbzKFwL50USG7MqIpDG1%252FaDsIhpOxH3j2aBz4sFzxJ3J%252BuFu4SjO2%252BEVPrdmEoQoYhgw%252FxAQyBmnGOFHzQoQzrA09ZAdoO0XCsh10dQ0AgEYwvr25RJ4hH5ecRaswm%252B00yy6zSEwajG6QH9g3U2VZY97tkbEMx7MXHK9I3l7vG1ajOFw6p8slieByMnGRPVamV8w5TrI3GI0D5N3ZLtwj24NgAjeTcXCtNY5HYA3nl0HYxvpgHB4pT7OZJO%252Bd7Q2vbU9Vo7LG%252BIgxYg%252Fg2AFST6pS8tKULeuq2XS1Nl3%252FpsviiHnrcs4SMlMULfbKzaEk8kwSOQxV23dGExLtJAwcdGXfugH49%252F5OFp9YHM02scsTsvG%252Fsl0%252FX86WkJCQ102XdLZ8hqk3urG9e%252FgD3UOdRo11ruwoElsBoU8JPJO3Yl1lTA6TM1LmUrPdAHl5znKD7ThQL0ytmg%252FN2Tj%252FB%252F0fn%252BFq4qHRp7HqX%252BByAzx0hTw0HiK%252FyBU1haaNxj7yAkHSSdlsFKhjbIliCFoYigfGmgKGSLhRyq9RTGdDu7PdW%252BQD1E0DdDumIdEN0C85e010Q7IMdPPsrN1sm822KWxYOLUehMcF%252BYEXy5i0QrYQJtPqwA1exgT8jBOS5W3dnrjp9XpN8SDuu%252Bb2arfbbT1nmrjqHQGAPfwKh1ypuQuh0%252FkFhF7%252F5HQXgrhKELoG6MMY8zwcn1lC0mMwTk9PT3eRWJYFl2zFH1hSheXktNc%252FgKUcjp4BuoNfaCQ8%252FQXN5seQdE56e1kxTfiC42SVVeHoWJZ5AIdVwtE3QEerJA%252FJFxrHFC%252BORqV70t3D0oEJfq7C0e0cxNEp4TgxQL%252FiOHkWnkP24xiCs9OTvbR04JqGz4y%252FVaE4Oz05lJVuCcWpAfonwvgTxRJHTBYkORqNk%252F5%252BZixA8aKyWAQr93H0SjjODNCv82B4b%252FhosXS6vf6BhExpUlktlmVZBzD0SxjMtgH6iNNUglAEqa7Yzl7JWiZ4cxaxiCQRr66WTqfTOYDlpIxF7GOfcSgD8onGx%252BvW7Ha63T1AJlxSHs4rWdrt7m1ipmmelnFYBujuKnwTnu5bEh6tWrPf6%252BzVSgc8Er0yFlVuZb1e7wCQszIQsZ3eYC4Dcs9WydMvAmIdqJjhnPDqgun3%252B4dwtPWGdl4%252ByNbCUJxihbPQKKhCoyAFjaLIM8qiztgIufUhpkkqtg0wDTDbplFWIdBpt1vCmqziWP3KMNWlq0jXXp9mRwxmnbXOzgzIuIBAk4zwFxyD3tPhjWAul5irwIkHHpq69hCJU3%252BI4LM%252FGYOHXDsYTca2C86t7Y6Ce7gboS8wvPWDyQ3y%252FJlzB2D7oEHhoL%252BgSUp4BherpdRYFxGJSUa0nzKzQgvLawA0Usm%252BEqJgLBTj2lKQCdLDxSUHGBTEg3RwcEYmj5eUZ%252FPcQUgKaUJSZG6vQS40pNHO9ebWuJYf0vyVLgFKff%252Bmy63uHCvZqSxFpVLQnkMhPXOXgnwpKFGBfe1R1KYb8elnOFula5eSHNXeKwWXyI1kXwp1%252FWdtRqPaACwDaptw1wZQ%252B0wXNQNq6wDXBjUpXESbDJzw%252BU4X%252B6JJeOTREz7dXr8kXoT5K13WBiDr7V29y7LFgmZK%252BOI4IzzBGX0h8Inj5fxPF9K3JMM%252FQGs292l5lJCbgszFa%252FQCco7KF1w1GomlEcfgrHBMsze4o%252BQVUrbiIYngkbMFNJsbtgNOIph4DvJ8vSHFuy9%252BCy96af78M%252F8%252FQPit4Tjtt35Hyb91O0LlrdNBQhcd8m1usL6pKuZtj%252F9A4WF38etrIkYbOZtZy5FYX1O1leZ%252Bg%252BLWWjXmpngGxc33gHcgtuM7%252BcotvYvbc9Xgfv4Svhk838B3%252FN8Lz3%252Fl9%252B%252FnsnzlT7MJGccRXXPNceHf4P%252FpKus%252FpPqajTNFxJlzpw7M%252FwfXK5gudngfuWgYKGg%252FN0e0PqORLja21sjZHtz6htnKdjXy%252FGA2tm9QwWfNauXi2vseOaGVg%252B04HvL9gvkrXSrT19G00Kx4KyybNpAsPbyCTb%252BcoaIja40c%252BDIKrmE8uZ1uP1eU%252FDdcVT2m3sS5HQazYhzKYwumKl8Z2FnFuFueKudgEtjuTMqEqpFzmhYH969H0ylySh3eS09X3uQmTzGwvYHV8iE%252FWFQQwEEuClDZ98s18hAAa214OXLgoyREIQF%252FaVsAcuIt08S33nz8A7t4qPkogLAlvljBR%252FiesmSWcZykj4wv1McpYcoP8%252BkUjR3Qf2utSfAxT%252FcmuwPTEqKylDyrLQRvKUdl%252BbWTE7Mn1Ni71tDU4hW6ljxh4aN5vj3mUhKTMIPfVXUdWB6D1zkR3yZLQ4B5rjWbaDL8H0yub9RQFwAA&code_language=PL_SQL&code_format=false

What I Love About This Approach

Domains eliminate copy-paste constraints. In a real production schema, you might have emails in five different tables (customers, employees, vendors, contacts, users). With domains, the validation regex lives in one place. Change it once, and every column using that domain picks up the update.

Annotations are self-documenting schemas. You can query USER_ANNOTATIONS_USAGE to discover what every domain, table, and column does. No more hunting through Confluence pages or README files to understand what a column means.

Duality Views solve the ORM problem at the database level. Your frontend developers can work with clean JSON documents. Your DBAs can work with normalized relational tables. Both see the same data, and the database keeps them in sync. No impedance mismatch, no complex mapping layer, no stale caches.

The fact that you can now experience all of this directly in your browser through FreeSQL makes it incredibly easy to learn and prototype. Select the 23ai engine, and all these features are available immediately.

Regards
Osama