When data lives across Object Storage, Autonomous Database, and MySQL in multiple compartments, you need a central place to discover what exists, understand structure, and document meaning. OCI Data Catalog harvests metadata from data sources on a schedule, builds a searchable catalog of tables and files, and connects them through a business glossary. This post covers setup with Terraform.

Step 1: IAM Policy

resource "oci_identity_policy" "data_catalog_policy" {
  compartment_id = var.compartment_id
  name           = "data-catalog-policy"
  statements = [
    "Allow group DataStewards to manage data-catalog-family in compartment id COMPARTMENT_OCID",
    "Allow group DataEngineers to read data-catalog-family in compartment id COMPARTMENT_OCID",
    "Allow service datacatalog to read object-family in compartment id COMPARTMENT_OCID",
    "Allow service datacatalog to read autonomous-database-family in compartment id COMPARTMENT_OCID"
  ]
}

Step 2: Catalog, Data Assets, and Connections

resource "oci_datacatalog_catalog" "production" {
  compartment_id = var.compartment_id
  display_name   = "production-data-catalog"
  defined_tags   = { "Operations.Environment" = "production", "Operations.ManagedBy" = "terraform" }
}

resource "oci_datacatalog_data_asset" "orders_object_storage" {
  catalog_id   = oci_datacatalog_catalog.production.id
  display_name = "orders-object-storage"
  description  = "Order data files in Object Storage production bucket"
  type_key     = "ORACLEOBJECTSTORAGE"
}

resource "oci_datacatalog_connection" "os_connection" {
  catalog_id     = oci_datacatalog_catalog.production.id
  data_asset_key = oci_datacatalog_data_asset.orders_object_storage.key
  display_name   = "orders-os-connection"
  type_key       = "ORACLEOBJECTSTORAGE"
  is_default     = true
}

resource "oci_datacatalog_data_asset" "orders_adb" {
  catalog_id   = oci_datacatalog_catalog.production.id
  display_name = "orders-autonomous-database"
  description  = "Production Autonomous Database for orders schema"
  type_key     = "ORACLE"
}

resource "oci_datacatalog_connection" "adb_connection" {
  catalog_id     = oci_datacatalog_catalog.production.id
  data_asset_key = oci_datacatalog_data_asset.orders_adb.key
  display_name   = "orders-adb-connection"
  type_key       = "ORACLE"
  is_default     = true
}

Step 3: Harvest Jobs

resource "oci_datacatalog_job" "harvest_os" {
  catalog_id     = oci_datacatalog_catalog.production.id
  display_name   = "harvest-orders-object-storage"
  job_type       = "HARVEST"
  data_asset_key = oci_datacatalog_data_asset.orders_object_storage.key
  connection_key = oci_datacatalog_connection.os_connection.key

  schedule_cron_expression           = "0 2 * * 0"
  schedule_timezone_id               = "UTC"
  is_incremental_harvest             = false
  is_include_unregistered_partitions = true
}

resource "oci_datacatalog_job" "harvest_adb" {
  catalog_id     = oci_datacatalog_catalog.production.id
  display_name   = "harvest-orders-adb"
  job_type       = "HARVEST"
  data_asset_key = oci_datacatalog_data_asset.orders_adb.key
  connection_key = oci_datacatalog_connection.adb_connection.key

  schedule_cron_expression = "0 3 * * *"
  schedule_timezone_id     = "UTC"
  is_incremental_harvest   = true
}

Operational Notes

Metadata harvesting reads schema and structural information, not data values. No actual data leaves your databases or Object Storage. The catalog stores table names, column names, data types, and record counts. Sensitive column names can be masked in the catalog display using custom attribute rules even if the underlying column contains PII.

Use the business glossary to document what your data entities mean in business terms. A table named ord_hdr becomes the canonical entity Order Header in the glossary, linked to the physical table with a description any engineer can understand. This connection from business term to physical asset transforms a technical schema catalog into shared organizational knowledge.

Regards,
Osama

#OCI #OracleCloud #DataCatalog #DataGovernance #Terraform #IaC #TechBlog #Oracle #DataPlatform #Analytics #Metadata #DataDiscovery #OracleCloudInfrastructure #DataEngineering #BusinessGlossary #DataLineage #CloudData #DataManagement #ObjectStorage #AutonomousDatabase