Azure Data Factory is the orchestration backbone of most Azure data platforms: the thing that moves data on schedule, calls the transformations, and wakes someone when a source system changes its schema on a Friday. It is easy to start with and easy to grow into a swamp of four hundred hand built pipelines. The difference is a few patterns adopted early.

Metadata Driven, Not Pipeline per Table

The cardinal pattern: one parameterized ingestion pipeline driven by a control table, not one pipeline per source table. A Lookup activity reads the control table (source, query or table name, destination path, watermark column, enabled flag), a ForEach fans out over the rows with controlled parallelism, and a parameterized Copy activity inside does the movement, updating the watermark on success. Onboarding source table number two hundred becomes an INSERT into the control table rather than another pipeline to maintain. Incremental loads ride the watermark column (a modified timestamp or ascending key) with the high water mark stored back after each run; where the source supports change data capture, prefer the CDC resources for cleaner delta extraction.

{
  "name": "ForEachSourceTable",
  "type": "ForEach",
  "typeProperties": {
    "items": { "value": "@activity('GetTableList').output.value", "type": "Expression" },
    "batchCount": 8,
    "activities": [
      {
        "name": "CopyIncremental",
        "type": "Copy",
        "typeProperties": {
          "source": {
            "type": "SqlServerSource",
            "sqlReaderQuery": {
              "value": "SELECT * FROM @{item().schemaName}.@{item().tableName} WHERE modified_at > '@{item().watermark}'",
              "type": "Expression"
            }
          },
          "sink": { "type": "ParquetSink" }
        }
      }
    ]
  }
}

Integration Runtimes and the Network Story

Three runtime types decide where the work executes. The Azure IR is the default serverless compute; run it inside a managed VNet so its connections to your stores go over managed private endpoints, consistent with everything this series stands for. The self hosted IR is your agent inside networks Azure cannot reach, on premises SQL Servers behind the ExpressRoute, partner systems, and it deserves production treatment: at least two nodes for HA, sized for the concurrent copy load, patched by routine. SSIS IR exists to lift and shift existing SSIS packages; treat it as a bridge, not a destination. Credentials never live in linked services: managed identity where the connector supports it, Key Vault references everywhere else, exactly as the Key Vault post prescribed.

Triggers and Recovery Semantics

Schedule triggers fire on wall clock time and that is all they promise. Tumbling window triggers are the serious tool for periodic loads: contiguous non overlapping windows, per window state, automatic retry, dependency chaining on other windows, and, critically, backfill, rerun a failed Tuesday without hand crafting parameters. Event triggers fire on blob arrival for the drop file driven feeds. Design every pipeline for rerunnability: idempotent sinks (overwrite the partition, merge on keys), watermark updates only on success, and failure alerts through the monitoring stack rather than someone noticing stale dashboards.

CI/CD Without the Publish Button Fights

Git integration on the dev factory only, and skip the classic publish button workflow in favor of the npm build approach: the ADF utilities package validates and generates the ARM templates from the collaboration branch in your pipeline, which then deploys to test and production factories with environment parameters swapped in. Production factories stay disconnected from Git and receive only pipeline deployments, so no one edits production by hand. Add the pre and post deployment script to stop and restart triggers around deployments, keep linked service definitions parameterized per environment, and pull request review applies to data pipelines exactly as it does to application code, because that is what they are.

Cheers
Osama