Event Hubs is Azure’s distributed log: millions of events per second, partitioned, replayable, and Kafka protocol compatible. It is the front door for telemetry pipelines, clickstreams, IoT ingestion, and increasingly as a drop in Kafka endpoint for applications you do not want to re-platform. The concepts are simple, but the sizing and consumer design decisions are where pipelines succeed or melt.

Partitions Are Forever, Think First

An event hub is a set of partitions, each an ordered append only log. Ordering exists only within a partition, and your maximum consumer parallelism equals your partition count, because each partition is owned by exactly one consumer per consumer group. On Standard tier the count is fixed at creation, so size for the future: I default to 32 for anything serious. Send events with a partition key (device ID, tenant ID) when per entity ordering matters, and without one when you just want even distribution.

resource "azurerm_eventhub_namespace" "this" {
  name                          = "evhns-telemetry-prod"
  location                      = "westeurope"
  resource_group_name           = azurerm_resource_group.stream.name
  sku                           = "Standard"
  capacity                      = 4
  auto_inflate_enabled          = true
  maximum_throughput_units      = 20
  public_network_access_enabled = false
  minimum_tls_version           = "1.2"
}

resource "azurerm_eventhub" "clicks" {
  name                = "clickstream"
  namespace_id        = azurerm_eventhub_namespace.this.id
  partition_count     = 32
  message_retention   = 3
}

resource "azurerm_eventhub_consumer_group" "enrichment" {
  name                = "cg-enrichment"
  namespace_name      = azurerm_eventhub_namespace.this.name
  eventhub_name       = azurerm_eventhub.clicks.name
  resource_group_name = azurerm_resource_group.stream.name
}

Throughput units are the capacity currency: each buys 1 MB per second or 1000 events per second in, and 2 MB per second out. Auto inflate scales TUs up under load (never down, so review the setting after spikes). Beyond roughly 40 TUs or when you need more egress headroom for multiple consumer groups, move to Premium or Dedicated. Remember every consumer group reads the full stream, so three consumer groups triple your egress requirement.

Consumers, Checkpoints, and Idempotency

Use the EventProcessorClient (or Kafka consumer groups over the Kafka endpoint) and let it balance partition ownership across your instances, storing checkpoints in a blob container. Two rules save you from the classic streaming outages. Checkpoint after processing, not before, and accept that redelivery after a crash means your processing must be idempotent: upserts keyed on event ID, not blind inserts. And never checkpoint per event at high volume, batch your checkpoints every N events or T seconds, because checkpoint writes are blob operations with real latency and cost.

Watch consumer lag as your primary health metric: the difference between the latest sequence number and your checkpoint per partition. Lag that grows monotonically means you need more partitions consumed in parallel, faster processing, or both. Lag on one partition only means a hot partition key, which is a producer side design bug.

Capture: The Free Data Lake Feed

Event Hubs Capture writes every event to Data Lake Gen2 in Avro on a schedule you define, no consumer code at all. It is the cheapest possible bridge from streaming to batch analytics: real time consumers handle the hot path while Synapse, Databricks, or Fabric read the captured files for historical processing. Enable it on any stream you might ever want to replay beyond the retention window, because retention on Standard caps at seven days and the question “can we reprocess last quarter” always arrives eventually.

Security follows the same pattern as the rest of this series: private endpoints, public access disabled, managed identities with the Event Hubs Data Sender and Data Receiver roles instead of shared access signatures wherever the client supports it.

Cheers
Osama