The Event Hubs post covered getting events in; today is the simplest serious way to compute over them. Azure Stream Analytics runs SQL over streams: no cluster, no JVM tuning, a query that reads like the reporting SQL your team already writes, except the tables never end. For a large class of real time problems, aggregations, threshold alerts, enrichment, stream to lake landing, it is the highest leverage per line of code in the Azure data stack.
Windowing: The Part to Actually Understand
Aggregating an infinite stream requires bounding it in time, and the window types are the vocabulary. Tumbling windows are fixed, non overlapping buckets, events per device per minute, each event in exactly one window. Hopping windows overlap, a five minute average recalculated every minute, events counted in multiple windows, for smooth rolling metrics. Sliding windows emit only when content changes, giving exact rolling computations for alerting. Session windows group by activity gaps, all the events of one user visit bounded by idle timeout, the natural shape for behavioral analytics. And snapshot windows group events sharing the exact same timestamp. Choosing the wrong one produces numbers that are subtly wrong forever, so this paragraph is the one to reread.
SELECT
deviceId,
AVG(temperature) AS avgTemp,
System.Timestamp() AS windowEnd
INTO alerts
FROM telemetry TIMESTAMP BY eventTime
GROUP BY deviceId, TumblingWindow(minute, 5)
HAVING AVG(temperature) > 75;
SELECT t.deviceId, t.temperature, r.siteName, r.threshold
INTO enriched
FROM telemetry t TIMESTAMP BY eventTime
JOIN deviceReference r ON t.deviceId = r.deviceId;
TIMESTAMP BY is the difference between event time (when it happened) and arrival time (when it showed up), and real systems must use event time. That drags in the late arrival and out of order policies: how long to wait for stragglers before sealing a window, and whether to adjust or drop events beyond tolerance. Set them consciously from your source characteristics, because the defaults silently rewrite timestamps on late data, and “why does the 09:05 window disagree with the raw data” is this service’s signature mystery.
Scaling: SUs and Embarrassing Parallelism
Capacity is streaming units, and the scaling cheat code is the embarrassingly parallel topology: input partitions (your Event Hubs partitions), query steps partitioned by the same key via PARTITION BY, and a partitioned output line up so every partition flows independently, and throughput scales linearly with SUs. Break the alignment, a global aggregate, a repartitioning join, and the job funnels through a merge step that caps everything. Watch two metrics religiously: watermark delay, the truthful lag indicator, and SU utilization above 80 percent, either one trending up means scale or redesign. Backpressure shows up as growing input backlog on the Event Hubs side, tying back to the consumer lag discipline from that post.
Where It Fits and Where It Does Not
Reference data joins (the second query above) enrich streams against slowly changing lookups from blob or SQL, refreshed on schedule, covering the overwhelmingly common enrich then route pattern. Outputs land everywhere that matters: the lake in Parquet with custom path patterns, SQL, Cosmos, Event Hubs onward, Power BI for genuinely real time dashboards. Choose Stream Analytics when the logic expresses in SQL plus windows plus reference joins, which is most operational streaming. Reach for Spark Structured Streaming or Flink when you need arbitrary code, huge state, complex event time logic beyond the built in policies, or stream to stream joins at serious scale. And for the edge, the same queries deploy to IoT Edge modules, filtering at the source before bandwidth is spent, which foreshadows the IoT posts coming later in this series.
Cheers
Osama
Leave a comment