Observability on Azure lives or dies on two things: whether your logs all land somewhere queryable, and whether anyone on the team can actually write the query at 2 AM. The first is architecture, the second is KQL fluency, and this post covers both, plus the part everyone learns the expensive way: log ingestion is often the biggest surprise line on the Azure bill.

Workspace Design

Default to one central Log Analytics workspace per region, in the management subscription, receiving diagnostic settings from everything via the policy driven approach from earlier this month. Cross workspace queries exist, but every workspace boundary is friction during an incident. Split only for hard reasons: data sovereignty, a separate security workspace for Sentinel with different retention and access, or a chargeback model that genuinely requires isolation. Use table level RBAC and resource context access to keep teams seeing their own resources without fragmenting the data.

KQL That Earns Its Keep

The pattern behind ninety percent of incident queries: filter time first, filter hard next, then summarize. Here are four I reach for constantly. Error spike by operation from Application Insights:

requests
| where timestamp > ago(1h)
| summarize total = count(), failures = countif(success == false)
    by name, bin(timestamp, 5m)
| extend failureRate = round(100.0 * failures / total, 2)
| where failureRate > 5
| order by timestamp desc

Latency percentiles, because averages lie:

requests
| where timestamp > ago(24h)
| summarize p50 = percentile(duration, 50),
            p95 = percentile(duration, 95),
            p99 = percentile(duration, 99)
    by name
| order by p99 desc

Finding what changed, joining deployments against error onset:

exceptions
| where timestamp > ago(6h)
| summarize errs = count() by bin(timestamp, 10m), cloud_RoleName
| join kind=leftouter (
    customEvents
    | where name == "DeploymentCompleted"
    | project deployTime = timestamp, cloud_RoleName
) on cloud_RoleName
| where timestamp between (deployTime .. deployTime + 1h)

And the firewall denial hunt against resource specific tables:

AZFWApplicationRule
| where TimeGenerated > ago(30m)
| where Action == "Deny"
| summarize hits = count() by SourceIp, Fqdn
| top 20 by hits

Alerting Philosophy

Page on symptoms, ticket on causes. Users experience failure rate, latency, and availability, so those page. CPU, memory, and queue depth are contributing causes that belong on dashboards and low severity tickets, because paging on causes generates the alert fatigue that trains people to ignore the one that matters. Route everything through action groups attached to alert processing rules, so maintenance windows suppress noise in one place. And test your alerts the way you test code: break something in staging and confirm the page arrives, because an unfired alert rule is a hypothesis, not a safety net.

The Cost Conversation

Ingestion is priced per GB and verbose diagnostics add up brutally. The levers, in order of impact: move high volume, low query tables (debug traces, verbose platform logs) to the Basic table plan, use data collection rule transformations to drop columns and rows you never query before ingestion, buy commitment tiers once you know your steady daily GB, and set retention per table instead of workspace wide, keeping security tables long and chatty app traces short. Review the Usage table monthly, sorted by billable GB per table, and you will find at least one surprise every quarter. I always do.

Cheers
Osama