Closing the AKS week with the layer that makes the rest operable. The Kubernetes ecosystem standardized on Prometheus metrics and Grafana dashboards long ago; the historical tax was running that stack, storage sizing, HA for the monitoring that monitors HA. Azure’s managed versions remove the tax while keeping the standards, and the design work shifts to what to collect and when to wake someone.

The Stack and Its Division of Labor

Azure Monitor managed Prometheus scrapes your cluster (platform targets on by default, application pods via the standard annotations or pod monitor CRDs) into a Monitor workspace with long retention and PromQL compatibility, so every community dashboard and alert rule you have ever used works unchanged. Azure Managed Grafana fronts it with identity from Entra (viewer access via groups, no local Grafana accounts, consistent with the whole series) and dashboards provisioned as code from the GitOps repo, because a dashboard built by hand in an incident is a dashboard the next incident cannot find. Container Insights remains the logs half, stdout, stderr, and events into Log Analytics, and its cost lesson from the September Monitor post applies doubled here: enable the cost preset or custom data collection settings, drop the namespaces you never query, push chatty dev clusters to Basic table plans, and question every log line at ingestion because cluster logs are the fastest growing bill in most estates. The division that keeps costs sane: metrics for everything continuous (cheap, aggregated), logs for the discrete and diagnostic, traces via the mesh or SDK instrumentation from earlier posts for the request path.

What to Watch: Four Layers

Cluster layer: node readiness, pending pods age (the single best early warning, pods pending five minutes means capacity, taints, or storage zone problems from this week’s posts), and API server latency. Capacity layer: requests versus allocatable per pool (the Karpenter post made requests load bearing, watch their honesty), OOM kill counts, and CPU throttling, which silently ruins latency on limit strangled workloads. Workload layer: restarts and crash loops, HPA at max replicas (an autoscaler pinned at its ceiling is an outage rehearsing), and PDB violations during disruptions. Application layer: the golden signals per service, which the mesh emits uniformly if you adopted it, or the SDKs emit if you did not, aggregated to the SLOs the WAF review documented. Wire the layers into Grafana as one drill down path, symptom at the top, cause four clicks down, because the 3 AM responder navigates dashboards in exactly one direction.

groups:
  - name: capacity
    rules:
      - alert: PodsPendingTooLong
        expr: sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Pods pending >10m in {{ $labels.namespace }}"

      - alert: ContainerCpuThrottlingHigh
        expr: sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (namespace, pod)
              / sum(rate(container_cpu_cfs_periods_total[5m])) by (namespace, pod) > 0.25
        for: 15m
        labels:
          severity: ticket

Alerting That Respects Sleep

Prometheus rule groups deploy through ARM and the GitOps repo, and the philosophy from the Monitor post governs them: page on user symptoms (SLO burn, ingress error rate, pending pods that block deployments), ticket on causes (throttling, restarts, capacity trends), dashboard the rest. The cluster specific temptation to resist is paging on every restart and every node event, Kubernetes heals constantly and the noise trains the exact numbness the Sentinel post warned about; a restart is a page only when the crash loop persists past the platform’s ability to self repair. Review alert volume monthly with the same discipline as cost: every page in the last month either led to action or gets tuned away. The AKS week ends where the series keeps ending, with the machinery declarative, the humans woken rarely and for reasons, and the evidence in dashboards anyone can find.

Cheers
Osama