Application Insights Deep Dive: Distributed Tracing That Answers Questions

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Application Insights has appeared in a dozen posts as the destination for diagnostics; today it gets the deep treatment, because the difference between telemetry that exists and telemetry that answers questions is instrumentation design, sampling strategy, and query fluency.

OpenTelemetry First

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

The instrumentation guidance consolidated: OpenTelemetry is the standard, the Azure Monitor OpenTelemetry distro is the implementation, and the classic SDKs are the past. OTel gives you vendor neutral instrumentation, automatic capture for the common frameworks (HTTP, database clients, the messaging SDKs from September), and the semantic conventions that make traces comparable across services and languages, with the distro wiring export to Application Insights and the correlation headers propagating W3C trace context end to end. The design work on top: enrich spans with the business dimensions queries will need (tenant, order type, feature flag state from the App Configuration post, because “is the new code path slower” should be a filter, not a deploy), custom metrics for the domain numbers (orders per minute beats requests per second for the 2 AM judgment call), and instrument the boundaries hardest, every queue hop, every external call, since gaps in distributed traces cluster exactly where blame disputes do.

Sampling: The Bill and the Truth

Full fidelity tracing on a busy estate is an ingestion bill the September cost lessons predicted, so sampling is a design decision, not a default to discover. The principles: sample at the head consistently by operation ID so traces stay complete (a sampled out middle service is a mystery, not a saving), keep every failure and every rare operation (fixed rate sampling of errors is deleting the evidence), and tier by traffic class, the checkout path at high fidelity, the health checks at nearly none, with ingestion side transformations from the data collection rules trimming the verbose properties nobody queries. The workspace based resource model puts app telemetry in Log Analytics beside everything else this series centralized, which is what makes the joined queries possible, the trace that ends at a database call joining to the database’s own diagnostics, the request spike correlating with the deployment marker from the pipeline annotation.

The Queries and the Outer Loop

Fluency means a handful of KQL shapes ready at hand: the dependency duration percentiles by target that find the slow downstream, the requests join to dependencies that decomposes an endpoint’s latency budget, the exceptions summarize by problemId that turns error noise into a ranked fix list, and the funnel over custom events that answers whether users complete the flow the feature exists for. The application map draws the estate’s call graph from the trace data, the first place to look when the architecture diagram and reality have drifted, and the failures and performance experiences are the triage front doors before the raw KQL. Close the outer loop with availability tests, the standard URL pings and the TrackAvailability custom tests running your synthetic journeys from regions your users occupy, alerting through the same action groups as everything else, because internal metrics all green while the login page times out from the outside is the classic observability blind spot. Instrumented this way, the estate’s answer to “why was it slow at 14:32” is a trace ID away, which is the entire point of paying the observability bill in the first place.

Cheers
Osama

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.