AWS X-Ray: Distributed Tracing Across Lambda, ECS, and API Gateway

Debugging distributed systems without distributed tracing is archaeology. You gather logs from each service, correlate timestamps, and try to reconstruct what happened across ten services that each have their own logging format and timezone. By the time you figure out which service caused the problem, you have spent hours on what should take minutes.

AWS X-Ray traces requests as they move through your system, showing you exactly where time was spent, which downstream calls succeeded or failed, and how errors propagate across services. In this article I will walk through instrumenting Lambda, ECS services, and API Gateway, and the X-Ray query patterns that surface performance problems before users report them.

How X-Ray Works

X-Ray uses a sampling model. It records a configurable percentage of requests as traces rather than every request. Each trace has a root segment created by the entry point of the request, typically API Gateway or an ALB, and child subsegments created by each downstream service the request touches. Subsegments can represent downstream HTTP calls, database queries, Lambda invocations, SQS sends, and anything else you instrument.

The trace propagates via an HTTP header, X-Amzn-Trace-Id, that each service passes downstream. AWS SDK calls automatically forward this header. You pass it manually in HTTP calls to services you control that are not instrumented with the X-Ray SDK.

Enabling X-Ray on Lambda

resource "aws_lambda_function" "api" {
  function_name = "api-handler"
  role          = aws_iam_role.lambda.arn
  handler       = "handler.lambda_handler"
  runtime       = "python3.12"
  architectures = ["arm64"]

  tracing_config {
    mode = "Active"
  }

  filename         = "${path.module}/functions/api.zip"
  source_code_hash = filebase64sha256("${path.module}/functions/api.zip")

  environment {
    variables = {
      AWS_XRAY_SDK_ENABLED = "true"
    }
  }
}

resource "aws_iam_role_policy_attachment" "xray" {
  role       = aws_iam_role.lambda.name
  policy_arn = "arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess"
}

tracing_config mode = Active means Lambda samples every request regardless of whether the caller requested tracing. PassThrough mode samples only when the incoming request already has an X-Ray trace header. Use Active for all Lambda functions so you have complete trace coverage across your entire request path, not just the portions that happen to originate from an instrumented entry point.

Instrumenting Python with the X-Ray SDK

from aws_xray_sdk.core import xray_recorder, patch_all
from aws_xray_sdk.core import xray_recorder
import boto3
import requests

patch_all()

dynamodb = boto3.resource("dynamodb")
table    = dynamodb.Table("orders")

def lambda_handler(event, context):
    order_id = event["pathParameters"]["orderId"]

    with xray_recorder.in_subsegment("fetch-order") as subsegment:
        subsegment.put_annotation("order_id", order_id)
        order = table.get_item(Key={"PK": f"ORDER#{order_id}"})["Item"]

    with xray_recorder.in_subsegment("enrich-with-inventory") as subsegment:
        response = requests.get(
            f"https://inventory.internal/api/stock/{order['productId']}",
            headers={"X-Amzn-Trace-Id": xray_recorder.current_trace_id()},
            timeout=2
        )
        subsegment.put_annotation("product_id", order["productId"])
        inventory = response.json()

    return {
        "statusCode": 200,
        "body": json.dumps({"order": order, "inventory": inventory})
    }

patch_all() automatically instruments all AWS SDK calls so DynamoDB, SQS, S3, and any other boto3 call automatically appears as a subsegment in the trace without any additional code. Manual subsegments with in_subsegment let you add custom annotations to specific business operations. Annotations are indexed and searchable in the X-Ray console and via the GetTraceSummaries API.

Sampling Rules

resource "aws_xray_sampling_rule" "health_checks" {
  rule_name      = "health-check-low-sampling"
  priority       = 1
  version        = 1
  reservoir_size = 0
  fixed_rate     = 0.01
  url_path       = "/health"
  host           = "*"
  http_method    = "GET"
  service_type   = "*"
  service_name   = "*"
  resource_arn   = "*"
}

resource "aws_xray_sampling_rule" "critical_paths" {
  rule_name      = "payment-full-sampling"
  priority       = 2
  version        = 1
  reservoir_size = 10
  fixed_rate     = 1.0
  url_path       = "/api/payments/*"
  host           = "*"
  http_method    = "*"
  service_type   = "*"
  service_name   = "*"
  resource_arn   = "*"
}

Health check endpoints are called hundreds of times per minute from load balancer health checkers. Sampling them at 1 percent keeps your trace data clean without losing visibility into actual application traffic. Payment endpoints are sampled at 100 percent because every payment trace has high business value and you need complete coverage for troubleshooting payment failures.

Analytics and Querying

X-Ray Analytics lets you filter traces by annotations, response time, error status, and service. The most useful queries for production troubleshooting are finding traces with specific error types, identifying the slowest traces in a time window, and finding all traces for a specific customer or order ID that you annotated in your code.

Use GetTraceSummaries with a filter expression to programmatically surface problematic traces during an incident rather than clicking through the console:

import boto3
from datetime import datetime, timedelta

xray = boto3.client("xray", region_name="us-east-1")

def find_slow_traces(service_name: str, min_response_ms: int = 2000):
    end_time   = datetime.utcnow()
    start_time = end_time - timedelta(hours=1)

    response = xray.get_trace_summaries(
        StartTime      = start_time,
        EndTime        = end_time,
        FilterExpression = f'service("{service_name}") AND responsetime > {min_response_ms / 1000}',
        Sampling       = False
    )

    return [
        {
            "trace_id":     trace["Id"],
            "response_ms":  int(trace["ResponseTime"] * 1000),
            "has_error":    trace.get("HasError", False),
            "http_status":  trace.get("Http", {}).get("HttpStatus")
        }
        for trace in response["TraceSummaries"]
    ]

Closing Thoughts

X-Ray changes how you investigate distributed system problems. Instead of gathering logs from multiple services and mentally assembling a timeline, you open a trace and see the entire request as a waterfall diagram showing exactly where each millisecond was spent. The service map shows you which services are affecting which other services under real production load.

Enable Active tracing on all Lambda functions. Use patch_all in Python and the equivalent auto-instrumentation in other runtimes. Add annotations for high-cardinality business identifiers like order IDs and customer IDs. Set sampling rules to sample 100 percent of high-value paths and reduce sampling on health checks and other high-frequency low-value paths. The investment is minimal and the diagnostic value during incidents is significant.

Enjoy the cloud.

Osama


#AWS #XRay #DistributedTracing #Observability #CloudArchitecture #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudComputing #Lambda #ECS #APIGateway #TechBlog #CloudInfrastructure #PerformanceEngineering #SRE #Monitoring #BackendEngineering #DevOps

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.