X-Ray

What is X-Ray

AWS X-Ray is a service that collects data about requests that your application serves, and provides tools you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization.

For any traced request to your application, you can see detailed information not only about the request and response, but also about calls that your application makes to downstream AWS resources, microservices, databases, and HTTP web APIs

X-Ray Architecture

X-ray SDK -> X-Ray Daemon (listening on UDP) -> X-ray API -> X-Ray Console

We can see everything interacting with the API

X-Ray SDK

The X-Ray SDK provides:

  • interceptors to add to your code to trace incoming HTTP requests
  • Client Handlers to instrument AWS SDK clients that your application uses to call other AWS services
  • An HTTP client to use to instrument calls to other internal and external HTTP web services

X-Ray Integration

The X-Ray integrates with the following AWS services:

  • Elastic Load Balancing
  • AWS Lambda
  • Amazon API Gateway
  • Amazon Elastic Compute Cloud
  • AWS Elastic Beanstalk

X-Ray Languages

X-Ray Supported Languages:

  • Java
  • Go
  • Node.js
  • Python
  • Ruby
  • .Net

X-Ray Exam Tips

The X-Ray SDK provides:

  • Interceptors to add to your code to trace incoming HTTP requests
  • Client handlers to instrument AWS SDK clients that your application uses to call other AWS services
  • An HTTP client to use to instrument calls to other internal and external HTTP web services

The X-Ray Integrates with the following AWS Services:

  • Elastic Load Balancing
  • AWS Lambda
  • Amazon API Gateway
  • Amazon Elastic Compute Cloud
  • AWS Elastic Beanstalk

The X-Ray integrates with the following languages

  • Java
  • Go
  • Node.Js
  • Python
  • Ruby
  • .Net

Extras

  • X-Ray is used to provide tracind data and debug your applications.
  • Can also be used cross accounts.
  • X-Ray Sampling
    • X-Ray SDK applies a sampling algorithm to determine which requests get traced however because our application is failing to send data to X-Ray it does not help in determining the cause of failure.
    • By default, records the first request each second and five percent of any additional requests
    • By default, records the first request each second and five percent of any additional requests
  • If you are using fargate, you have no access to anything involving the EC2 instance
    • this means we can't put instances roles or control what is installed on the EC2 instance
  • To view all traces across multiple accounts, In the X-Ray Daemon:
    • Create a role in the target unified account and allow roles in each sub account to assume the role
    • Configure the X-Ray daemon to use an IAM instance role
    • https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-configuration.html
  • For Lambda, the environment variables used to facilitate communication with X-Ray:
    • _X_AMZN_TRACE_ID
      • Contains the tracing header, which includes the sampling decision, trace ID, and parent segment ID
      • If lambda receives a tracing header when your function is invoked, that header will be used to populate this environment variable
      • If a tracing header was not received, Lambda will generate one for you
    • AWS_XRAY_CONTEXT_MISSING
      • The X-Ray SDK uses this variable to determine its behavior in the event that your function tries to record X-Ray data, but a tracing header is not available
      • Lambda sets this value to LOG_ERROR by default

Segment Documents

A trace segment is a JSON representation of a request that your application serves. A trace segment records information about the original request, information about the work that your application does locally, and subsegments with information about downstream calls that your application makes to AWS resources, HTTP APIs, and SQL databases.

  • A Segment document conveys information about a segment to X-Ray
  • A segment can be up to 64 kB and contain a whole segment with subsegments, a fragment of a segment that indicates that a request is in progress, or a single subsegment that is sent separately.
  • You can send segment documents directly to X-Ray by using the PutTraceSegments API

-

  • X-Ray compiles and processes segmnet documents to generate queryable trace summaries and full traces that you can access by using the GetTraceSummaries and BatchGetTraces APIs, respectively.
  • In addition to the segments and subsegments that you send to X-Ray, the service uses information in subsegments to generate inferred segments and adds them to the full trace
  • Inferred segments represent downstream services and resources in the service map
  • X-Ray provides a JSON Schema for segment documents.
    • can download the schema here: xray-segmentdocument-schema-v1.0.0
  • A subset of segment fields are indexed by X-Ray for use with filter expressions
    • I.e., if you set the user field on a segment to a unique identifier, you can search for segments associated with specific users in the X-Ray console or by using the GetTraceSummaries API

You can create subsegments to record calls to AWS services and resources that you make with the AWS SDK, calls to internal or external HTTP web APIs, or SQL database queries. You can also create subsegments to debug or annotate blocks of code in your applications. Subsegments can contain other subsegments, so a custom subsegment that records metadata about an internal function call can contain other custom subsegments and subsegments for downstream calls.

A subsegment records a downstream call from the point of view of the service that calls it. X-Ray uses subsegments to identifiy downstream services that don't send segments and create entires for them on the service graph

A subsegment can be embedded in a full segment document or sent independently. Send subsegments separately to asynchronously trace downstream calls for long-running requests, or to avoid exceeding the maximum segment document size

  • Optional Subsegment Fields:
    • Namespace
      • aws for AWS SDK Calls
      • Remote for other downstream calls
    • http
      • http object with information about an ongoing HTTP call
    • aws
      • aws object with information about the downstream AWS resource that your application called
    • Error, throttle, fault, and cause
      • error fields that indicate an error ocurred and that include information about the exception that caused the error
    • Annotations
      • Annotations object with key-value pairs that you want X-Ray to index for search
      • If you want to get downstream information (not just calls to your application), put the annotations in subsegment section, not the segment document itself.
    • metadata
      • metadata object with any additional data that you want to store in the segment
    • Subsegments
      • array of subsegment objects
    • Precursor_ids
      • array of subsegment IDs that identifies subsegments with the same parent that completed prior to this subsegment
  • AWS X-Ray Segment Documents
  • What is AWS X-Ray

Sampling

  • To ensure efficient tracing and provide a representative sample of the requests that your application serves, the X-Ray SDK applies a sampling algorithm to determine which requests get traced
    • By default, the X-Ray SDK records the first request each second, and five percent of eany additional requests
  • To avoid incurring service charges when you are getting started, the default sampling rate is conservative.
  • You can configure X-Ray to modify the default sampling rule and configure additional rules that apply sampling based on properties of the service or request
  • For Example, you might want to disable sampling and trace all requests for calls that modify state or handle user accounts or transactions
  • For high-volume read-only calls, like background polling, health checks, or connection maintenance, you can sample at a low rate and still get enough data to see any issues that arise

Reservoir size is the target number of traces to record per second before applying the fixed rate. The reservoir applies across all services cumulatively, so you can't use it directly. However, if it is non-zero, you can borrow one trace per second from the reservoir until X-Ray assigns a quota. Before receiving a quota, record the first request each second, and apply the fixed rate to additional requests. The fixed rate is a decimal between 0 and 1.00 (100%)

The default rule will record the first request each second, and five percent of any additional requests. One request each second is referred to as the reservoir, which ensures that at least one trace is recorded each second. The five percent of additional requests is what we refer to as the fixed rate. Both the reservoir and the fixed rate are configurable.

To calculate the total sampled requests per second, you can use this formula:

 = reservoir size + ( (incoming requests per second - reservoir size) * fixed rate)

If you set the reservoir size to 50 and the fixed rate to 10%, then if 100 requests per second match the rule, the total number of requests sampled is 55 per second.

= 50 + ( (100 - 50) * 10%)
= 50 + (50 * .10)
= 50 + 5
= 55

AWS Blog Post - API Gateway and Sampling Rules AWS X-Ray Concepts