What is distributed tracing?

Distributed tracing can be said to have begun with the Dapper paper—introduced by Google in 2010—which laid the foundational groundwork for large-scale distributed systems tracing infrastructure. Interestingly, Ben Sigelman, the founder of Lightstep (which later became ServiceNow Cloud Observability), was instrumental in the creation of Dapper. Following Dapper, Twitter released Zipkin in 2012, the first open-source distributed tracing project. Then in 2015, Uber launched Jaeger, which was itself inspired by Dapper.

In 2016, Sigelman wrote a blog post ("Toward Turnkey Distributed Tracing," which would come to be known as the OpenTracing Manifesto). This pivotal text introduced OpenTracing as a single standard, addressing the lack of standardization within the tracing ecosystem and laying the foundation for OpenTracing to become a project under the Cloud Native Computing Foundation (CNCF) and eventually merge with OpenTelemetry in 2019.

OpenTelemetry version 1.0 was released in 2021, and has since become the de facto standard for tracing, metrics, and logging. From Dapper in 2010 to today's OpenTelemetry capabilities, in little over a decade, distributed tracing has evolved from a single backend system to a widely used end-to-end solution, ultimately paving the way for modern comprehensive observability practices.

Distributed tracing allows organizations to profile and monitor their full range of applications, especially those built using a microservices architecture. This approach provides visibility into how individual services within a distributed system interact with one another, building an accurate picture of individual requests as they flow through the system.

By tracking the journey of requests and measuring how long each part takes, distributed tracing aids in pinpointing performance bottlenecks, latency issues, and potential failures. As such, distributed tracing is a crucial tool for DevOps and IT teams, allowing them to optimize, troubleshoot, and maintain their systems more effectively.

Distributed tracing is built around three core components:

• Span
  A span is a single unit of work carried out by a service within the system, marked with start and end time intervals and possibly including metadata such as logs or tags. Spans are the building blocks of a trace, representing different parts of the workflow. Root spans (also called parent spans) can have their own spans (called child spans), which in turn can also have spans.
• Trace
  A trace is made up of a span or multiple spans which together represent the complete execution path of the service request as it moves through the distributed system. Traces are often visualized as trees, where the root node represents the user's interaction, and the other nodes represent the various microservices involved in processing the request and preparing the response.
• Tags
  Tags are metadata elements attached to spans. These provide context and classification.

The trace/span structure offers a request-centric view—bridging the gaps between independent microservices and providing a unified perspective of the system's performance. With this information, organizations are better prepared to understand and improve the user's experience.

Tracing, logging, and metrics play pivotal roles in observability, but they are not the same concepts. Each serves distinct purposes, and understanding the differences and complementary nature of these concepts is essential for comprehensive system monitoring and debugging:

• Tracing
  Distributed tracing provides a detailed view of requests as they traverse through the components of a distributed system, capturing the flow of a request through assorted services playing a role in performance tuning and troubleshooting. Unlike logging and metrics, distributed tracing focuses on the journey of specific requests, giving a clear picture of the interactions between microservices.
• Logging
  Logging is the practice of recording specific, individual events in a system, such as user actions, system errors, or other activities. Time-stamped logs provide granular information about what happened in the system at a particular moment, often essential for debugging and auditing. While distributed tracing tracks the flow of requests, logging offers a more static snapshot of events, without necessarily showing the relationships between different parts of the system.

Metrics are numerical values that represent the state of a system at a particular point in time or over a time interval, and may include response times, error rates, and system resource utilization. Metrics play a vital role in distributed tracing, offering a quantifiable way to monitor and analyze the performance of various services within a distributed system. These numerical values are extrapolated from traces and logs, providing "at-a-glance" information, or even detailed reporting on specific aspects such as response times, error rates, and system throughput.

By considering trace and log data through the lens of metrics that summarize key performance indicators, organizations can gain a comprehensive understanding of their distributed architecture, allowing for quick diagnostics and actionable insights, and facilitating effective system optimization.

Microservices are a software architectural design where an application is structured as a collection of loosely coupled, independently deployable services. Each microservice focuses on a specific functional area and operates as an individual component within the broader system. This modular approach promotes flexibility, scalability, and can enhance development speed. In the context of distributed tracing, microservices play a significant role as the individual nodes that a request passes through.

As a request travels from one microservice to another, distributed tracing captures the details of these interactions, including the time taken at each step. This information details how the request flows through the numerous services, identifying bottlenecks, latencies, and potential failures.

Understanding how microservices interact within a distributed system can be complex; distributed tracing provides invaluable insights into these interactions, empowering organizations to visualize the paths, monitor system performance, and troubleshoot any problems that may arise to foster a more robust and efficient system architecture.

Distributed tracing has become an indispensable tool for organizations working with distributed systems, particularly in the context of microservices and dynamic architectures. By comprehensively tracking and recording every interaction that a request has with each service, distributed tracing provides crucial insights into monitoring, debugging, and performance optimization. Attributes can be added to traces for further clarification, and aach span is recorded with detailed metadata, including span parent-child relationships, allowing a complete understanding of how requests move through and across services.

As such, more and more organizations are turning to distributed tracing to manage the complexity of their modern application environments. With numerous potential failure points in today's intricate application stacks, pinpointing root causes of issues can be difficult, time-consuming, and potentially fraught with errors. Distributed tracing streamlines this process, facilitating quicker and more accurate identification of problems, thereby directly enhancing a company's ability to provide an excellent user experience.

At the same time, distributed tracing is an effective answer to the problem of cardinality, where data volumes increase to the point where data storage and computing power become difficult to manage.

The benefits of distributed tracing extend to enhancing microservices' performance understanding, fostering quick issue resolution, and boosting customer satisfaction. By providing a detailed view of how each microservice performs, organizations can ensure steady revenue streams while also dedicating more time to strategy and innovation.

The data provided through distributed tracing is crucial, but at the end of the day it is still just data. Without a clear understanding of what the data represents, it cannot positively impact the decision-making process. The true value in the data is the actionable insight that can be derived from the numbers—provided they are recent, relevant, and reliable.

It’s in the intelligent analysis and contextual understanding of this data where organizations can pinpoint issues, identify causes, and implement effective solutions. How does distributed tracing move beyond mere data collection to provide profound insights into various scenarios? Consider the following:

As the modern business IT landscape continues to expand in terms of size and complexity, the benefits of distributed tracing are becoming ever more obvious. ServiceNow Cloud Observability—leveraging the award-winning Now Platform®—sets a new standard for tracing, delivering complete visibility across requests in distributed systems.

Integrate with existing tools. Bridge metrics and tracing to create unified telemetry. Significantly reduce your organization’s MTTR. And, through it all, align pricing with business outcomes, for enhanced value without scaling costs for increased usage.

Cloud Observability is revolutionizing distributed tracing to benefit your business. Contact ServiceNow to learn more!