Skip to content

tracing.adoc

Latest commit

 

History

History
910 lines (727 loc) · 28 KB

tracing.adoc

File metadata and controls

910 lines (727 loc) · 28 KB

Tracing

Overview

Distributed tracing is a critical feature of micro-service based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon MP includes support for distributed tracing through the OpenTracing API. Tracing is integrated with WebServer, gRPC Server, and Security.

<dependency>
    <groupId>io.helidon.microprofile.tracing</groupId>
    <artifactId>helidon-microprofile-tracing</artifactId>
</dependency>

Usage

This section explains a few concepts that you need to understand before you get started with tracing.

  • In the context of this document, a service is synonymous with an application.

  • A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.

  • A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts. The OpenTracing Data Model describes the details at The OpenTracing Semantic Specification. Spans are automatically created by Helidon as needed during execution of the REST request. Additional spans can be added through MP annotation @Traced or through OpenTracing APIs.

Configuration

Enabling and Disabling Tracing

You can configure a custom service name using the tracing.service configuration property. If this property is undefined, name is created from JAX-RS Application name, or Helidon MP is used if no application is defined.

To disable Helidon tracing for WebServer and security:

tracing.components.web-server.enabled=false
tracing.components.security.enabled=false

To disable MP Tracing per specification:

mp.opentracing.server.skip-pattern=.*

Tracing configuration can be defined in application.yaml file.

Tracing configuration example
tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
  components:
    web-server:
      spans:
        - name: "HTTP Request"
          logs:
            - name: "content-write"
              enabled: false

Controlling Tracing Output

For WebServer we have path-based support for configuring tracing, in addition to the configuration described above.

Path configuration can use any path string supported by the WebServer. The configuration itself has the same possibilities as traced configuration described above. The path specific configuration will be merged with global configuration (path is the "newer" configuration, global is the "older")

Renaming top level span using request properties

To have a better overview in the search pane of a tracer, you can customize the top-level span name using configuration.

Example:

Configuration properties
tracing.components.web-server.spans.0.name="HTTP Request"
tracing.components.web-server.spans.0.new-name: "HTTP %1$s %2$s"

This is supported ONLY for the span named "HTTP Request" on component "web-server".

Parameters provided:

  1. Method - HTTP method

  2. Path - path of the request (such as '/greet')

  3. Query - query of the request (may be null)

Examples

The examples in this guide demonstrate how to integrate tracing with Helidon, how to view traces, how to trace across multiple services, and how to integrate tracing with Kubernetes. All examples use Zipkin and traces will be viewed using both the Zipkin API and UI.

Set Up Zipkin

First, you need to run the Zipkin tracer. Helidon will communicate with this tracer at runtime.

Run Zipkin within a docker container, then check the Zipkin server health:
docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin  // (1)

  1. Run the Zipkin docker image named openzipkin/zipkin.

Check the Zipkin server health:
curl http://localhost:9411/health # (1)

  1. Invoke the Zipkin REST API to check the Zipkin server health.

Response body
{
  "status": "UP", // (1)
  "zipkin": {
    "status": "UP",
    "details": {
      "InMemoryStorage{}": {
        "status": "UP"
      }
    }
  }
}

  1. All status fields should be UP.

Enable Tracing in the Helidon Application

Update the pom.xml file and add the following Zipkin dependency to the <dependencies> section (not <dependencyManagement>). This will enable Helidon to use Zipkin at the default host and port, localhost:9411.

Add the following dependency to pom.xml:
<dependency>
  <groupId>io.helidon.tracing</groupId>
  <artifactId>helidon-tracing-zipkin</artifactId>
</dependency>

All spans sent by Helidon to Zipkin need to be associated with a service. Specify the service name below.

Add the following line to META-INF/microprofile-config.properties:
tracing.service=helidon-mp-1
Build the application, skipping unit tests, then run it:
mvn package -DskipTests=true
java -jar target/helidon-quickstart-mp.jar
Run the curl command in a new terminal window and check the response:
curl http://localhost:8080/greet
Response body
{
  "message": "Hello World!"
}

View Tracing Using Zipkin REST API

Because tracing is now enabled, the previous /greet endpoint invocation resulted in a new trace being created. Let’s get the trace data that was generated using the Zipkin API. First, get the service information.

Note
 Helidon automatically enables tracing for JAX-RS resources methods so you don’t need to use annotations with JAX-RS. See MicroProfile OpenTracing for more details.
Run the curl command and check the response:
curl http://localhost:9411/api/v2/services
Response body
["helidon-mp-1"] // (1)

  1. This is the tracing service name specified in META-INF/microprofile-config.properties.

Each span used by a service has a name, which is unique within a trace. If you invoke the /greet endpoint multiple times, you will still get the same set of names.

Invoke the endpoint below and check the response:
curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-mp-1" -H "accept: application/json" # (1)

  1. Get the span names for the helidon-mp-1 service.

Response body
[ // (1)
  "content-read",
  "content-write",
  "get:io.helidon.examples.quickstart.mp.greetresource.getdefaultmessage",
  "security",
  "security:atn",
  "security:atz",
  "security:response"
]

  1. These are the span names. If you invoke the /greet endpoint again, then invoke the /spans endpoint, you will get the same response.

Next, get the contents of the trace as shown below. Notice that each span has a parentId field, except the get:io.helidon.examples.quickstart.mp.greetresource.getdefaultmessage span, which is the root.

Invoke the endpoint below and check the response:
curl -X GET "http://localhost:9411/api/v2/traces?serviceName=helidon-mp-1&limit=1" -H "accept: application/json" # (1)

  1. Get the newest trace only, using the limit=1 query param. There are other query params that let you restrict results to a specific time window.

Response body
[
  [ // (1)
    {
      "traceId": "2e0af8866efdef35",
      "parentId": "2e0af8866efdef35",
      "id": "b5d61690f230fde4",
      "kind": "SERVER",
      "name": "content-read",
      "timestamp": 1568077339998659,
      "duration": 41,
      "localEndpoint": {
        "serviceName": "helidon-mp-1",
        "ipv4": "192.168.1.115"
      },
      "tags": {
        "requested.type": "java.io.InputStream"
      }
    }
  ]
]

  1. The request will return seven spans, one for each name, along with an unnamed JSON node, which has the status.

View Tracing Using Zipkin UI

The tracing output data is verbose and can be difficult to interpret using the REST API, especially since it represents a structure of spans. Zipkin provides a web-based UI at http://localhost:9411/zipkin, where you can see a visual representation of the same data and the relationship between spans within a trace. If you see a Lens UI button at the top center then click on it and it will take you to the specific UI used by this guide.

Click on the UI refresh button (the search icon) as shown in the image below. Notice that you can change the look-back time to restrict the trace list.

Trace Refresh
Figure 1. Trace refresh

The image below shows the trace summary, including start time and duration of each trace. There are two traces, each one generated in response to a curl http://localhost:8080/greet invocation. The oldest trace will have a much longer duration since there is one-time initialization that occurs.

Traces
Figure 2. Tracing list view

Click on a trace and you will see the trace detail page where the spans are listed. You can clearly see the root span and the relationship among all the spans in the trace, along with timing information.

Trace Detail
Figure 3. Trace detail page
Note
 A parent span might not depend on the result of the child. This is called a FollowsFrom reference, see Open Tracing Semantic Spec. Note that the last span that writes the response after the root span ends falls into this category.

You can examine span details by clicking on the span row. Refer to the image below, which shows the security span details, including timing information. You can see times for each space relative to the root span. These rows are annotated with Server Start and Server Finish, as shown in the third column.

Span Details
Figure 4. Span detail page

Enable Tracing on CDI Beans

So far in this tutorial you have used tracing with JAX-RS without needing to annotate. You can enable tracing on other CDI beans, either at the class level or at the method level, as shown by the following examples.

Tracing at the Method Level

To trace at the method level, you just annotate a method with @Traced.

Update the GreetingProvider class; 1) Add a new import and 2) Add the @Traced annotation to the getMessage method:
import org.eclipse.microprofile.opentracing.Traced; // (1)

@Traced  // (2)
String getMessage() {
    return message.get();
}

  1. Import the Traced annotation.

  2. Enable tracing for getMessage.

Build and run the application, then invoke the endpoints and check the response:
curl http://localhost:8080/greet
curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-mp-1" -H "accept: application/json"
Response body
[
  "content-read",
  "content-write",
  "dosomework",
  "get:io.helidon.examples.quickstart.mp.greetresource.getdefaultmessage",
  "io.helidon.examples.quickstart.mp.greetingprovider.getmessage", // (1)
  "security",
  "security:atn",
  "security:atz",
  "security:response"
]

  1. There is a new span name for the getmessage method, since your code called that method during the invocation of /greet.

Click the back button on your browser, then click on the UI refresh button to see the new trace. Select the newest trace in the list to see the trace detail page like the one below. Notice the new span named io.helidon.examples.quickstart.mp.greetingprovider.getmessage.

Trace getmessage
Figure 5. Trace detail page with new span getmessage

Tracing at the Class Level

To trace at the class level, annotate the class with @Traced. This will enable tracing for all class methods, except for the constructor and private methods.

  • Update the GreetingProvider class

  • Add @Traced to the GreetingProvider class

  • Remove @Traced from the getMessage method

@Traced // (1)
@ApplicationScoped
public class GreetingProvider {

    String getMessage() { // (2)
        return message.get();
    }
}

  1. This will enable tracing for all class methods, except for the constructor and methods that are private.

  2. Remove @Traced for the getMessage method.

Build and run the application, then invoke the endpoints and check the response:
curl http://localhost:8080/greet
curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-mp-1" -H "accept: application/json"
Response body
[
  "io.helidon.examples.quickstart.mp.greetingprovider.getmessage" // (1)
]

  1. The service has the same set of span names as above, since getmessage was the only method called in this bean.

Next, invoke HTTP PUT to change the greeting, which will cause setMessage to be called.

Invoke the endpoints and check the response:
curl -i -X PUT -H "Content-Type: application/json" -d '{"greeting": "Hi"}'  http://localhost:8080/greet/greeting # (1)
curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-mp-1" -H "accept: application/json"

  1. Invoke the endpoint to change the greeting.

Response body
[
  "content-read",
  "content-write",
  "get:io.helidon.examples.quickstart.mp.greetresource.getdefaultmessage",
  "io.helidon.examples.quickstart.mp.greetingprovider.getmessage",
  "io.helidon.examples.quickstart.mp.greetingprovider.setmessage", // (1)
  "put:io.helidon.examples.quickstart.mp.greetresource.updategreeting", // (2)
  "security",
  "security:atn",
  "security:atz",
  "security:response"
]

  1. The GreetingProvider.setmessage method was traced since you enabled class level tracing.

  2. The JAX-RS method GreetResource.updategreeting was traced automatically by Helidon.

You can refresh the UI view and drill down the trace to see the new spans.

Note
 Methods invoked directly by your code are not enabled for tracing, even if you explicitly annotate them with @Traced. Tracing only works for methods invoked on CDI beans. See the example below.
Update the GreetingProvider class with the following code:
@ApplicationScoped
public class GreetingProvider {
    private final AtomicReference<String> message = new AtomicReference<>();

    @Inject
    public GreetingProvider(@ConfigProperty(name = "app.greeting") String message) {
        this.message.set(message);
    }

    @Traced // (1)
    String getMessage() {
        return getMessage2();
    }

    @Traced  // (2)
    String getMessage2() {
        return message.get();
    }

    void setMessage(String message) {
        this.message.set(message);
    }
}

  1. The getMessage method will be traced since it is externally invoked by GreetResource.

  2. The getMessage2 method will not be traced, even with the @Traced annotation, since it is called internally by getMessage.

Build and run the application, then invoke the endpoints and check the response:
curl http://localhost:8080/greet
curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-mp-1" -H "accept: application/json"
Response body
[
  "io.helidon.examples.quickstart.mp.greetingprovider.getmessage" // (1)
]

  1. The getMessage method is traced, but getMessage2 is not.

Trace Across Services

Helidon automatically traces across services as long as the services use the same tracer, for example, the same instance of Zipkin. This means a single trace can include spans from multiple services and hosts. OpenTracing uses a SpanContext to propagate tracing information across process boundaries. When you make client API calls, Helidon will internally call OpenTracing APIs to propagate the SpanContext. There is nothing you need to do in your application to make this work.

To demonstrate distributed tracing, you will need to create a second project, where the server listens on port 8081. Create a new root directory to hold this new project, then do the following steps, similar to what you did at the start of this guide:

Create a second service

Run the Maven archetype:
mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-mp \
    -DarchetypeVersion={helidon-version} \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-mp-2 \
    -Dpackage=io.helidon.examples.quickstart.mp
The project will be built and run from the helidon-quickstart-mp directory:
cd helidon-quickstart-mp-2
Add the following dependency to pom.xml:
<dependency>
  <groupId>io.helidon.tracing</groupId>
  <artifactId>helidon-tracing-zipkin</artifactId>
</dependency>
Replace META-INF/microprofile-config.properties with the following:
app.greeting=Hello From MP-2
tracing.service=helidon-mp-2

# Microprofile server properties
server.port=8081
Build the application, skipping unit tests, then run it:
mvn package -DskipTests=true
java -jar target/helidon-quickstart-mp-2.jar
Run the curl command in a new terminal window and check the response (notice the port is 8081) :
curl http://localhost:8081/greet
Response body
{
  "message": "Hello From MP-2 World!"
}

Modify the first service

Once you have validated that the second service is running correctly, you need to modify the original application to call it.

Replace the GreetResource class with the following code:
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import jakarta.enterprise.context.RequestScoped;
import jakarta.inject.Inject;
import jakarta.json.Json;
import jakarta.json.JsonBuilderFactory;
import jakarta.json.JsonObject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.client.WebTarget;
import jakarta.ws.rs.core.MediaType;
import org.glassfish.jersey.server.Uri;

@Path("/greet")
@RequestScoped
public class GreetResource {

  @Uri("http://localhost:8081/greet")
  private WebTarget target; // (1)

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final GreetingProvider greetingProvider;

  @Inject
  public GreetResource(GreetingProvider greetingConfig) {
    this.greetingProvider = greetingConfig;
  }

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public JsonObject getDefaultMessage() {
    return createResponse("World");
  }

  @GET
  @Path("/outbound") // (2)
  public JsonObject outbound() {
    return target.request().accept(MediaType.APPLICATION_JSON_TYPE).get(JsonObject.class);
  }

  private JsonObject createResponse(String who) {
    String msg = String.format("%s %s!", greetingProvider.getMessage(), who);

    return JSON.createObjectBuilder().add("message", msg).build();
  }
}

  1. This is the WebTarget needed to send a request to the second service at port 8081.

  2. This is the new endpoint that will call the second service.

Build and run the application, then invoke the endpoint and check the response:
curl -i http://localhost:8080/greet/outbound # (1)

  1. The request went to the service on 8080, which then invoked the service at 8081 to get the greeting.

Response body
{
  "message": "Hello From MP-2 World!" // (1)
}

  1. Notice the greeting came from the second service.

Refresh the Zipkin UI trace listing page and notice that there is a trace across two services.

Traces
Figure 6. Tracing multiple service list view

Click on the trace with two services to see the detail view.

Traces
Figure 7. Tracing across multiple services detail view

In the image above, you can see that the trace includes spans from two services. You will notice there is a gap before the sixth span, which is a get operation. This is a one-time client initialization delay. Run the /outbound curl command again and look at the new trace to see that the delay no longer exists.

You can now stop your second service, it is no longer used in this guide.

Integration with Kubernetes

The following example demonstrate how to use Zipkin from a Helidon application running in Kubernetes.

Update application.yaml:
tracing:
  host: "zipkin"
Stop the application and build the docker image for your application:
docker build -t helidon-tracing-mp .

Deploy Zipkin into Kubernetes

Create the Kubernetes YAML specification, named zipkin.yaml, with the following contents:
apiVersion: v1
kind: Service
metadata:
  name: zipkin
spec:
  ports:
    - port: 9411
      protocol: TCP
  selector:
    app: zipkin
---
kind: Pod
apiVersion: v1
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  containers:
    - name: zipkin
      image: openzipkin/zipkin
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 9411
Create the Zipkin pod and ClusterIP service:
kubectl apply -f ./zipkin.yaml
Create a Zipkin external server and expose it on port 9142:
kubectl expose pod  zipkin --name=zipkin-external --port=9412 --target-port=9411 --type=LoadBalancer # (1)

  1. Create a service so that you can access the Zipkin UI.

Navigate to http://localhost:9412/zipkin to validate that you can access Zipkin running in Kubernetes. It may take a few seconds before it is ready.

Deploy Your Helidon Application into Kubernetes

Create the Kubernetes YAML specification, named tracing.yaml, with the following contents:
kind: Service
apiVersion: v1
metadata:
  name: helidon-tracing # (1)
  labels:
    app: helidon-tracing
spec:
  type: NodePort
  selector:
    app: helidon-tracing
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-tracing
spec:
  replicas: 1 # (2)
  selector:
    matchLabels:
      app: helidon-tracing
  template:
    metadata:
      labels:
        app: helidon-tracing
        version: v1
    spec:
      containers:
        - name: helidon-tracing
          image: helidon-tracing-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

  1. A service of type NodePort that serves the default routes on port 8080.

  2. A deployment with one replica of a pod.

Create and deploy the application into Kubernetes:
kubectl apply -f ./tracing.yaml

Access Your Application and the Zipkin Trace

Get the application service information:
kubectl get service/helidon-tracing
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-tracing   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s // (1)

  1. A service of type NodePort that serves the default routes on port 31143.

Verify the tracing endpoint using port 31143, your port will likely be different:
curl http://localhost:31143/greet
Response body
{
  "message": "Hello World!"
}

Access the Zipkin UI at http://localhost:9412/zipkin and click on the refresh icon to see the trace that was just created.

Cleanup

You can now delete the Kubernetes resources that were just created during this example.

Delete the Kubernetes resources:
kubectl delete -f ./zipkin.yaml
kubectl delete -f ./tracing.yaml
kubectl delete service zipkin-external
docker rm -f zipkin

Creating Custom Spans

Helidon MP fully supports MicroProfile OpenTracing. You can add custom spans using @Traced annotation on methods of CDI beans.

Note for invoking methods on same class: If you invoke a method on the same class, @Traced annotation would be ignored, as it is not invoked through a CDI proxy and as such cannot be intercepted. To make sure @Traced is honored, use it on JAX-RS resource methods and on CDI bean methods used from other beans.

Trace Propagation Across Services

Automated trace propagation is supported currently only with Jersey client.

Tracing propagation works automatically as long as you run within the scope of Helidon MP and use Helidon components to invoke external services.

Manual handling of traces in Jersey Client

There is an option to provide SpanContext programmatically (such as when writing a command line application that starts the span manually).

You can either configure the span context as the active span, or explicitly define it as client property.

Tracing propagation with Jersey client
import static io.helidon.tracing.jersey.client.ClientTracingFilter.CURRENT_SPAN_CONTEXT_PROPERTY_NAME;
import static io.helidon.tracing.jersey.client.ClientTracingFilter.TRACER_PROPERTY_NAME;

Response response = client.target(serviceEndpoint)
    .request()
    // tracer should be provided unless available as GlobalTracer
    .property(TRACER_PROPERTY_NAME, tracer)
    .property(CURRENT_SPAN_CONTEXT_PROPERTY_NAME, spanContext)
    .get();

Additional Information

Jaeger Tracing Metrics

As the Jaeger Tracing section describes, you can use Jaeger tracing in your Helidon application.