Add telemetry filter helper feature so developer code can influence automatic span creation (helidon-io#9552)

* Add telemetry filter helper feature so developer code can influence whether incoming or outgoing spans are automatically created

* Add doc describing filter helpers; also fix some Javadoc warnings in related files

* Use CDI instead of Java service loading to locate helpers

Author: tjquinno

docs/src/main/asciidoc/includes/attributes.adoc

@@ -225,6 +225,7 @@ endif::[]
 :scheduling-javadoc-base-url: {javadoc-base-url}/io.helidon.microprofile.scheduling
 :security-integration-jersey-base-url: {javadoc-base-url}/io.helidon.security.integration.jersey
 :security-integration-webserver-base-url: {javadoc-base-url}/io.helidon.webserver.security
+:telemetry-javadoc-base-url: {javadoc-base-url}/io.helidon.microprofile.telemetry
 :tracing-javadoc-base-url: {javadoc-base-url}/io.helidon.tracing
 
 :webclient-javadoc-base-url: {javadoc-base-url}/io.helidon.webclient

docs/src/main/asciidoc/mp/telemetry.adoc

@@ -204,6 +204,40 @@ include::{sourcedir}/mp/TelemetrySnippets.java[tag=snippet_6, indent=0]
 
 include::{rootdir}/includes/tracing/common-callbacks.adoc[tags=defs;detailed,leveloffset=+1]
 
+=== Controlling Automatic Span Creation
+By default, Helidon MP Telemetry creates a new child span for each incoming REST request and for each outgoing REST client request. You can selectively control if Helidon creates these automatic spans on a request-by-request basis by adding a very small amount of code to your project.
+
+==== Controlling Automatic Spans for Incoming REST Requests
+To selectively suppress child span creation for incoming REST requests implement the link:{telemetry-javadoc-base-url}/io/helidon/microprofile/telemetry/spi/HelidonTelemetryContainerFilterHelper.html[HelidonTelemetryContainerFilterHelper interface].
+
+When Helidon receives an incoming REST request it invokes the `shouldStartSpan` method on each such implementation, passing the link:{jakarta-jaxrs-javadoc-url}/jakarta.ws.rs/jakarta/ws/rs/container/containerrequestcontext[Jakarta REST container request context] for the request. If at least one implementation returns `false` then Helidon suppresses the automatic child span. If all implementations return `true` then Helidon creates the automatic child span.
+
+The following example shows how to allow automatic spans in the Helidon greet example app for requests for the default greeting but not for the personalized greeting or the `PUT` request to change the greeting message (because the update path ends with `greeting` not `greet`).
+
+Your implementation of `HelidonTelemetryContainerFilterHelper` must have a CDI bean-defining annotation. The example shows `@ApplicationScoped`.
+
+.Example container helper for the Helidon MP Greeting app
+[source,java]
+----
+include::{sourcedir}/mp/TelemetrySnippets.java[tag=snippet_11, indent=0]
+----
+
+
+==== Controlling Automatic Spans for Outgoing REST Client Requests
+To selectively suppress child span creation for outgoing REST client requests implement the link:{telemetry-javadoc-base-url}/io/helidon/microprofile/telemetry/spi/HelidonTelemetryClientFilterHelper.html[HelidonTelemetryClientFilterHelper interface].
+
+When your application sends an outgoing REST client request Helidon invokes the `shouldStartSpan` method on each such implementation, passing the link:{jakarta-jaxrs-javadoc-url}/jakarta.ws.rs/jakarta/ws/rs/client/clientrequestcontext[Jakarta REST client request context] for the request. If at least one implementation returns `false` then Helidon suppresses the automatic child span. If all implementations return `true` then Helidon creates the automatic child span.
+
+The following example shows how to allow automatic spans in an app that invokes the Helidon greet example app. The example permits automatic child spans for outgoing requests for the default greeting but not for the personalized greeting or the `PUT` request to change the greeting message (because the update path ends with `greeting` not `greet`).
+
+Your implementation of `HelidonTelemetryClientFilterHelper` must have a CDI bean-defining annotation. The example shows `@ApplicationScoped`.
+
+.Example Client Helper for the Helidon MP Greeting App
+[source,java]
+----
+include::{sourcedir}/mp/TelemetrySnippets.java[tag=snippet_12, indent=0]
+----
+
 == Configuration
 
 IMPORTANT: MicroProfile Telemetry is not activated by default. To activate this feature, you need to specify the configuration `otel.sdk.disabled=false` in one of the MicroProfile Config or other config sources.

docs/src/main/java/io/helidon/docs/mp/TelemetrySnippets.java

@@ -15,6 +15,9 @@
  */
 package io.helidon.docs.mp;
 
+import io.helidon.microprofile.telemetry.spi.HelidonTelemetryClientFilterHelper;
+import io.helidon.microprofile.telemetry.spi.HelidonTelemetryContainerFilterHelper;
+
 import io.opentelemetry.api.baggage.Baggage;
 import io.opentelemetry.api.trace.Span;
 import io.opentelemetry.api.trace.SpanKind;
@@ -28,7 +31,9 @@
 import jakarta.ws.rs.GET;
 import jakarta.ws.rs.Path;
 import jakarta.ws.rs.Produces;
+import jakarta.ws.rs.client.ClientRequestContext;
 import jakarta.ws.rs.client.WebTarget;
+import jakarta.ws.rs.container.ContainerRequestContext;
 import jakarta.ws.rs.core.MediaType;
 import jakarta.ws.rs.core.Response;
 import org.glassfish.jersey.server.Uri;
@@ -225,4 +230,35 @@ public String getSecondaryMessage() {
         // end::snippet_10[]
     }
 
+    class FilterHelperSnippets_11_to_12 {
+
+        // tag::snippet_11[]
+        @ApplicationScoped
+        public class CustomRestRequestFilterHelper implements HelidonTelemetryContainerFilterHelper {
+
+            @Override
+            public boolean shouldStartSpan(ContainerRequestContext containerRequestContext) {
+
+                // Allows automatic spans for incoming requests for the default greeting but not for
+                // personalized greetings or the PUT request to update the greeting message.
+                return containerRequestContext.getUriInfo().getPath().endsWith("greet");
+            }
+        }
+        // end::snippet_11[]
+
+        // tag::snippet_12[]
+        @ApplicationScoped
+        public class CustomRestClientRequestFilterHelper implements HelidonTelemetryClientFilterHelper {
+
+            @Override
+            public boolean shouldStartSpan(ClientRequestContext clientRequestContext) {
+
+                // Allows automatic spans for outgoing requests for the default greeting but not for
+                // personalized greetings or the PUT request to update the greeting message.
+                return clientRequestContext.getUri().getPath().endsWith("greet");
+            }
+        }
+        // end::snippet_12[]
+    }
+
 }

docs/src/main/java/io/helidon/docs/mp/restclient/RestclientMetricsSnippets.java

@@ -13,7 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package io.helidon.docs.mp;
+package io.helidon.docs.mp.restclient;
 
 import java.net.URI;
 
@@ -33,7 +33,7 @@
 import org.eclipse.microprofile.metrics.annotation.Timed;
 import org.eclipse.microprofile.rest.client.RestClientBuilder;
 
-import static io.helidon.docs.mp.RestclientMetricsSnippets.Snippet1.GreetRestClient;
+import static io.helidon.docs.mp.restclient.RestclientMetricsSnippets.Snippet1.GreetRestClient;
 
 @SuppressWarnings("ALL")
 class RestclientMetricsSnippets {

docs/src/main/java/io/helidon/docs/mp/restclient/RestclientSnippets.java

@@ -13,7 +13,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package io.helidon.docs.mp;
+package io.helidon.docs.mp.restclient;
 
 import java.io.IOException;
 import java.net.URI;

microprofile/telemetry/src/main/java/io/helidon/microprofile/telemetry/HelidonTelemetryClientFilter.java

@@ -17,15 +17,19 @@
 
 import java.util.List;
 import java.util.Optional;
+import java.util.ServiceLoader;
 import java.util.Set;
 
+import io.helidon.common.HelidonServiceLoader;
+import io.helidon.microprofile.telemetry.spi.HelidonTelemetryClientFilterHelper;
 import io.helidon.tracing.HeaderConsumer;
 import io.helidon.tracing.HeaderProvider;
 import io.helidon.tracing.Scope;
 import io.helidon.tracing.Span;
 
 import io.opentelemetry.api.baggage.Baggage;
 import io.opentelemetry.context.Context;
+import jakarta.enterprise.inject.Instance;
 import jakarta.inject.Inject;
 import jakarta.ws.rs.client.ClientRequestContext;
 import jakarta.ws.rs.client.ClientRequestFilter;
@@ -55,17 +59,32 @@ class HelidonTelemetryClientFilter implements ClientRequestFilter, ClientRespons
             Response.Status.Family.CLIENT_ERROR,
             Response.Status.Family.SERVER_ERROR);
 
+    private static final String HELPER_START_SPAN_PROPERTY = HelidonTelemetryClientFilterHelper.class.getName() + ".startSpan";
+
     private final io.helidon.tracing.Tracer helidonTracer;
 
+    private final List<HelidonTelemetryClientFilterHelper> helpers;
+
     @Inject
-    HelidonTelemetryClientFilter(io.helidon.tracing.Tracer helidonTracer) {
+    HelidonTelemetryClientFilter(io.helidon.tracing.Tracer helidonTracer,
+                                 Instance<HelidonTelemetryClientFilterHelper> helpersInstance) {
         this.helidonTracer = helidonTracer;
+        helpers = helpersInstance.stream().toList();
     }
 
-
     @Override
     public void filter(ClientRequestContext clientRequestContext) {
 
+        boolean startSpan = helpers.stream().allMatch(h -> h.shouldStartSpan(clientRequestContext));
+        clientRequestContext.setProperty(HELPER_START_SPAN_PROPERTY, startSpan);
+        if (!startSpan) {
+            if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
+                LOGGER.log(System.Logger.Level.TRACE,
+                           "Client filter helper(s) voted to not start a span for " + clientRequestContext.getUri());
+            }
+            return;
+        }
+
         if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
             LOGGER.log(System.Logger.Level.TRACE, "Starting Span in a Client Request");
         }
@@ -99,10 +118,14 @@ public void filter(ClientRequestContext clientRequestContext) {
                              new RequestContextHeaderInjector(clientRequestContext.getHeaders()));
     }
 
-
     @Override
     public void filter(ClientRequestContext clientRequestContext, ClientResponseContext clientResponseContext) {
 
+        Boolean startSpanObj = (Boolean) clientRequestContext.getProperty(HELPER_START_SPAN_PROPERTY);
+        if (startSpanObj != null && !startSpanObj) {
+            return;
+        }
+
         if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
             LOGGER.log(System.Logger.Level.TRACE, "Closing Span in a Client Response");
         }
@@ -128,6 +151,10 @@ public void filter(ClientRequestContext clientRequestContext, ClientResponseCont
         clientRequestContext.removeProperty(SPAN_SCOPE);
     }
 
+    private static List<HelidonTelemetryClientFilterHelper> helpers() {
+        return HelidonServiceLoader.create(ServiceLoader.load(HelidonTelemetryClientFilterHelper.class)).asList();
+    }
+
     private static class RequestContextHeaderInjector implements HeaderConsumer {
 
         private final MultivaluedMap<String, Object> requestHeaders;

microprofile/telemetry/src/main/java/io/helidon/microprofile/telemetry/HelidonTelemetryContainerFilter.java

@@ -23,6 +23,7 @@
 
 import io.helidon.common.context.Contexts;
 import io.helidon.config.mp.MpConfig;
+import io.helidon.microprofile.telemetry.spi.HelidonTelemetryContainerFilterHelper;
 import io.helidon.tracing.Scope;
 import io.helidon.tracing.Span;
 import io.helidon.tracing.SpanContext;
@@ -32,6 +33,7 @@
 import io.opentelemetry.api.baggage.BaggageEntryMetadata;
 import io.opentelemetry.context.Context;
 import io.opentelemetry.semconv.trace.attributes.SemanticAttributes;
+import jakarta.enterprise.inject.Instance;
 import jakarta.inject.Inject;
 import jakarta.ws.rs.ApplicationPath;
 import jakarta.ws.rs.container.ContainerRequestContext;
@@ -63,6 +65,8 @@ class HelidonTelemetryContainerFilter implements ContainerRequestFilter, Contain
 
     private static final String SPAN_NAME_FULL_URL = "telemetry.span.full.url";
 
+    private static final String HELPER_START_SPAN_PROPERTY = HelidonTelemetryContainerFilterHelper.class + ".startSpan";
+
     @Deprecated(forRemoval = true, since = "4.1")
     static final String SPAN_NAME_INCLUDES_METHOD = "telemetry.span.name-includes-method";
 
@@ -81,12 +85,15 @@ class HelidonTelemetryContainerFilter implements ContainerRequestFilter, Contain
      */
     private final boolean restSpanNameIncludesMethod;
 
+    private final List<HelidonTelemetryContainerFilterHelper> helpers;
+
     @jakarta.ws.rs.core.Context
     private ResourceInfo resourceInfo;
 
     @Inject
     HelidonTelemetryContainerFilter(io.helidon.tracing.Tracer helidonTracer,
-                                    org.eclipse.microprofile.config.Config mpConfig) {
+                                    org.eclipse.microprofile.config.Config mpConfig,
+                                    Instance<HelidonTelemetryContainerFilterHelper> helpersInstance) {
         this.helidonTracer = helidonTracer;
         isAgentPresent = HelidonOpenTelemetry.AgentDetector.isAgentPresent(MpConfig.toHelidonConfig(mpConfig));
 
@@ -106,6 +113,8 @@ class HelidonTelemetryContainerFilter implements ContainerRequestFilter, Contain
                                SPAN_NAME_INCLUDES_METHOD));
         }
         // end of code to remove in 5.x.
+
+        helpers = helpersInstance.stream().toList();
     }
 
     @Override
@@ -115,6 +124,16 @@ public void filter(ContainerRequestContext requestContext) {
             return;
         }
 
+        boolean startSpan = helpers.stream().allMatch(h -> h.shouldStartSpan(requestContext));
+        requestContext.setProperty(HELPER_START_SPAN_PROPERTY, startSpan);
+        if (!startSpan) {
+            if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
+                LOGGER.log(System.Logger.Level.TRACE,
+                           "Container filter helper(s) voted to not start a span for " + requestContext);
+            }
+            return;
+        }
+
         if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
             LOGGER.log(System.Logger.Level.TRACE, "Starting Span in a Container Request");
         }
@@ -151,6 +170,11 @@ public void filter(final ContainerRequestContext request, final ContainerRespons
             return;
         }
 
+        Boolean startSpanObj = (Boolean) request.getProperty(HELPER_START_SPAN_PROPERTY);
+        if (startSpanObj != null && !startSpanObj) {
+            return;
+        }
+
         if (LOGGER.isLoggable(System.Logger.Level.TRACE)) {
             LOGGER.log(System.Logger.Level.TRACE, "Closing Span in a Container Request");
         }
@@ -179,6 +203,10 @@ public void filter(final ContainerRequestContext request, final ContainerRespons
         }
     }
 
+//    private static List<HelidonTelemetryContainerFilterHelper> helpers() {
+//        return HelidonServiceLoader.create(ServiceLoader.load(HelidonTelemetryContainerFilterHelper.class)).asList();
+//    }
+
     private String spanName(ContainerRequestContext requestContext, String route) {
         // @Deprecated(forRemoval = true) In 5.x remove the option of excluding the HTTP method from the REST span name.
         // Starting in 5.x this method should be:

microprofile/telemetry/src/main/java/io/helidon/microprofile/telemetry/TelemetryAutoDiscoverable.java

@@ -22,6 +22,12 @@
  */
 public class TelemetryAutoDiscoverable implements AutoDiscoverable {
 
+    /**
+     * For service loading.
+     */
+    public TelemetryAutoDiscoverable() {
+    }
+
     /**
      * Used to register {@code HelidonTelemetryContainerFilter} and {@code HelidonTelemetryClientFilter}
      * filters.

microprofile/telemetry/src/main/java/io/helidon/microprofile/telemetry/TelemetryCdiExtension.java

@@ -36,6 +36,12 @@ public class TelemetryCdiExtension implements Extension {
 
     private static final System.Logger LOGGER = System.getLogger(TelemetryCdiExtension.class.getName());
 
+    /**
+     * For service loading.
+     */
+    public TelemetryCdiExtension() {
+    }
+
     /**
      * Add {@code HelidonWithSpan} annotation with interceptor.
      *

microprofile/telemetry/src/main/java/io/helidon/microprofile/telemetry/spi/HelidonTelemetryClientFilterHelper.java

@@ -0,0 +1,33 @@
+/*
+ * Copyright (c) 2024 Oracle and/or its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.helidon.microprofile.telemetry.spi;
+
+import jakarta.ws.rs.client.ClientRequestContext;
+
+/**
+ * Service-loaded type applied while the Helidon-provided client filter executes.
+ */
+public interface HelidonTelemetryClientFilterHelper {
+
+    /**
+     * Invoked to see if this helper votes to create and start a new span for the outgoing client request reflected
+     * in the provided client request context.
+     *
+     * @param clientRequestContext the {@link jakarta.ws.rs.client.ClientRequestContext} passed to the filter
+     * @return true to vote to start a span; false to vote not to start a span
+     */
+    boolean shouldStartSpan(ClientRequestContext clientRequestContext);
+}