///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2019 Oracle and/or its affiliates. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

= Metrics SE Guide
:description: Helidon metrics
:keywords: helidon, metrics, microprofile, guide

This guide describes how to create a sample Helidon SE project
that can be used to run some basic examples using both built-in and custom metrics.

== What you need

[width=50%,role="flex, sm7"]
|===
|About 30 minutes
|<<about/03_prerequisites.adoc,Helidon Prerequisites>>
|https://github.com/helm/helm[Helm]
|===

=== Create a sample Helidon SE project

Use the Helidon SE Maven archetype to create a simple project that can be used for the examples in this guide.

[source,bash,subs="attributes+"]
.Run the Maven archetype
----
mvn archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-se \
    -DarchetypeVersion={helidon-version} \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-se \
    -Dpackage=io.helidon.examples.quickstart.se
----

=== Using the built-in metrics

Helidon provides three scopes of metrics: base, vendor, and application. Here are the metric endpoints:

1. `/metrics/base` - Base metrics data as specified by the MicroProfile Metrics specification.
2. `/metrics/vendor` - Helidon-specific metrics data.
3. `/metrics/application` - Application-specific metrics data.

NOTE: The `/metrics` endpoint will return data for all scopes.

The following example will demonstrate how to use the built-in metrics.  All examples are executed
from the root directory of your project (helidon-quickstart-se).  The generated source code is
already configured for both metrics and health-checks, but the following example removes health-checks.


[source,xml]
.Notice that the metrics dependency is already in the project's pom.xml file:
----
<dependency>
    <groupId>io.helidon.metrics</groupId>
    <artifactId>helidon-metrics</artifactId>
</dependency>
----


[source,java]
.Replace the `Main.createRouting` method with the following code:
----
    private static Routing createRouting(Config config) {

      GreetService greetService = new GreetService(config);

      return Routing.builder()
          .register(JsonSupport.create())
          .register(MetricsSupport.create()) // <1>
          .register("/greet", greetService)
          .build();
    }
----
<1> Register the built-in base and vendor metrics.

[source,bash]
.Build the application, skipping unit tests, then run it:
----
mvn package -DskipTests=true
java -jar target/helidon-quickstart-se.jar
----

NOTE: Metrics can be returned in either text format (the default), or JSON.  The text format uses Prometheus Text Format,
see https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details.

[source,bash]
.Verify the metrics endpoint in a new terminal window:
----
curl http://localhost:8080/metrics
----

[source,text]
.Text response:
----
# TYPE base:classloader_current_loaded_class_count counter
# HELP base:classloader_current_loaded_class_count Displays the number of classes that are currently loaded in the Java virtual machine.
base:classloader_current_loaded_class_count 7511
# TYPE base:classloader_total_loaded_class_count counter
# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
base:classloader_total_loaded_class_count 7512
...
----

You can get the same data in JSON format.

[source,bash]
.Verify the metrics endpoint with an HTTP accept header:
----
curl -H "Accept: application/json"  http://localhost:8080/metrics
----

[source,json]
.JSON response:
----
{
  "base": {
    "classloader.currentLoadedClass.count": 7534,
    "classloader.totalLoadedClass.count": 7538,
    "classloader.totalUnloadedClass.count": 1,
    "cpu.availableProcessors": 4,
    "cpu.systemLoadAverage": 2.83349609375,
    "gc.PS MarkSweep.count": 2,
    "gc.PS MarkSweep.time": 77,
    "gc.PS Scavenge.count": 5,
    "gc.PS Scavenge.time": 37,
    "jvm.uptime": 727588,
    "memory.committedHeap": 284164096,
    "memory.maxHeap": 3817865216,
    "memory.usedHeap": 53283088,
    "thread.count": 44,
    "thread.daemon.count": 35,
    "thread.max.count": 44
  },
  "vendor": {
    "grpc.requests.count": 0,
    "grpc.requests.meter": {
      "count": 0,
      "meanRate": 0.0,
      "oneMinRate": 0.0,
      "fiveMinRate": 0.0,
      "fifteenMinRate": 0.0
    },
    "requests.count": 6,
    "requests.meter": {
      "count": 6,
      "meanRate": 0.008275992296704147,
      "oneMinRate": 0.01576418632772332,
      "fiveMinRate": 0.006695060022357365,
      "fifteenMinRate": 0.0036382699664488415
    }
  }
}
----

You can get a single metric by specifying the name in the URL path.

[source,bash]
.Get the Helidon `grpc.requests.meter` metric:
----
curl -H "Accept: application/json"  http://localhost:8080/metrics/vendor/grpc.requests.meter
----

[source,json]
.JSON response:
----
{
  "grpc.requests.meter": {
    "count": 0,
    "meanRate": 0.0,
    "oneMinRate": 0.0,
    "fiveMinRate": 0.0,
    "fifteenMinRate": 0.0
  }
}
----

NOTE: You cannot get the individual fields of a metric. For example, you cannot target http://localhost:8080/metrics/vendor/grpc.requests.meter.count.

=== Metrics metadata

Each metric has associated metadata that describes:

1. name: The name of the metric.
2. units: The unit of the metric such as time (seconds, millisecond), size (bytes, megabytes), etc.
3. type: The type of metric: `Counter`, `Timer`, `Meter`, `Histogram`, or `Gauge`.

You can get the metadata for any scope, such as `/metrics/base`, as shown below:

[source,bash]
.Get the metrics metadata using HTTP OPTIONS method:
----
 curl -X OPTIONS -H "Accept: application/json"  http://localhost:8080/metrics/base
----

[source,json]
.JSON response (truncated):
----
{
  "classloader.currentLoadedClass.count": {
    "unit": "none",
    "type": "counter",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine.",
    "displayName": "Current Loaded Class Count"
  },
...
  "jvm.uptime": {
    "unit": "milliseconds",
    "type": "gauge",
    "description": "Displays the start time of the Java virtual machine in milliseconds. This attribute displays the approximate time when the Java virtual machine started.",
    "displayName": "JVM Uptime"
  },
...
  "memory.usedHeap": {
    "unit": "bytes",
    "type": "gauge",
    "description": "Displays the amount of used heap memory in bytes.",
    "displayName": "Used Heap Memory"
  }
}
----


=== Application-specific metrics data

This section demonstrates how to use application-specific metrics and integrate them with Helidon.
It is the application's responsibility to create and update the metrics at runtime.  The application has
complete control over when and how each metric is used. For example, an application may use the
same counter for multiple methods, or one counter per method.  Helidon maintains an application
`MetricRegistry` which is used to manage all of the application metrics.
Helidon returns these metrics in response to a `/metrics/application` REST request.

In all of these examples, the scope and lifetime of the metric is at the application-level.
Each metric, except `Gauge`, is updated in response to a REST request and the contents of the
metric is cumulative.

==== Counter metric

The `Counter` metric is a monotonically increasing or decreasing number. The following example
will demonstrate how to use a `Counter` to track the number of times the `/cards` endpoint is called.

[source,java]
.Create a new class named `GreetingCards` with the following code:
----
package io.helidon.examples.quickstart.se;

import io.helidon.metrics.RegistryFactory;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import java.util.Collections;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import org.eclipse.microprofile.metrics.Counter;  // <1>
import org.eclipse.microprofile.metrics.MetricRegistry;

public class GreetingCards implements Service {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final Counter cardCounter;   // <2>

  GreetingCards() {
    RegistryFactory metricsRegistry = RegistryFactory.getInstance();
    MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
    cardCounter = appRegistry.counter("cardCount");  // <3>
  }

  @Override
  public void update(Routing.Rules rules) {
    rules.get("/", this::getDefaultMessageHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {
    cardCounter.inc();   // <4>
    sendResponse(response, "Here are some cards ...");
  }

  private void sendResponse(ServerResponse response, String msg) {
    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }
}
----
<1> Import metrics classes.
<2> Declare a `Counter` member variable.
<3> Create and register the `Counter` metric in the `MetricRegistry`.  This `Counter` will exist for the lifetime of
the application.
<4> Increment the count.

[source,java]
.Update the `Main.createRouting` method as follows:
----
    private static Routing createRouting(Config config) {

        MetricsSupport metrics = MetricsSupport.create();
        GreetService greetService = new GreetService(config);

        return Routing.builder()
                .register(JsonSupport.create())
                .register(metrics)
                .register("/greet", greetService)
                .register("/cards", new GreetingCards()) //<1>
            .build();
    }
----
<1> Add the `GreetingCards` service to the `Routing.builder`.  Helidon will route any REST requests with
the `/cards` root path to the `GreetingCards` service.

[source,bash]
.Build and run the application, then invoke the endpoints below:
----
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response:
----
{
  "cardCount": 1 // <1>
}
----
<1> The count value is one since the method was called once.


==== Meter metric

The `Meter` metric is used to measure throughput, the number of times an event occurs within a certain time period.
When a `Meter` object is created, its internal clock starts running.  That clock is used to calculate the various rates
stored this metric.  The `Meter` also includes the `count` field from the `Counter` metric.  When you mark an event,
the count is incremented.

The following example marks an event each time the `/cards` endpoint is called.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.se;

import io.helidon.metrics.RegistryFactory;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import java.util.Collections;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import org.eclipse.microprofile.metrics.Meter;
import org.eclipse.microprofile.metrics.MetricRegistry;

public class GreetingCards implements Service {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final Meter cardMeter; // <2>

  GreetingCards() {
    RegistryFactory metricsRegistry = RegistryFactory.getInstance();
    MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
    cardMeter = appRegistry.meter("cardMeter"); // <3>
  }

  @Override
  public void update(Routing.Rules rules) {
    rules.get("/", this::getDefaultMessageHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {
    cardMeter.mark(); // <4>
    sendResponse(response, "Here are some cards ...");
  }

  private void sendResponse(ServerResponse response, String msg) {
    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }
}
----
<1> Import metrics classes.
<2> Declare a `Meter` member variable.
<3> Create and register the `Meter` metric in the `MetricRegistry`.
<4> Mark the occurrence of an event.

TIP:  Note: you can specify a count parameter such as `mark(100)` to mark multiple events.

[source,bash]
.Build and run the application, then invoke the endpoints below:
----
curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response:
----
{
  "cardMeter": { // <1>
    "count": 3, // <2>
    "meanRate": 0.17566568722974535,
    "oneMinRate": 0.04413761384322548,
    "fiveMinRate": 0.009753212003766951,
    "fifteenMinRate": 0.0033056752265846544
  }
}
----
<1> The `Meter` metric has a set of fields to show various rates, along with the count.
<2> The `/cards` endpoint was called three times.


==== Timer metric

The `Timer` metric aggregates durations, provides timing statistics, and includes throughput statistics
using an internal `Meter` metric. The `Timer` measures duration in nanoseconds. In the following example,
a `Timer` metric is used to measure the duration of a method's execution.  Whenever the REST `/cards`
endpoint is called, the `Timer` will be updated with additional timing information.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.se;

import io.helidon.metrics.RegistryFactory;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import java.util.Collections;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import org.eclipse.microprofile.metrics.MetricRegistry; // <1>
import org.eclipse.microprofile.metrics.Timer;

public class GreetingCards implements Service {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final Timer cardTimer; // <2>

  GreetingCards() {
    RegistryFactory metricsRegistry = RegistryFactory.getInstance();
    MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
    cardTimer = appRegistry.timer("cardTimer"); // <3>
  }

  @Override
  public void update(Routing.Rules rules) {
    rules.get("/", this::getDefaultMessageHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {
    Timer.Context timerContext = cardTimer.time(); // <4>
    sendResponse(response, "Here are some cards ...");
    response.whenSent().thenAccept(res -> timerContext.stop()); // <5>
  }

  private void sendResponse(ServerResponse response, String msg) {
    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }
}
----
<1> Import metrics classes.
<2> Declare a `Timer` member variable.
<3> Create and register the `Timer` metric in the `MetricRegistry`.
<4> Start the timer.
<5> Stop the timer.


[source,bash]
.Build and run the application, then invoke the endpoints below:
----
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----


[source,json]
.JSON response:
----
{
  "cardTimer": {
    "count": 1,
    "meanRate": 0.03843465264149663, // <1>
    "oneMinRate": 0.014712537947741825,
    "fiveMinRate": 0.0032510706679223173,
    "fifteenMinRate": 0.0011018917421948848,
    "min": 40876527,  // <2>
    "max": 40876527,
    "mean": 40876527,
    "stddev": 0.0,
    "p50": 40876527,
    "p75": 40876527,
    "p95": 40876527,
    "p98": 40876527,
    "p99": 40876527,
    "p999": 40876527
  }
}
----
<1> These are the same fields used by `Meter`.
<2> These are the `Timer` fields that measure the duration of the `getDefaultMessageHandler` method. Some of these values
will change each time you invoke the `/cards` endpoint.


==== Histogram metric

The `Histogram` metric calculates the distribution of a set of values within ranges.  This metric does
not relate to time at all.  The following example will record a set of random numbers in a `Histogram` metric when
the `/cards` endpoint is invoked.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.se;

import io.helidon.metrics.RegistryFactory;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import java.util.Collections;
import java.util.Random;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import org.eclipse.microprofile.metrics.Histogram;
import org.eclipse.microprofile.metrics.MetricRegistry;

public class GreetingCards implements Service {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final Histogram cardHistogram; // <2>

  GreetingCards() {
    RegistryFactory metricsRegistry = RegistryFactory.getInstance();
    MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
    cardHistogram = appRegistry.histogram("cardHistogram"); // <3>
  }

  @Override
  public void update(Routing.Rules rules) {
    rules.get("/", this::getDefaultMessageHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {

    Random r = new Random();
    for (int i = 0; i < 1000; i++) {  // <4>
      cardHistogram.update(1 + r.nextInt(25)); // <5>
    }
    sendResponse(response, "Here are some cards ...");
  }

  private void sendResponse(ServerResponse response, String msg) {
    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }
}

----
<1> Import metrics classes.
<2> Declare a `Histogram` member variable.
<3> Create and register the `Histogram` metric in the `MetricRegistry`.
<5> Loop, loading the histogram with numbers.
<4> Update the `Histogram` metric with a random number.


[source,bash]
.Build and run the application, then invoke the endpoints below:
----
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----


[source,json]
.JSON response:
----
{
  "cardHistogram": { //<1>
    "count": 1000,
    "min": 1,
    "max": 25,
    "mean": 12.743999999999915,
    "stddev": 7.308793607702962,
    "p50": 13.0,
    "p75": 19.0,
    "p95": 24.0,
    "p98": 25.0,
    "p99": 25.0,
    "p999": 25.0
  }
}
----
<1> This is the histogram data.  Some of these values will change each time you invoke the `/cards` endpoint.


==== Gauge metric

The `Gauge` metric measures a discreet value at a point in time, such as a temperature.  The metric is not normally
tied to a REST endpoint, rather it should be registered during application startup.  When the `/metrics/application` endpoint
is invoked, Helidon will call the `getValue` method of each registered `Gauge`.  The following example demonstrates
how a `Gauge` is used to get the current temperature.

[source,java]
.Add new imports to `Main.java` and replace the `Main.createRouting` method with the following code:
----

import io.helidon.metrics.RegistryFactory;
import java.util.Random;
import org.eclipse.microprofile.metrics.Gauge;
import org.eclipse.microprofile.metrics.MetricRegistry;

...

    private static Routing createRouting(Config config) {

      MetricsSupport metrics = MetricsSupport.create();

      RegistryFactory metricsRegistry = RegistryFactory.getInstance();
      MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
      appRegistry.register("temperature", (Gauge<Integer>)() ->  new Random().nextInt(100)); //<1>

      GreetService greetService = new GreetService(config);
      return Routing.builder()
          .register(JsonSupport.create())
          .register(metrics)                  // Metrics at "/metrics"
          .register("/greet", greetService)
          .register("/cards", new GreetingCards())
          .build();
    }
----
<1> Register the `Gauge`, providing a lambda function that will return a random temperature.

[source,java]
.Update the `GreetingCards` class with the following code to use the `Counter` metric which will simplify the JSON output:
----
package io.helidon.examples.quickstart.se;

import io.helidon.metrics.RegistryFactory;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import java.util.Collections;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import org.eclipse.microprofile.metrics.Counter;
import org.eclipse.microprofile.metrics.MetricRegistry;

public class GreetingCards implements Service {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());
  private final Counter cardCounter;

  GreetingCards() {
    RegistryFactory metricsRegistry = RegistryFactory.getInstance();
    MetricRegistry appRegistry = metricsRegistry.getRegistry(MetricRegistry.Type.APPLICATION);
    cardCounter = appRegistry.counter("cardCount");
  }

  @Override
  public void update(Routing.Rules rules) {
    rules.get("/", this::getDefaultMessageHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {
    cardCounter.inc();   // <4>
    sendResponse(response, "Here are some cards ...");
  }

  private void sendResponse(ServerResponse response, String msg) {
    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }
}
----

[source,bash]
.Build and run the application, then invoke the endpoints below:
----
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response from `/metrics/application`:
----
{
  "cardCount": 1,
  "temperature": 11 // <1>
}
----
<1> The current temperature is returned.  Invoke the `/metrics/application` endpoint again and you should get a different value.


=== Integration with Kubernetes and Prometheus

The following example shows how to integrate the Helidon SE application with Kubernetes.

[source,bash]
.Stop the application and build the docker image:
----
docker build -t helidon-metrics-se .
----

[source,yaml]
.Create the Kubernetes YAML specification, named `metrics.yaml`, with the following content:
----
kind: Service
apiVersion: v1
metadata:
  name: helidon-metrics // <1>
  labels:
    app: helidon-metrics
  annotations:
    prometheus.io/scrape: 'true' // <2>
spec:
  type: NodePort
  selector:
    app: helidon-metrics
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: helidon-metrics
spec:
  replicas: 1 // <3>
  template:
    metadata:
      labels:
        app: helidon-metrics
        version: v1
    spec:
      containers:
        - name: helidon-metrics
          image: helidon-metrics-se
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
----
<1> A service of type `NodePort` that serves the default routes on port `8080`.
<2> An annotation that will allow Prometheus to discover and scrape the application pod.
<3> A deployment with one replica of a pod.


[source,bash]
.Create and deploy the application into Kubernetes:
----
kubectl apply -f ./metrics.yaml
----

[source,bash]
.Get the service information:
----
kubectl get service/helidon-metrics
----

[source,bash]
----
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-metrics   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s // <1>
----
<1> A service of type `NodePort` that serves the default routes on port `31143`

[source,bash]
.Verify the metrics endpoint using port `31143`, your port will likely be different:
----
curl http://localhost:31143/metrics
----

NOTE: Leave the application running in Kubernetes since it will be used for Prometheus integration.

==== Prometheus integration

The metrics service that you just deployed into Kubernetes is already annotated with `prometheus.io/scrape:`.  This will allow
Prometheus to discover the service and scrape the metrics.  In this exercise, you will install Prometheus
into Kubernetes, then verify that it discovered the Helidon metrics in your application.

[source,bash]
.Install Prometheus and wait until the pod is ready:
----
helm install stable/prometheus --name metrics
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl get pod $POD_NAME
----

You will see output similar to the following.  Repeat the `kubectl get pod` command until you see `2/2` and `Running`. This may take up to one minute.

[source,bash]
----
metrics-prometheus-server-5fc5dc86cb-79lk4   2/2     Running   0          46s
----

[source,bash]
.Create a port-forward so you can access the server URL:
----
kubectl --namespace default port-forward $POD_NAME 7090:9090
----

Now open your browser and navigate to `http://localhost:7090/targets`.  Search for helidon on the page and you will see your
Helidon application as one of the Prometheus targets.

==== Final cleanup

You can now delete the Kubernetes resources that were just created during this example.

[source,bash]
.Delete the Prometheus Kubernetes resources:
----
helm delete --purge metrics
----

[source,bash]
.Delete the application Kubernetes resources:
----
kubectl delete -f ./metrics.yaml
----

=== Summary

This guide demonstrated how to use metrics in a Helidon SE application using various combinations of
metrics and scopes.

* Access metrics for all three scopes: base, vendor, and application
* Configure metrics that are updated by the application when an application REST endpoint is invoked
* Configure a `Gauge` metric
* Integrate Helidon metrics with Kubernetes and Prometheus

Refer to the following references for additional information:

* MicroProfile Metrics specification at https://github.com/eclipse/microprofile-metrics/releases/tag/1.1
* MicroProfile Metrics Javadoc at https://javadoc.io/doc/org.eclipse.microprofile.metrics/microprofile-metrics-api/1.1.1
* Helidon Javadoc at https://helidon.io/docs/latest/apidocs/index.html?overview-summary.html

