Collecting Metrics in Knative¶

Knative supports different popular tools for collecting metrics:

• Prometheus
• OpenTelemetry Collector

Grafana dashboards are available for metrics collected directly with Prometheus.

You can also set up the OpenTelemetry Collector to receive metrics from Knative components and distribute them to other metrics providers that support OpenTelemetry.

About the Prometheus Stack¶

Prometheus is an open-source tool for collecting, aggregating timeseries metrics and alerting. It can also be used to scrape the OpenTelemetry Collector that is demonstrated below when Prometheus is used.

Grafana is an open-source platform for data analytics and visualization, enabling users to create customizable dashboards for monitoring and analyzing metrics from various data sources.

Prometheus Stack is a preconfigured collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, combined to provide end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. The stack includes by default some Prometheus packages and Grafana.

Setting up the Prometheus Stack¶

1. Install the Prometheus Stack by using Helm:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install prometheus prometheus-community/kube-prometheus-stack -n default -f values.yaml
   # values.yaml contains at minimum the configuration below
   

   Caution

   You will need to ensure that the helm chart has following values configured, otherwise the ServiceMonitors/Podmonitors will not work.

   kube-state-metrics:
     metricLabelsAllowlist:
       - pods=[*]
       - deployments=[app.kubernetes.io/name,app.kubernetes.io/component,app.kubernetes.io/instance]
   prometheus:
     prometheusSpec:
       serviceMonitorSelectorNilUsesHelmValues: false
       podMonitorSelectorNilUsesHelmValues: false
   

2. Apply the ServiceMonitors/PodMonitors to collect metrics from Knative.

   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/servicemonitor.yaml
   

Access the Prometheus instance locally¶

By default, the Prometheus instance is only exposed on a private service named prometheus-kube-prometheus-prometheus.

To access the console in your web browser:

1. Enter the command:

   kubectl port-forward -n default svc/prometheus-kube-prometheus-prometheus 9090:9090
   

2. Access the console in your browser via http://localhost:9090.

Access the Grafana instance locally¶

By default, the Grafana instance is only exposed on a private service named prometheus-grafana.

To access the dashboards in your web browser:

1. Enter the command:

   kubectl port-forward -n default svc/prometheus-grafana 3000:80
   

2. Access the dashboards in your browser via http://localhost:3000.

3. Use the default credentials to login:

   username: admin
   password: prom-operator
   

Import Grafana dashboards¶

1. Grafana dashboards can be imported from the monitoring repository.

2. If you are using the Grafana Helm Chart with the Dashboard Sidecar enabled, you can load the dashboards by applying the following configmaps.

   kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/grafana/dashboards.yaml
   

   Caution

   You will need to ensure that the helm chart has following values configured, otherwise the dashboards loading will not work.

   grafana:
     sidecar:
       dashboards:
         enabled: true
         searchNamespace: ALL
   

   If you have an existing configmap and the dashboards loading doesn't work, add the labelValue: true attribute to the helm chart after the searchNamespace: ALL declaration.

About OpenTelemetry¶

OpenTelemetry is a CNCF observability framework for cloud-native software, which provides a collection of tools, APIs, and SDKs.

You can use OpenTelemetry to instrument, generate, collect, and export telemetry data. This data includes metrics, logs, and traces, that you can analyze to understand the performance and behavior of Knative components.

OpenTelemetry allows you to easily export metrics to multiple monitoring services without needing to rebuild or reconfigure the Knative binaries.

Understanding the collector¶

The collector provides a location where various Knative components can push metrics to be retained and collected by a monitoring service.

In the following example, you can configure a single collector instance using a ConfigMap and a Deployment.

Set up the collector¶

1. Create a namespace for the collector to run in, by entering the following command:

   kubectl create namespace metrics
   

   The next step uses the metrics namespace for creating the collector.

2. Create a Deployment, Service, and ConfigMap for the collector by entering the following command:

   kubectl apply -f https://raw.githubusercontent.com/knative/docs/main/docs/serving/observability/metrics/collector.yaml
   

3. Update the config-observability ConfigMaps in the Knative Serving and Eventing namespaces, by entering the follow command:

   kubectl patch --namespace knative-serving configmap/config-observability \
     --type merge \
     --patch '{"data":{"metrics.backend-destination":"opencensus","metrics.request-metrics-backend-destination":"opencensus","metrics.opencensus-address":"otel-collector.metrics:55678"}}'
   kubectl patch --namespace knative-eventing configmap/config-observability \
     --type merge \
     --patch '{"data":{"metrics.backend-destination":"opencensus","metrics.opencensus-address":"otel-collector.metrics:55678"}}'
   

Verify the collector setup¶

1. You can check that metrics are being forwarded by loading the Prometheus export port on the collector, by entering the following command:

   kubectl port-forward --namespace metrics deployment/otel-collector 8889
   

2. Fetch http://localhost:8889/metrics to see the exported metrics.