Service metrics¶

Every Knative Service has a proxy container that proxies the connections to the application container. A number of metrics are reported for the queue proxy performance.

Using the following metrics, you can measure if requests are queued at the proxy side (need for backpressure) and what is the actual delay in serving requests at the application side.

Queue proxy metrics¶

Requests endpoint.

Metric Name	Description	Type	Tags	Unit	Status
`revision_request_count`	The number of requests that are routed to queue-proxy	Counter	`configuration_name` `container_name` `namespace_name` `pod_name` `response_code` `response_code_class` `revision_name` `service_name`	Dimensionless	Stable
`revision_request_latencies`	The response time in millisecond	Histogram	`configuration_name` `container_name` `namespace_name` `pod_name` `response_code` `response_code_class` `revision_name` `service_name`	Milliseconds	Stable
`revision_app_request_count`	The number of requests that are routed to user-container	Counter	`configuration_name` `container_name` `namespace_name` `pod_name` `response_code` `response_code_class` `revision_name` `service_name`	Dimensionless	Stable
`revision_app_request_latencies`	The response time in millisecond	Histogram	`configuration_name` `namespace_name` `pod_name` `response_code` `response_code_class` `revision_name` `service_name`	Milliseconds	Stable
`revision_queue_depth`	The current number of items in the serving and waiting queue, or not reported if unlimited concurrency	Gauge	`configuration_name` `event-display` `container_name` `namespace_name` `pod_name` `response_code_class` `revision_name` `service_name`	Dimensionless	Stable

Note

The revision_queue_depth metric will be exported only if the revision concurrency hard limit is set to a value greater than 1.

Exposing Queue proxy metrics¶

Queue proxy exports metrics for the requests endpoint on port 9091. The metrics can be scraped by Prometheus when metrics.request-metrics-backend-destination is set to prometheus (default) in the configmap observability. The backend can be changed to opencensus which uses a push model and requires a destination address which can be set in the same configmap via metrics.opencensus-address. User can control the reporting period for both backends with metrics.request-metrics-reporting-period-seconds. If metrics.request-metrics-reporting-period-seconds is not set at all then the reporting period depends on the value of the global reporting period, metrics.reporting-period-seconds, that affects both control and data planes. If both properties are not available then the reporting period defaults to 5s for the Prometheus backend and 60s for the Opencensus one.

Here is a sample configuration for the observability configmap in order to connect to the OpenTelemetry collector:

metrics.request-metrics-backend-destination: "opencensus"
metrics.opencensus-address: "otel-collector.metrics:55678"
metrics.request-metrics-reporting-period-seconds: "1"

Note

The reporting period is to 1s so that we can push metrics as soon as possible but this could be overwhelming for the targeted metrics backend. Setting a value of zero or a negative value defaults to 10s (does not mean no delay) which is the default reporting period defined by the Opencensus metrics client library. The latter is used by Knative Serving for exporting metrics.