This repository was archived by the owner on Oct 23, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 23, 2026. It is now read-only.
mtail fails to scrape with > 70k metrics presented. #903
Description
We find that with our current deployment, even though the scrape time is under 2.5 seconds, HTTP GET on the /metrics endpoint will just fail if mtail has more than 70k metrics. There are no errors in the logs, no issues, just a failed response and a connection close after 2 seconds. Reducing the metric count appears to restore operation
However, we need more metrics.
Activity
jaqx0r commentedon Jul 19, 2024
Which version please?
https://github.com/google/mtail/blob/main/docs/Troubleshooting.md#reporting-a-problem
Does it look like
mtailhas also stopped processing lines when a GET is being processed?terencehonles commentedon Aug 6, 2024
This may indeed be related to the issue I was seeing and the change #908. I was testing with the
/jsonhandler and it does emit the headers and then stream the response. I had noticed that testing/metricswas returning an empty response (when I was in the browser).For
/jsonI was seeing aE0805 16:26:05.490112 435505 json.go:27] write tcp [::1]:3903->[::1]:55250: i/o timeoutmessage.From
curlwith verbose logging I was seeing:When rebuilding mtail without #908 (I need #906 for my mtail program) and testing
/metricsagain, I do see that there's nothing written to the logs, and curl looks like:@JWThorne you can probably look at one of the other exporters to confirm you're seeing partial output from them, and you can either build from the source or wait till #908 is released
terencehonles commentedon Aug 6, 2024
This is the number of outputted metrics or the number of log lines you're processing?
For my case we had a number of counters with a large number of labels, so it was generating a large JSON payload and hitting the timeout.
Add reproducing test for google#903: scrape failure with >70k metrics