Repository navigation
Dashboard panels fail to load simultaneously #846
Description
Description:
Multiple dashboard panels fail to load at the same time, even though each panel loads correctly when viewed individually. Grafana appears to not send some queries to ClickHouse when rendering the entire dashboard.
The issue seems to be related to how Grafana batches or throttles queries when multiple panels are rendered at once. Each panel works in isolation, suggesting the queries themselves are valid, but the system fails under combined load.
Key observations from logs:
In Grafana UI, panels show: Query execution failed: Unexpected error
Grafana logs:
logger=context userId=1 orgId=1 uname=admin t=2026-01-21T11:06:59.377516695Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/PDEE91DDB90597936/ status=499 remote_addr=172.22.254.221 time_ms=34 duration=34.613047ms size=0 referer="https://grafana.pgw.internal/d/beueev132i3ggf/health-on-clickhouse?from=now-6h&orgId=1&to=now" handler=/api/datasources/proxy/uid/:uid status_source=downstream
logger=plugin.context t=2026-01-21T11:07:01.366730304Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.388783856Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.414983793Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.455126679Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.477803362Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.498109591Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.523216782Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.546203671Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.568825792Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.591535768Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.609274211Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=plugin.context t=2026-01-21T11:07:01.630196269Z level=warn msg="Could not create user agent" error="invalid user agent format"
logger=context userId=1 orgId=1 uname=admin t=2026-01-21T11:07:01.67213073Z level=info msg="Request Completed" method=GET path=/api/datasources/proxy/uid/PDEE91DDB90597936/ status=499 remote_addr=172.22.254.221 time_ms=7 duration=7.042584ms size=0 referer="https://grafana.pgw.internal/d/beueev132i3ggf/health-on-clickhouse?from=now-6h&orgId=1&to=now" handler=/api/datasources/proxy/uid/:uid status_source=downstream
logger=authn.service t=2026-01-21T11:07:08.591038583Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2026-01-21T11:07:08.591147943Z level=warn msg=Unauthorized error="user token not found" remote_addr=192.168.242.103 traceID=
logger=context userId=0 orgId=0 uname= t=2026-01-21T11:07:08.591473976Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.242.103 time_ms=1 duration=1.211845ms size=40 referer= handler=/api/live/ws status_source=server
Environment:
- Grafana version: 10.4.18-security-01
- ClickHouse Grafana plugin version: v3.4.8
- ClickHouse version: 25.3.4.190
Expected behavior:
All panels should load simultaneously when opening the dashboard.
Activity
Slach commentedon Jan 23, 2026
do you see any error when move mouse cursor to red icons?
do you see any error in developer tools -> javascript console?
Slach commentedon Jan 30, 2026
@Anbu42 any news from your side? could you answer questions please?
Anbu42 commentedon Jan 30, 2026
@Slach Sorry for the long replies
Yes, When I point at the red button I see only this message
Query execution failed: Unexpected errorThere are many such messages in the javascript console:
Slach commentedon Jan 30, 2026
ok. could you check devtools Network tab and look to network traffic to datasource?
Anbu42 commentedon Jan 30, 2026
@Slach There should be no problems with the network connection, I checked using a clickhouse client from the same cluster. We also did not change anything in the network configuration. I also tried to configure the datasource to different clickhouse nodes, the problem occurs when connecting to any of the nodes
Slach commentedon Jan 30, 2026
@Anbu42 could you filter network by /api/datasource ? all requests return 200 ?
Slach commentedon Jan 31, 2026
@Anbu42 what is your network environmnet? is your grafana browser connect to your grafana server directly? status=499 from logs means client close connection before grafana server send response
do you have something between browser and grafana?
and between grafana and clickhouse server?
Anbu42 commentedon Feb 2, 2026
@Slach Grafana is deployed in a K8s cluster, Clickhouse is running on virtual machines. There are no network issues, as data source is active, and I can easily connect to Clickhouse from the same cluster using the cli.
Let me remind you once again that all dashboards load one by one without any problems.
Slach commentedon Feb 3, 2026
Actually you got zero size response from ClickHouse and this looks like a network issue
could you click to highlighted requests in Network Tools and share more screenshots?
need to see response headers
Anbu42 commentedon Feb 9, 2026
@Slach No response data available for this requests
Anbu42 commentedon Feb 16, 2026
@Slach Are there any updates on this issue? The problem is definitely not in the network, other requests are working.
The problem is most likely in specific queries, for example, now some queries have stopped working at all, even one at a time. But by mistake it is not possible to debug it to understand the problem. How we you debug this plugin?
frvade commentedon Feb 17, 2026
@Slach it looks like it happens when the query is too long. When using HTTP/1.1 the server responds with 414 error normally and only the problematic query gets broken, with HTTP/2 it breaks the whole connection and whole page doesn't load because of the one long query.
So the root cause is in the method of sending SQL query in the URL. Is it possible to migrate it to POST maybe?
Slach commentedon Feb 17, 2026
@frvade we already have option to use Post look to datasource settings
look providion examples
https://github.com/Altinity/clickhouse-grafana/blob/master/docker/grafana/provisioning/datasources/clickhouse.yaml#L15
Slach commentedon Feb 17, 2026
@Anbu42 please change use Post checkbox in datasource settings and check again
Slach commentedon Mar 16, 2026
@Anbu42 is error still present?