8000 fix(query): bound log queries and release database resources by kl7sn · Pull Request #1271 · clickvisual/clickvisual · GitHub
[go: up one dir, main page]

Skip to content

fix(query): bound log queries and release database resources - #1271

Merged
kl7sn merged 1 commit into
masterfrom
kl7sn/261108-ch-query-hang
Nov 13, 2026
Merged

kl7sn merged 1 commit into
masterfrom
kl7sn/261108-ch-query-hang

Conversation

@kl7sn
@kl7sn kl7sn commented Nov 8, 2026
Copy link
Copy Markdown
Collaborator

Log queries could wait indefinitely before reaching ClickHouse, and scan/transaction failures could retain connections in the shared ten-connection pool. This change gives logs/charts, metadata reads, permission-lock waits, connection initialization, timeline optimization, and optional counts one request deadline (configurable app.queryTimeout, default 30s) and returns identifiable timeout/cancellation errors.

It also closes ClickHouse/Databend/trace rows on every exit, checks iteration errors, rolls back failed trace batches, and emits in-flight/completed slow-query diagnostics with correlated request/table/instance IDs, the actual stage and pool, safe error classifications, and sql.DB.Stats() (app.querySlowThreshold, default 2s). The pinned ClickHouse v2.3.0 driver requires a contextual connector and native socket cancellation during handshake/network reads. The pinned Databend v0.2.1 driver also ignores HTTP contexts; a local reader adapter now applies the shared deadline to response headers, bodies and pages, while retaining its parsers and write/batch implementation. Reader bind arguments and named tls_config fail explicitly instead of falling back to the unbounded path; ordinary ClickVisual reader SQL uses neither.

Validation: fault-injection tests cover pool exhaustion/recovery, cancellation without retries/fallback, shared retry/timeline/log/count budgets, metadata timeouts, stream/scan errors, and transaction failures. Real TCP tests cover native handshake deadlines and cancellation after handshake; real HTTP tests cover Databend headers, bodies, pagination, client cancellation, early close, schema/empty pages, and connection reuse. Backend build and scoped go vet pass. Existing unrelated full-package failures were reproduced before edits: SQL-format expectations in Test_adaSelectPart/Test_adaSelectPart1, and TestSettingUpdate requiring an external default configuration.

Incident analysis found independent prom2click pools, a shared page/trace pool, pre-query MySQL reads, and a permission mutex whose holders can access external stores; CMDB event enqueue is nonblocking. Without incident goroutine stacks, none can be claimed as the unique production cause. This PR targets GitHub master; the divergent Shimo upstream needs its own adaptation and verification.

Agent smoke passed against the built backend with temporary SQLite and a synthetic ClickHouse HTTP peer: normal logs/count/charts; ten concurrent blocked queries returned an explicit timeout at ~1s; the eleventh recorded InUse=10, WaitCount=1 and ~0.991s of pool wait; a disconnected client canceled its downstream query and left the query pool at InUse=0; the next query succeeded. No production or UI changes were exercised.

@kl7sn
kl7sn merged commit b1be7f5 into master Nov 13, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0