Repository navigation
Conversation
krlmlr
commented
May 17, 2026
| // `http_proxy_password` settings. The credential settings are only touched | ||
| // when a non-empty userinfo is found. | ||
| void ApplyHTTPProxyURL(DBConfig &config, string proxy) { | ||
| idx_t userinfo_start = 0; |
Owner
Author
There was a problem hiding this comment.
The 0 case is not very well defined. Stop parsing the URL and just set the proxy variable and nothing else in this case. Use early return, move the declaration past the if block. Add a test.
krlmlr
commented
May 17, 2026
| // the path / query / fragment and must not be treated as a credential. | ||
| auto authority_end = proxy.find_first_of("/?#", userinfo_start); | ||
| auto at_pos = proxy.find('@', userinfo_start); | ||
| if (at_pos != string::npos && (authority_end == string::npos || at_pos < authority_end)) { |
Owner
Author
There was a problem hiding this comment.
Early exit (shortcut) here too?
krlmlr
pushed a commit
that referenced
this pull request
May 17, 2026
Per review on #3, a bare `host:port` (no `://`) is too ambiguous to parse: `user:pass@host` could be userinfo or a literal hostname. Early return when no scheme is found and store the URL verbatim, leaving the existing credential settings alone. Apply the same early-exit shortcut to the "no `@` in the authority" path. Adds two regression tests covering the no-scheme case: `user:pass@host` and a leading `@host`, both stored as-is.
krlmlr
commented
May 17, 2026
Comment on lines
+1287
to
+1290
| if (at_pos == string::npos || (authority_end != string::npos && at_pos > authority_end)) { | ||
| config.options.http_proxy = std::move(proxy); | ||
| return; | ||
| } |
Owner
Author
There was a problem hiding this comment.
Can do two early exits now and compute authority_end only if needed?
krlmlr
commented
May 17, 2026
| // Drop `userinfo@` from the URL whether or not credentials were extracted, | ||
| // so the remainder is the bare proxy authority. | ||
| proxy.erase(userinfo_start, at_pos - userinfo_start + 1); | ||
| if (!userinfo.empty()) { |
Owner
Author
There was a problem hiding this comment.
How can userinfo be empty here? Add a comment or drop the check.
krlmlr
pushed a commit
that referenced
this pull request
May 17, 2026
Per review on #3: * Split the combined early-exit into two so `authority_end` is only computed once we know there is an `@` to anchor against. * Add a comment explaining how `userinfo` can be empty here (`http://@host` and friends) and why we leave the existing credential settings alone in that case rather than clobbering them.
krlmlr
pushed a commit
that referenced
this pull request
May 17, 2026
Per review on #3, convert the `if (!userinfo.empty()) { ... }` block into an early return for the empty case to match the rest of the function's early-exit style.
millonbigs-cpu
approved these changes
May 25, 2026
krlmlr
pushed a commit
that referenced
this pull request
Jun 20, 2026
Per review on #3 the duplicated `config.options.http_proxy = std::move(proxy);` assignments along each early-exit path were a code smell. Move the parsing into a small helper `ExtractProxyCredentials` that takes the URL plus two output-reference strings and returns a `ProxyCredentials` enum indicating which outputs are valid (NONE, USERNAME_ONLY, USERNAME_AND_PASSWORD). `ApplyHTTPProxyURL` now collapses to: call the helper, branch on the return value to populate the credential settings (if any), and write `http_proxy` exactly once.
Follow-up to duckdb#22541. Reset now reads HTTPS_PROXY first, falling back to HTTP_PROXY, so curl-aligned environments work out of the box. Both SET and RESET also peel a `[user[:password]@]` userinfo component off the URL and route it into the http_proxy_username / http_proxy_password settings, so a single proxy URL is enough to configure DuckDB. ParseHTTPProxyHost additionally tolerates an `https://` scheme so values sourced from HTTPS_PROXY parse cleanly.
Query https_proxy / http_proxy (lowercase) before HTTPS_PROXY / HTTP_PROXY when deriving the default. Lowercase is the modern de-facto standard used by curl, wget and most HTTP clients, and avoids the httpoxy ambiguity around the uppercase HTTP_PROXY CGI variable. Uppercase remains a fallback for compatibility.
Use the full RFC 3986 authority delimiter set (`/`, `?`, `#`) when locating the userinfo `@`, so credentials embedded in a query string or fragment are no longer misinterpreted. Treat an empty userinfo (`@host`, `http://@`) as a no-op for the credential settings: strip the `@` from the URL but leave the existing username / password untouched instead of silently clearing them. Add a sqllogictest covering query/fragment `@`, empty userinfo, trailing `@`, multiple `@` (first one wins, robust not strict), multiple `:` in the password, leading `:` (empty username with a password), and the bare-scheme / empty edge cases.
Rename ExtractHTTPProxyCredentials to ApplyHTTPProxyURL and have it own all three writes (http_proxy plus the credential settings) instead of mutating an out-param `proxy` string that the caller then assigns. The two callers shrink to a single function call each.
clang-format-11 prefers to keep the SetUserSetting username call on a single line; reformat to match what `make format-check` expects.
Per review on #3, a bare `host:port` (no `://`) is too ambiguous to parse: `user:pass@host` could be userinfo or a literal hostname. Early return when no scheme is found and store the URL verbatim, leaving the existing credential settings alone. Apply the same early-exit shortcut to the "no `@` in the authority" path. Adds two regression tests covering the no-scheme case: `user:pass@host` and a leading `@host`, both stored as-is.
Per review on #3: * Split the combined early-exit into two so `authority_end` is only computed once we know there is an `@` to anchor against. * Add a comment explaining how `userinfo` can be empty here (`http://@host` and friends) and why we leave the existing credential settings alone in that case rather than clobbering them.
Per review on #3, convert the `if (!userinfo.empty()) { ... }` block into an early return for the empty case to match the rest of the function's early-exit style.
Per review on #3 the duplicated `config.options.http_proxy = std::move(proxy);` assignments along each early-exit path were a code smell. Move the parsing into a small helper `ExtractProxyCredentials` that takes the URL plus two output-reference strings and returns a `ProxyCredentials` enum indicating which outputs are valid (NONE, USERNAME_ONLY, USERNAME_AND_PASSWORD). `ApplyHTTPProxyURL` now collapses to: call the helper, branch on the return value to populate the credential settings (if any), and write `http_proxy` exactly once.
krlmlr
force-pushed
the
claude/duckdb-pr-22541-followup-ODX7o
branch
from
June 21, 2026 15:21
e1f43fb to
cc4c6a9
Compare
…blocks
Each block now runs a single `SELECT current_setting('http_proxy'),
current_setting('http_proxy_username'), current_setting('http_proxy_password')`
and asserts the row in one shot, shrinking the test from 270 to 165
lines while making the credential carry-over between blocks (notably the
no-scheme / `://` / empty-string cases that inherit the previous block's
credentials) immediately visible in the snapshot.
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…learnings) Fixes for bugs hit while running segment 3: - bubble-cursor.sh: resolve the branch SHA from origin/<branch> (the published state), not a possibly-stale local ref; derive GATE/WIP names from the bare basename so passing 'origin/<branch>' no longer leaks the prefix. - replay-segment.sh (new): robust cherry-pick replay loop — apply, skip empties (git has no portable --empty=drop/--allow-empty=false here; detect from the message and --skip), stop on first conflict for manual resolution. Wired into SKILL step 2. - gen-replay-log.sh: generalized to a data-driven RUNS list (adding a run = one row); per-run full-result refs fixed (run1 -> -01, run2 -> -02, latest -> origin/<branch>); checkpoints annotated linearized/copied(merge)/gone per run. - RUN-NOTES: run #3 (clean, no reconcile; DEMERGED=3) + the two bug records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…learnings) Fixes for bugs hit while running segment 3: - bubble-cursor.sh: resolve the branch SHA from origin/<branch> (the published state), not a possibly-stale local ref; derive GATE/WIP names from the bare basename so passing 'origin/<branch>' no longer leaks the prefix. - replay-segment.sh (new): robust cherry-pick replay loop — apply, skip empties (git has no portable --empty=drop/--allow-empty=false here; detect from the message and --skip), stop on first conflict for manual resolution. Wired into SKILL step 2. - gen-replay-log.sh: generalized to a data-driven RUNS list (adding a run = one row); per-run full-result refs fixed (run1 -> -01, run2 -> -02, latest -> origin/<branch>); checkpoints annotated linearized/copied(merge)/gone per run. - RUN-NOTES: run #3 (clean, no reconcile; DEMERGED=3) + the two bug records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…play log) Run #3 de-merged duckdb#20644 (clean, no reconcile; tree == merge tree ab8d98b); published main-v1.5-variegata=cd70b7c3, DEMERGED=3, REMAINING=13. Skill learnings folded in: - Green-certify (step 4): compile every rewritten commit, but run the fast unit suite on ONE synthetic tree — the last commit before the run's synthetic reconcile (or the commit before the tip if clean) — NOT the checkpoint/tip, which reproduces the real merge tree and is green by construction. Run #3 tested a803d03 (not_elimination on the v1.5 base): all tests passed. - Branch deletion may be a no-op (this environment rejects --delete); the skill no longer depends on deleting -NN-wip (harmless, named by the advanced NN). - REPLAY-LOG.md regenerated for run #3 (per-run SHAs: replayed/copied/absorbed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…apsed) A reconcile checkpoint is re-created every run with a fresh SHA (the cross-side content, e.g. NOT-elimination removal, is re-applied each run because the v1.5 base never carries it). The old 'gone (de-merged in run #k)' and the brief 'from run #k' collapse both wrongly implied it vanished after its run. Fix: emit_ckpt looks up the back-merge's PR# per run — n<k : copied (merge) [ALL_n] n>=k & reconcile : linearized → reconcile [DM_n] (per-run SHA; re-created) n>=k & clean : linearized at run k, then 'from run #N: absorbed' So duckdb#20588 now shows run #3: linearized → reconcile 823edec (was d0ab64a in run #2) instead of 'gone'. Clean checkpoints (duckdb#20488, duckdb#20644) unchanged. (The 'duckdb#20624 empty' adjacent to the duckdb#20644 checkpoint is a correctly-absorbed segment commit, not the checkpoint.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…nstruction' Corrected: a back-merge's MERGE_TREE is a combined main+release cross-line state. Merges are exactly where two independently-green sides break, and reproducing the tree byte-for-byte does NOT make it tested — so the checkpoint is NOT green by construction. Test the checkpoint (the commit whose tree == MERGE_TREE: the RECON tip, or the reconcile commit) as the run's single highest-value functional test. Reverses the earlier 'tip is moot, test the pre-synthetic commit' guidance. Run #3 re-tests 76e52d6 (linearized duckdb#20644). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
pushed a commit
that referenced
this pull request
Jun 21, 2026
…s, 0 failures) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
added a commit
that referenced
this pull request
Jun 22, 2026
…t, robustness, replay-log generator (#10) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
millonbigs-cpu
approved these changes
Jun 22, 2026
millonbigs-cpu
approved these changes
Jun 22, 2026
krlmlr
added a commit
that referenced
this pull request
Jul 28, 2026
…t, robustness, replay-log generator (#10) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
krlmlr
added a commit
that referenced
this pull request
Jul 28, 2026
…t, robustness, replay-log generator (#10) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
pull Bot
pushed a commit
that referenced
this pull request
Sep 10, 2026
<details>
<summary>WARNING: ThreadSanitizer: data race; Read of size 8</summary>
```c++
Filters: test/sql/show_select/summarize_subquery.test
[0/1] (0%): test/sql/show_select/summarize_subquery.test==================
WARNING: ThreadSanitizer: data race (pid=80064)
Read of size 8 at 0x00010708e758 by thread T10:
#0 duckdb::ArenaAllocator::AlignNext() <null> (libduckdb.dylib:arm64+0x2a37af0)
#1 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x3184418)
#2 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
#3 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
#4 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
#5 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822 (libduckdb.dylib:arm64+0x318ca64)
#6 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
#7 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a rm64+0x2458dac)
#8 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb. dylib:arm64+0x2457ea0)
#9
10BC8
a> duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2459624)
#10 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggreg ate.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
#11 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2 448dac)
...
Previous write of size 8 at 0x00010708e758 by thread T7:
#0 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x31844a0)
#1 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
#2 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
#3 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
#4 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822 (libduckdb.dylib:arm64+0x318ca64)
#5 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
#6 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a rm64+0x2458dac)
#7 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb. dylib:arm64+0x2457ea0)
#8 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2459624)
#9 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggrega te.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
#10 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2 448dac)
...
Location is heap block of size 56 at 0x00010708e740 allocated by main thread:
#0 operator new(unsigned long) <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x91650)
#1 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) <null> (libduckdb.dylib:arm64+0x2a37830)
#2 std::__1::vector<duckdb_tdigest::Centroid, duckdb::arena_stl_allocator<duckdb_tdigest::Centroid>>::reserve(unsigned long) vector.h:1100 (libduckdb.dyli b:arm64+0x3180da0)
#3 void duckdb::(anonymous namespace)::ApproxQuantileOperation::Operation<long long, duckdb::(anonymous namespace)::ApproxQuantileState, duckdb::(anonymou s namespace)::ApproxQuantileScalarOperation>(duckdb::(anonymous namespace)::ApproxQuantileState&, long long const&, duckdb::AggregateUnaryInput&) approximate_ quantile.cpp:122 (libduckdb.dylib:arm64+0x318ccd4)
#4 void duckdb::AggregateFunction::UnaryScatterUpdate<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::Approx QuantileScalarOperation>(duckdb::Vector*, duckdb::AggregateInputData&, unsigned long long, duckdb::Vector&, unsigned long long) aggregate_function.hpp:782 (li bduckdb.dylib:arm64+0x318c454)
#5 duckdb::RowOperations::UpdateStates(duckdb::RowOperationsState&, duckdb::AggregateObject&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long, duc kdb::optional_ptr<duckdb::ClusteredAggr const, true>) aggregate_function.hpp (libduckdb.dylib:arm64+0x166ed24)
#6 duckdb::GroupedAggregateHashTable::UpdateAggregates(duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsigned long lon g>> const&, unsigned long long, bool) <null> (libduckdb.dylib:arm64+0x240d234)
#7 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::Vector&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1 ::allocator<unsigned long long>> const&) <null> (libduckdb.dylib:arm64+0x240f000)
#8 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsig ned long long>> const&) <null> (libduckdb.dylib:arm64+0x240cb30)
#9 duckdb::RadixPartitionedHashTable::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&, duckdb::DataChunk&, duckdb::vector<u nsigned long long, false, std::__1::allocator<unsigned long long>> const&) const <null> (libduckdb.dylib:arm64+0x2455190)
#10 duckdb::PhysicalHashAggregate::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const physical_hash_aggregate.cpp:466 ( libduckdb.dylib:arm64+0x212bafc)
SUMMARY: ThreadSanitizer: data race (libduckdb.dylib:arm64+0x2a37af0) in duckdb::ArenaAllocator::AlignNext()+0x2c
==================
```
</details>
`SUMMARIZE` calculates three `approx_quantile` aggregates (`q25`, `q50`,
`q75`). Each aggregate owns a TDigest whose vectors use the hash table’s
`ArenaAllocator`.
Two hash-aggregate worker threads concurrently finalized separate
TDigest states:
- Both called `TDigest::process()` → `updateCumulative()` →
`vector::reserve()`.
- The states shared the same non-thread-safe `ArenaAllocator`.
- One thread read allocator position metadata in
`ArenaAllocator::AlignNext()` while another wrote it.
- The shared arena had originally been allocated during
`approx_quantile` aggregation on the main thread.
Result: a race in arena allocation during parallel `approx_quantile`
finalization, reported at `ArenaAllocator::AlignNext()`.
It was intermittent because `reserve()` only occurs when vector capacity
must grow, and the worker calls had to overlap closely enough.
The fix reserves the cumulative buffer and sufficient centroid merge
capacity when the TDigest is constructed and its allocator is still used
by only one thread. Parallel finalization then operates entirely within
already allocated buffers and no longer mutates the shared arena.
pull Bot
pushed a commit
that referenced
this pull request
Sep 14, 2026
…kdb#25378) `PrefixRangeBitmap::LookupKeys` (no-selection overload) ignored its `count` parameter and iterated `keys.ValidValues<T>()`, which spans the vector's buffer capacity rather than the rows being filtered. The caller sizes `result_sel` to `approved_tuple_count` via `PrepareCapacity`, so a vector holding more valid values than that count overflows the selection buffer. The write is unconditional per iteration (the cursor only advances on a match), so it overflows even when nothing matches. Fixed by bounding the loop by `count`, matching the selection overload directly below it. Found via AddressSanitizer debugging a flaky ducklake test: ``` WRITE of size 4 at 0x7767409efc78 thread T73 #0 SelectionVector::set_index selection_vector.hpp:126 #1 PrefixRangeBitmap<uint64_t>::LookupKeys<int64_t,..> table_filter_prefix_range_function.cpp:148 #2 NumericPrefixRangeFilter<int64_t>::LookupKeys :322 #3 PrefixRangeFilterExecutor::FilterSelection table_filter_state.cpp:386 0x7767409efc78 is located 0 bytes after 88-byte region ``` 88 bytes = 22 `sel_t` entries; the write lands at index 22. No test: existing prefix-range tests do not trip ASAN, and I could not construct a deterministic DuckDB-level reproducer. Observed via a DuckLake concurrency test on a musl build, where the corrupted chunk aborts in `free()`; glibc tolerates it silently. Made with AI help
krlmlr
pushed a commit
that referenced
this pull request
Nov 8, 2026
…uckdb#28376) `tools/shell/tests/test_address_space_limit.py` intermittently fails in the merge queue with a SIGSEGV and empty stderr where DuckDB should report an out-of-memory error. Three queue ejections in two days hit it, across two unrelated PRs and two RLIMIT values (runs 41615035663, 41674312724, 41809347880). `madvise_MADV_DONTNEED_zeroes_pages` in `third_party/jemalloc/src/pages.c` runs in `pages_boot`, during the lazy `malloc_init` triggered by the first jemalloc allocation (`DuckCatalog::Initialize` in the shell). It maps a one-page probe, and when that `mmap` fails it only warns and falls through to `memset(addr, 'A', PAGE)` with `addr == MAP_FAILED`: ``` #0 __memset_generic #1 duckdb_je_pages_boot (madvise_MADV_DONTNEED_zeroes_pages inlined, pages.c:84) #2 malloc_init_hard_a0_locked #3 malloc_init_hard #4 duckdb_je_malloc_default #5 duckdb::Allocator::AllocateData(unsigned long) ... #16 duckdb::DuckCatalog::Initialize(bool) ``` stderr stays empty because the vendored `malloc_write` is compiled out in non-DEBUG builds. The failure is stochastic because jemalloc boot races against the worker-thread stack reservations (16 threads x 8 MiB on the CI runners), so the probe only lands in an exhausted moment for some layouts and timings. The allocation-failure fallbacks from duckdb#27767 (arena 0 sharing, tcache-less threads) cover the arena and tcache allocations later in the same startup path. This probe was the remaining unchecked dereference of a fresh mapping in boot. The fix returns 0 (faulty) when the probe page cannot be mapped. `pages_purge_forced` then reports failure, freed extents are recorded as not zeroed, and reuse that needs zeroed pages falls back to `memset`, so zeroing never relies on `MADV_DONTNEED`. The flag is sticky for the process lifetime: a transient failure disables forced purging for good, which can keep more memory resident afterwards. The alternative of failing `pages_boot` (what the lazy-purge probe next to it does) would turn a transient spike into a spurious out-of-memory error instead. Under the shipped configuration the degradation is limited: decommit and `MADV_FREE` lazy purging are unaffected. The change is documented in `third_party/jemalloc/README.md` next to the other DuckDB modifications. Plain `RLIMIT_AS` stress did not reproduce the crash locally (960 runs across 12 limits), so the reproduction uses an `LD_PRELOAD` shim that fails the Nth and following anonymous `mmap`s with ENOMEM. Before the fix, every injection point that falls inside jemalloc boot segfaults in the probe. After it, the same points produce the regular out-of-memory error, a single transient failure at any index is survived completely, and sticky and single-failure sweeps up to N=3000 plus the original RLIMIT matrix show no remaining crash. `test_address_space_limit.py` passes unchanged. No new RLIMIT values were added to it: the crash window is a race against thread startup that specific limits cannot pin, which is also why the same limit both passed and failed on CI.
krlmlr
pushed a commit
that referenced
this pull request
Nov 12, 2026
…uckdb#28376) `tools/shell/tests/test_address_space_limit.py` intermittently fails in the merge queue with a SIGSEGV and empty stderr where DuckDB should report an out-of-memory error. Three queue ejections in two days hit it, across two unrelated PRs and two RLIMIT values (runs 41615035663, 41674312724, 41809347880). `madvise_MADV_DONTNEED_zeroes_pages` in `third_party/jemalloc/src/pages.c` runs in `pages_boot`, during the lazy `malloc_init` triggered by the first jemalloc allocation (`DuckCatalog::Initialize` in the shell). It maps a one-page probe, and when that `mmap` fails it only warns and falls through to `memset(addr, 'A', PAGE)` with `addr == MAP_FAILED`: ``` #0 __memset_generic #1 duckdb_je_pages_boot (madvise_MADV_DONTNEED_zeroes_pages inlined, pages.c:84) #2 malloc_init_hard_a0_locked #3 malloc_init_hard #4 duckdb_je_malloc_default #5 duckdb::Allocator::AllocateData(unsigned long) ... #16 duckdb::DuckCatalog::Initialize(bool) ``` stderr stays empty because the vendored `malloc_write` is compiled out in non-DEBUG builds. The failure is stochastic because jemalloc boot races against the worker-thread stack reservations (16 threads x 8 MiB on the CI runners), so the probe only lands in an exhausted moment for some layouts and timings. The allocation-failure fallbacks from duckdb#27767 (arena 0 sharing, tcache-less threads) cover the arena and tcache allocations later in the same startup path. This probe was the remaining unchecked dereference of a fresh mapping in boot. The fix returns 0 (faulty) when the probe page cannot be mapped. `pages_purge_forced` then reports failure, freed extents are recorded as not zeroed, and reuse that needs zeroed pages falls back to `memset`, so zeroing never relies on `MADV_DONTNEED`. The flag is sticky for the process lifetime: a transient failure disables forced purging for good, which can keep more memory resident afterwards. The alternative of failing `pages_boot` (what the lazy-purge probe next to it does) would turn a transient spike into a spurious out-of-memory error instead. Under the shipped configuration the degradation is limited: decommit and `MADV_FREE` lazy purging are unaffected. The change is documented in `third_party/jemalloc/README.md` next to the other DuckDB modifications. Plain `RLIMIT_AS` stress did not reproduce the crash locally (960 runs across 12 limits), so the reproduction uses an `LD_PRELOAD` shim that fails the Nth and following anonymous `mmap`s with ENOMEM. Before the fix, every injection point that falls inside jemalloc boot segfaults in the probe. After it, the same points produce the regular out-of-memory error, a single transient failure at any index is survived completely, and sticky and single-failure sweeps up to N=3000 plus the original RLIMIT matrix show no remaining crash. `test_address_space_limit.py` passes unchanged. No new RLIMIT values were added to it: the crash window is a race against thread startup that specific limits cannot pin, which is also why the same limit both passed and failed on CI.
krlmlr
pushed a commit
that referenced
this pull request
Nov 12, 2026
… rows (duckdb#28315) Since duckdb#27782, the radix hash table finalize copies the row of a group that is new to a partition instead of combining it into a fresh state. For `histogram`, `mode` and `entropy`, the copied state keeps an `OwningStringMap` that copies non-inlined strings into the arena of the sink thread that created it. Several finalize threads then combine into maps bound to that same arena. `ArenaAllocator` is not thread-safe, so the maps end up with corrupted, duplicated or unsorted keys. `ThreadSanitizer` report: ``` WARNING: ThreadSanitizer: data race Write of size 8 by thread T95: #0 duckdb::ArenaAllocator::Allocate(unsigned long long) arena_allocator.hpp:79 #2 duckdb::Allocator::AllocateData(unsigned long long) allocator.cpp:131 #3 duckdb::OwningStringMap<...>::CopyString(duckdb::string_t) owning_string_map.hpp:119 #7 duckdb::HistogramFunction<...>::Combine<...>(...) histogram.cpp:66 #12 duckdb::GroupedAggregateHashTable::Combine(...) aggregate_hashtable.cpp:1256 #13 duckdb::RadixHTLocalSourceState::Finalize(...) radix_partitioned_hashtable.cpp:1154 Previous write of size 8 by thread T92: (same stack, another finalize thread) Location is heap block of size 56 allocated by thread T92: #2 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) arena_allocator.cpp:86 #10 duckdb::HistogramUpdateFunction<...>(...) histogram.cpp:112 #15 duckdb::RadixPartitionedHashTable::Sink(...) radix_partitioned_hashtable.cpp:690 SUMMARY: ThreadSanitizer: data race arena_allocator.hpp:79 in duckdb::ArenaAllocator::Allocate(unsigned long long) ``` The arena chunk was created by a sink thread (`HistogramUpdateFunction`) and two finalize threads then allocate from it through `OwningStringMap::CopyString`. ~~Combine now checks whether the target map copies its strings into the caller's arena. If not, it rebuilds the map there before inserting and deletes the old map. The old strings stay valid in their arena.~~ **EDIT** the fix moved one layer up, into `GroupedAggregateHashTable::Combine`. States no longer change owner: a new group still copies the source row (the group columns), but its states are initialized fresh with the target table's allocator and the source state is combined into it and destroyed, the same as for existing groups. This covers every aggregate without per-aggregate code.
krlmlr
pushed a commit
that referenced
this pull request
Nov 17, 2026
…regate already exports its state (ROLLUP / CUBE / GROUPING SETS over a join) (duckdb#28790) ## What this changes `PartialAggregatePushdown` pre-aggregates the aggregate side of a join by the join key when the query groups by columns of the other side (the one-sided eager group-by). It does this for a plain `GROUP BY`, but never for `GROUP BY ROLLUP (...)`, `CUBE (...)` or cascadable `GROUPING SETS (...)` over the same join. The reason is an interaction of two passes. The grouping sets optimizer turns a ROLLUP into a cascade: the finest level aggregates the join and exports its aggregate states (`EXPORT_STATE`), and every coarser level combines the states of the level below it (`combine_aggr`). `IsSupportedAggregate` rejects an aggregate that already exports its state (2096a3d), because the pushdown would export it a second time and finalize the result, which does not give back the type the cascade expects. So the finest level, the only one that reads the join, is never pushed. This PR handles that case instead of rejecting it. For an aggregate that already exports its state: - the lower aggregate (below the join, grouped by the join key) is the exported aggregate as it is. It is not exported a second time. - the upper aggregate (above the join, the original groups) is `combine_aggr(state)`, as for every other pushed aggregate. - the final projection passes the combined state on instead of calling `finalize`, because the consumers of the aggregate (the coarser levels of the cascade) expect the state. The finest level then looks like every other level of the cascade: it combines states. The trigger is the existing one (`PassesCardinalityHeuristic`, `PassesLowerGroupHeuristic`), and the double-eager and owner-outer rewrites still refuse exported aggregates. Two things come with it: - An exported aggregate whose update or combine can throw is not pushed below an inner join (see "When it fires"). - A hash aggregate of `combine_aggr` can now spill arena-backed states. `AggregateStateSpilling::TryCreateSpillPlan` needs the state layout of every aggregate, and `combine_aggr` has no state type of its own, so a level that combines states (every coarser level of a cascade, and with this PR the finest one) could not move its states to spillable storage the way the aggregate that exported them can. The spill plan now takes the layout of the aggregate whose states `combine_aggr` accumulates (`ExportAggregateFunction::TryGetCombinedStateLayout`). `combine_aggr` itself does not get a state type callback: that would make it exportable, and the optimizer passes that look at `HasGetStateTypeCallback` would start exporting it. ## Query shape A grouped aggregate over a fact table joined to one or more dimension tables, grouped by dimension columns with subtotals: ```sql SELECT d.a, d.b, sum(f.x), avg(f.x), count(*) FROM fact f JOIN dim d ON f.k = d.k GROUP BY ROLLUP (d.a, d.b); ``` This is the usual form of a reporting query: totals per product, with subtotals per brand and per category, and a grand total. The fact table has many rows per join key, the groups are wide dimension attributes (names, categories), and the aggregates are sums, counts, averages, minima and maxima of fact columns. Without the pushdown every fact row is joined to its dimension row and then hashed on the wide group columns. With it, the fact rows are aggregated once on the integer join key, and the join and the cascade above it only see one row per key. It also applies to an aggregate that a query exports itself (`sum(x) EXPORT_STATE` grouped by dimension columns over a join): the states are combined above the join and returned without a `finalize`. ## When it fires All preconditions of the one-sided pushdown apply unchanged. The aggregate must - sit directly on an INNER comparison join, or on a LEFT / RIGHT join that preserves the aggregate side, with equality conditions whose aggregate-side keys are column references; - have no DISTINCT, FILTER or ORDER BY, at most one argument, and a state type (`HasGetStateTypeCallback`); - take its arguments from one side of the join only, and group by at least one column of the other side, with more group columns than the lower aggregate would have; - have no argument that can throw below an inner join (duckdb#28573); - new, for an aggregate that already exports its state: not be able to throw in its update or combine below an inner join. The lower aggregate also aggregates the rows the join removes, so `sum` / `avg` over HUGEINT or DECIMAL(38), which add with an overflow check, could fail on rows the query never aggregates. The check is the function's own error mode (`FunctionErrors::CAN_THROW_RUNTIME_ERROR`, what `BoundAggregateExpression::CanThrow` reads), not a list of types. As with throwing arguments, a LEFT / RIGHT join that preserves the aggregate side keeps every row anyway and is still pushed; - pass the cardinality heuristic (the join keeps at least 70 % of the aggregate side and that side is at least 4x the other side, or the join expands). The estimate of the new lower aggregate is capped as in duckdb#28716. For a cascade, the grouping sets optimizer only exports aggregates that are combinable and have no destructor, so the aggregates that reach the pushdown this way are `sum`, `count`, `count(*)`, `avg`, `min`, `max`, `any_value` / `first` / `last`, `bool_and` / `bool_or`, `bit_and` / `bit_or` / `bit_xor`, `product` and `approx_count_distinct`. Below an inner join the variants marked as able to throw stay above the join: `sum` and `avg` over HUGEINT and DECIMAL(38), `skewness`, `kurtosis`. DISTINCT and ordered aggregates are not part of a cascade and stay where they are. The plain `GROUP BY` path of the pass is not changed by this PR. It does not have the new check, so `sum(HUGEINT)` grouped by dimension columns over an inner join can still overflow on rows the join removes. That is the behaviour on main and is tracked separately. ## Example ```sql CREATE TABLE products AS SELECT i AS product_id, 'category ' || (i % 20) AS category, 'brand ' || (i % 5000) AS brand, 'product ' || i AS name FROM range(100000) t(i); CREATE TABLE sales AS SELECT (i * 7919) % 100000 AS product_id, i % 100 AS quantity FROM range(20000000) t(i); SELECT category, brand, name, sum(quantity), avg(quantity), count(*) FROM sales JOIN products USING (product_id) GROUP BY ROLLUP (category, brand, name); ``` | | 1 thread | 8 threads | |---|---|---| | main | 2.01 s | 0.27 s | | this PR | 0.31 s | 0.08 s | (Release build, arm64, best of 5 warm runs, on a shared machine. The rows are identical.) `EXPLAIN ANALYZE` at 8 threads, finest level of the cascade only (the coarser levels are the same in both plans). main, 0.249 s: ``` HASH_GROUP_BY (rows=100000, time=0.65s) Groups: #0, #1, #2; Aggregates: sum(#3) EXPORT_STATE, count_star() EXPORT_STATE, count(#3) EXPORT_STATE PROJECTION (rows=20000000, time=0.00s) Projections: category, brand, name, quantity PROJECTION (rows=20000000, time=1.05s) Projections: #0, __internal_compress_string_uhugeint(#1), __internal_compress_string_uhugeint(#2), __internal_compress_string_uhugeint(#3) HASH_JOIN (rows=20000000, time=0.08s) Join Type: INNER; Conditions: product_id = product_id TABLE_SCAN (rows=20000000, time=0.04s) Table: sales TABLE_SCAN (rows=100000, time=0.00s) Table: products ``` this PR, 0.071 s: ``` HASH_GROUP_BY (rows=100000, time=0.01s) Groups: #0, #1, #2; Aggregates: combine_aggr(#3), combine_aggr(#4), combine_aggr(#5), combine_aggr(#6) PROJECTION (rows=100000, time=0.00s) Projections: category, brand, name, #3, #4, #5, #6 PROJECTION (rows=100000, time=0.00s) Projections: __internal_compress_string_uhugeint(#0), __internal_compress_string_uhugeint(#1), __internal_compress_string_uhugeint(#2), #3, #4, #5, #6 HASH_JOIN (rows=100000, time=0.01s) Join Type: INNER; Conditions: product_id = #0 TABLE_SCAN (rows=100000, time=0.00s) Table: products PROJECTION (rows=100000, time=0.00s) HASH_GROUP_BY (rows=100000, time=0.34s) Groups: #0; Aggregates: sum(#1) EXPORT_STATE, count_star() EXPORT_STATE, count(#1) EXPORT_STATE TABLE_SCAN (rows=20000000, time=0.04s) Table: sales ``` The 20M-row join, the string compression of the group columns and the 20M-row aggregate on three string columns are replaced by one 20M-row aggregate on the integer join key. Everything above it handles 100k rows. ## Relation to duckdb#28573 and duckdb#28716 Both apply to the new case without changes, because it goes through the same code path: - duckdb#28573: an exported aggregate whose argument can throw stays above an inner join. `sum(CAST(s AS INTEGER))` under a ROLLUP is not pushed, `sum(TRY_CAST(s AS INTEGER))` is (tested, with rows that only fail on the fact rows the join drops). - duckdb#28716: the lower aggregate created for an exported aggregate gets the capped estimate (its input and the product of its group domains). Float sums: as with the existing pushdown of a plain `GROUP BY`, the states are combined in a different order than without the pushdown, so a floating-point `sum` / `avg` can differ in the last digits. This is the same reassociation the pass already does, and that parallel aggregation does anyway. ## Tests - `test/optimizer/partial_aggregate_pushdown_exported_state.test` - plans: the pre-aggregate is below the join for ROLLUP, CUBE, GROUPING SETS (with a fact column among the groups), with GROUPING() and HAVING on top, for LEFT / RIGHT joins that preserve the aggregate side, for an aggregate the query exports itself, and for every one-argument aggregate function the cascade exports; - plans that must not change: the pass disabled; no aggregate with an argument (`count(*)` only); arguments from both sides; FILTER; two arguments; DISTINCT; ordered aggregate; an argument that can throw below an inner join (and its `TRY_CAST` twin that is pushed); an aggregate that can throw below an inner join (and the LEFT JOIN twin that is pushed); the aggregate side not preserved (`dim LEFT JOIN fact`, FULL join); a non-equality join condition; an aggregate below the aggregate side; no group from the other side; a single group column; a many-to-many join, where a plain GROUP BY takes the double-eager rewrite and the ROLLUP gets exactly one pre-aggregate; - results with the pushdown and with the pass disabled, at 1 and 8 threads: NULL join keys, NULL arguments, NULL group values next to the NULLs of the ROLLUP, unmatched fact rows, a duplicated dimension key, an empty dimension side, a fact side that is empty only at run time (so the levels combine over an empty pre-aggregate), outer joins, INTEGER / HUGEINT / DECIMAL / DOUBLE / VARCHAR arguments, all supported aggregate functions. - `test/optimizer/partial_aggregate_pushdown_exported_state_star_join.test`: a ROLLUP over a fact table joined to a filtered dimension and to the dimension that provides the groups (tpcds extension as data source, sf 0.01). The pre-aggregate sits between the two joins; rows equal to the plan without the pushdown at 1 and 8 threads. - `test/optimizer/partial_aggregate_pushdown_exported_state_overflow.test`: a fact table with the HUGEINT and the DECIMAL(38) maximum on the rows of one key and a dimension without that key, `threads = 1` and `join_filter_pushdown` disabled. `sum` and `avg` under ROLLUP, CUBE and GROUPING SETS, and an aggregate the query exports itself, return the rows main returns (all sums 0) instead of an overflow error; the plans show no aggregate below the join, while the BIGINT sum and the LEFT JOIN variant are pushed. - `test/optimizer/partial_aggregate_pushdown_exported_state_external.test_slow`: 10M fact rows over 2M keys under `memory_limit='300MB'`, so the pre-aggregate and the levels that combine its states run as external aggregates; same rows with and without the pushdown. A second part runs `min` over 800 byte strings (62500 states of about 50 MB per level) under `memory_limit='64MB'`: it only completes when the levels that combine states spill them, with and without the pushdown. - `test/configs/verify_aggregate_state_export.json`: the first test is added to the list of tests whose EXPLAIN output differs under `debug_verify_aggregate_state_export`, next to the other partial aggregate pushdown plan tests. The skip "null pointer dereference when the partial aggregate pushdown optimizer encounters STATE_EXPORT aggregates" is removed: `test/sql/tpcds/tpcds_sf0.test` passes without it and `test/optimizer/partial_aggregate_pushdown.test` only differs in EXPLAIN text, so it moves to that list. No expected plan or result of an existing test changes. ## Regression check Same binary, the new case switched off and on (a local switch that is not part of this PR), release build, arm64: - Plans: TPC-H SF1 / SF10 (22 queries each), TPC-DS SF1 / SF10 (99 each), JOB (113), ClickBench (43), at 1, 8 and 64 threads. Two TPC-DS plans change at each scale factor and thread count; no other plan changes. - Results: TPC-H SF1, TPC-DS SF1 and SF10 at 1, 8 and 64 threads: identical with the case off and on, except for one query whose plan does not change and whose result also differs between two runs with the case off. - User-mode instructions at 1 thread (`perf stat -e instructions:u`): no query above 1.05x, and the two queries whose plan changes use fewer instructions. Geomean per suite: TPC-H 1.00 / 1.00, TPC-DS 0.99 / 0.98, JOB 1.00, ClickBench 1.00. - Peak buffer memory of the two changed queries is not higher with the pushdown. On the current head, against the commit it branches from: - `scripts/plan_cost_runner.py` on `benchmark/tpch_plan_cost`: no differences. - TPC-DS SF1, all 99 queries at 1 and 64 threads: results identical. - All 398 plans of the suites above are byte-identical to the ones measured above, so the instruction counts carry over. - The whole fast test suite passes on a release build, as do `test/optimizer`, `test/sql/aggregate`, `test/sql/join`, `test/sql/tpch` and `test/sql/tpcds` including their slow tests. - `test/optimizer`, `test/sql/aggregate` and `test/sql/join` (825 tests including the slow ones and the new files) under the 66 configs in `test/configs`, on a clang assertion build compiled with `-Werror`: no test fails that does not fail the same way on the commit the branch starts from. 52 configs ran all 825 tests; the other 14 ran the 725 tests that are not slow plus the new slow test. The new memory-limited test was also run on its own under every config (passes, or is skipped by the config). - The new tests on a `STANDARD_VECTOR_SIZE=2` build and on an `ALTERNATIVE_VERIFY=1 DISABLE_STRING_INLINE=1 DESTROY_UNPINNED_BLOCKS=1 DISABLE_POINTER_SALT=1` build: pass (the memory-limited test requires full vectors and is skipped on the first). Limits of these measurements: - The primary signal is instructions at 1 thread. Queries whose plan does not change were measured once; the two whose plan changes and every query that read above 1.03x in the first pass were repeated (median of 3 to 5). The instruction counts were taken before the last commit and not repeated, because no plan changed. - The config sweep ran on a loaded shared machine. Slow tests that exceeded the 30 minute limit there (six under `disable_optimizer`, nine under `verify_compression`, none of them new) were not counted as failures and were not all repeated. `encryption` (needs httpfs for tests that open a database file) and `io_metrics` (not a test config) could not run here; `storage_compatibility` and `one_schema_per_test` share one database file between tests and were judged from a single-worker run. - Wall clock was not measured on a quiet machine. The timings of the example above were taken on a shared, loaded machine and are indicative only. - The state spilling of `combine_aggr` is covered by one memory-limited test (`min` over long strings); it was not benchmarked. Note: this PR was generated by an AI coding agent; it is a draft pending review by the author.
krlmlr
pushed a commit
that referenced
this pull request
Nov 18, 2026
…duckdb#28799) Conditional aggregation is very often written as an aggregate over a `CASE`: pivot-style reports that compute one column per status / weekday / channel (`sum(CASE WHEN status = 'shipped' THEN amount ELSE 0 END)`), counts per category (`count(CASE WHEN .. THEN 1 END)`, `sum(CASE WHEN .. THEN 1 ELSE 0 END)`), and it is what SQL generators emit for dialects without `FILTER`. For such an aggregate we currently compute the `CASE` into a vector for every row and update the aggregate state for every row, including the rows that only contribute a NULL or a 0. This PR adds an optimizer rule, `case_to_filter_aggregate`, that turns these aggregates into filtered aggregates. It builds on duckdb#28264: the state is only updated for the rows that match, and aggregates that share a condition evaluate it once per chunk. ### The rewrite | aggregate | becomes | when | |---|---|---| | `f(CASE WHEN c THEN x END)` | `f(x) FILTER (WHERE c)` | `f` is `sum`, `count`, `avg`, `min`, `max`, `bool_and` or `bool_or`: both forms are NULL (0 for `count`) if no row matches | | `f(CASE WHEN c THEN x END) FILTER (WHERE p)` | `f(x) FILTER (WHERE p AND c)` | `p` is not volatile and cannot throw: a row is aggregated if `p` and `c` are both true, in either form | | `f(DISTINCT CASE WHEN c THEN x END)` | `f(DISTINCT x) FILTER (WHERE c)` | the distinct values of the `CASE` are the distinct `x` of the matching rows and a NULL, which these aggregates ignore (see "DISTINCT" below for when the rule meets such an aggregate) | | `count(CASE WHEN c THEN k END)` | `count(*) FILTER (WHERE c)` | `k` is a non-NULL constant and the count is not `DISTINCT`: `count(DISTINCT CASE WHEN c THEN k END)` is 0 or 1 and becomes `count(DISTINCT k) FILTER (WHERE c)` | | `sum(CASE WHEN c THEN x ELSE 0 END)` | `coalesce(sum(x) FILTER (WHERE c), 0)` | `x` is provably not NULL | | `sum(CASE WHEN c THEN x ELSE 0 END)` | `CASE WHEN count(*) > count(*) FILTER (WHERE c) THEN coalesce(sum(x) FILTER (WHERE c), 0) ELSE sum(x) FILTER (WHERE c) END` | `x` is nullable: a group in which every row matches and every `x` is NULL is NULL (a sum over NULLs); any other group without a non-NULL matching `x` has a row that contributed the `ELSE 0`. The two counts are shared by all aggregates with the same condition | | `sum(CASE WHEN c THEN k ELSE 0 END)` | `count(*) FILTER (WHERE c) * k` | `k` is an integer constant | Where the aggregate returns a row for an empty input (no `GROUP BY`, or a grouping set without columns), the `ELSE 0` forms are wrapped in `CASE WHEN count(*) > 0 THEN .. END`, because the sum over no rows is NULL and not 0. ### Preconditions The aggregate is left alone if * it has an `ORDER BY` or exports its state; * it has a `FILTER` that is volatile or can throw (the rewritten filter `p AND c` evaluates `p` or `c` for a part of the rows only); * it is an `ELSE 0` form that is `DISTINCT` or has a `FILTER`: the helper counts of these forms count all rows of the group, not the rows that pass the filter, and the 0 is one of the distinct values; * it is a `DISTINCT` `sum` / `avg` over `FLOAT` / `DOUBLE` values: the distinct values are added in the order of their hash table, which changes with the values in it; * the `CASE` has more than one `WHEN`, or an `ELSE` other than NULL (or other than the constant 0 for `sum`); * the condition is volatile or contains a subquery; * the `THEN` expression can throw or is volatile: the `CASE` evaluates it for the matching rows only, the filtered aggregate evaluates its argument for every row (so `sum(CASE WHEN is_number THEN s::INTEGER END)` keeps its `CASE`); * the `THEN` expression is a `FLOAT` / `DOUBLE` constant under `sum` / `avg`: for a constant input vector `sum` adds `value * count` instead of adding the value `count` times, and that is another floating-point number (100 times `0.1` is `9.99999999999998`, `100 * 0.1` is `10.0`). A constant `THEN` is such a vector in every chunk, so the filtered sum would never have the digits of the `CASE` form; * it is a `min` / `max` over a string or a nested type: these read the vector that the `CASE` produces faster than a slice of the argument (two such aggregates over the same `CASE` measured 1.12x - 1.16x slower as filtered aggregates when 90% or more of the rows match). The NOT NULL proof for the bare `coalesce` form only follows constants, columns that are proven to be free of NULLs, `+`, `-`, `*` and casts between numeric types. Anything else (a struct field, a list element, a `TRY_CAST`) takes the form with the two counts. A column is proven with the existing `NotNullExpressionAnalyzer`: a filter or join condition that rejects NULL, or the statistics of the scanned table (which is also how a NOT NULL constraint is seen). The proof has to hold for the rows that the aggregate reads, so the statistics are only consulted where the optimizer itself relies on them: * not in a plan with a data-modifying CTE (`WITH u AS (UPDATE t SET x = NULL RETURNING x) SELECT sum(CASE WHEN c THEN x ELSE 0 END) FROM t`): the scan reads rows that the statistics did not describe when the plan was made. Statistics propagation is skipped for these plans for the same rea C31D son, and they always take the form with the two counts, which is correct for any input; * not when statistics propagation is disabled. The table statistics include the uncommitted changes of the running transaction, they still cover NULLs that another connection has overwritten (and checkpointed) while a transaction can read them, and a prepared statement that reads a table is planned again when it is executed. Rows that do not come from a table (a table function, a recursive CTE, the NULL-extended side of an outer join) are never proven NOT NULL. The rule runs after aggregate reuse and partial aggregate pushdown. These only match aggregates without a filter, so they still see the `CASE`: a pre-aggregation that is pushed below a join stays as it is today. ### DISTINCT and existing filters An existing filter combines with the condition for every `ELSE NULL` form, with or without `DISTINCT`. `p AND c` has the three-valued logic that the `CASE` form has: a row with a false or NULL `p`, or a false or NULL `c`, contributes nothing. Measured like the example below (instructions, one thread, 10M rows, 1000 groups), `sum(CASE WHEN c THEN x END) FILTER (WHERE p)` next to a `count` with the same condition and filter runs at 0.57x - 0.71x of the `CASE` form for selectivities of `c` and `p` between 1% and 99%; three aggregates over the same `CASE` and filter with 99% / 99% are at 0.81x. `DISTINCT` aggregates are turned into grouped aggregates over their argument by `distinct_aggregate_rewrite`, which runs before this rule (and before aggregate reuse and partial aggregate pushdown, which this rule has to follow). The rule therefore only meets a `DISTINCT` aggregate when that optimizer is disabled; with the default settings the plan of `count(DISTINCT CASE WHEN c THEN x END)` is unchanged, with and without `ROLLUP` / grouping sets. I did not move the rewrite in front of that optimizer: it plans the `CASE` form as a `GROUP BY` over the groups and the `CASE`, and `count(DISTINCT x) FILTER (WHERE c)` as a `GROUP BY` over the groups, `x` and `c`, which are more groups - the hand-written `FILTER` form measures 1.03x - 1.61x of the `CASE` form there. With `distinct_aggregate_rewrite` disabled the filtered aggregate only adds the matching rows to its distinct hash table: one to four `DISTINCT` aggregates run at 0.35x - 0.97x of the `CASE` form (grouped and ungrouped, 1% - 99% of the rows matching), with one exception, a single grouped `count(DISTINCT ..)` with 99% of the rows matching at 1.10x. ### Floating-point sums `sum` / `avg` over a `DOUBLE` argument that is not a constant are rewritten (a `FLOAT` argument is not: its cast to `DOUBLE` sits between the aggregate and the `CASE`). The filtered aggregate adds the same values in the same order as the `CASE` form (the rows of the `ELSE 0` add `0.0`, which changes nothing), so that the digits are the same when the query runs on one thread. With more threads the digits of a `DOUBLE` sum already vary from run to run, in either form. There is one exception, and it is behaviour of `sum` and not of the rewrite: the constant input vector mentioned above. The `CASE` computes a flat vector and thereby hides a constant vector from the sum; the filtered aggregate receives it, as a hand-written `sum(x) FILTER (WHERE c)` and a plain `sum(x)` do. A column arrives as constant vectors from the one-row side of a cross join and from constant or RLE compressed segments of a persistent table, which is not known when the plan is made. In these cases the result is the one of the hand-written `FILTER` and can differ from the `CASE` form in the low digits: `10.0` instead of `9.99999999999998` for 100 times `0.1` from a one-row table, `9.951039336363154` instead of `9.95103933636457` for the `avg` over half of 1M rows of an RLE compressed column (ungrouped, one thread). For comparison, a plain `sum(x)` over 1M rows of `0.1` is `100000.00000000087`, `100000.00000006249` or `100000.00000133288` today, depending on whether the table is persistent and compressed, persistent and uncompressed, or in memory. Leaving `sum` / `avg` over `DOUBLE` alone instead would give up the rewrite for these columns: the example below with a `DOUBLE` amount runs at 0.71x the instructions of the `CASE` form on one thread, the same as with the integer amount. The rule has its own optimizer type, so `SET disabled_optimizers = 'case_to_filter_aggregate'` switches it off. ### Structure The rule analyzes an aggregate once. The analysis names the condition, the value, the `ELSE` and an existing filter of the aggregate, and returns one result: the kind of the rewrite (an enum with one value per row of the table above), the bound filtered aggregate, the helper counts by name, the constant and the result type. The projection above the aggregate is built from that kind. For this the aggregate function rewriter accepts an analyzer next to a rule; the rules that are driven by a matcher and callbacks (`avg`, `sum`, `list`) and their code path are untouched. ### Two supporting changes A filter that selects every row should cost as little as no filter, otherwise a condition that happens to be always true makes the rewritten form slower than the `CASE`: * the statistics propagator drops an aggregate filter that is proven to be true for every row (this also applies to a hand-written `FILTER`), and reports it like the other expressions it removes, so that a column which only the filter read is pruned from the scan; * the aggregate filter data passes the payload on without slicing it when a chunk passes the filter completely. With these two, the worst shape I could construct is three numeric aggregates over the *same* `CASE` expression with 99% of the rows matching, at 1.04x the instructions of the `CASE` form; the same shape is at 1.00x with 90% and 0.89x with 50% of the rows matching. One fix belongs to the `DISTINCT` form. `GroupedAggregateHashTable::TryAddConstantGroups` adds the single group of an all-constant chunk; release builds take that path for chunks with more than one row, DEBUG builds for every chunk, including one without rows, for which it created a group that no row has. With `distinct_aggregate_rewrite` disabled, a user-written `SELECT count(DISTINCT 1) FILTER (WHERE x > 100) FROM t` without a matching row returned 1 instead of 0 in a DEBUG build, and `count(DISTINCT CASE WHEN x > 100 THEN 1 END)` is rewritten to that aggregate. The function now returns before the fast path for a chunk without rows. ### Example ```sql CREATE TABLE orders AS SELECT i % 1000 AS customer, i % 4 AS status, (i * 7) % 100 AS amount FROM range(50000000) t(i); SELECT customer, sum(CASE WHEN status = 0 THEN amount ELSE 0 END) AS open_amount, sum(CASE WHEN status = 1 THEN amount ELSE 0 END) AS shipped_amount, count(CASE WHEN status = 2 THEN 1 END) AS returned, avg(CASE WHEN status = 3 THEN amount END) AS avg_cancelled FROM orders GROUP BY customer; ``` Before: ``` PROJECTION CAST(#0 AS BIGINT), #1, #2, #3, #4 PERFECT_HASH_GROUP_BY Groups: #0 Aggregates: sum_no_overflow(#1), sum_no_overflow(#2), count(#3), avg(#4) PROJECTION customer, CASE WHEN ((status = 0)) THEN (amount) ELSE 0 END, CASE WHEN ((status = 1)) THEN (amount) ELSE 0 END, CASE WHEN ((status = 2)) THEN (1) ELSE NULL END, CASE WHEN ((status = 3)) THEN (amount) ELSE NULL END SEQ_SCAN orders ``` After: ``` PROJECTION #0, COALESCE(#1, 0), COALESCE(#2, 0), #3, #4 PERFECT_HASH_GROUP_BY Groups: #0 Aggregates: sum_no_overflow(#1) FILTER (WHERE #3), sum_no_overflow(#1) FILTER (WHERE #4), count() FILTER (WHERE #5), avg(#1) FILTER (WHERE #6) PROJECTION customer, amount, status = 0, status = 1, status = 2, status = 3 SEQ_SCAN orders ``` On this example (release build, in-memory table, best of 3 on a shared machine): 88 ms -> 65 ms with 8 threads, 0.64 s -> 0.45 s with 1 thread. A single conditional aggregate gains as well: `sum(CASE WHEN c THEN amount ELSE 0 END)` grouped by `customer` runs at 0.92x / 0.85x / 0.77x the instructions of the `CASE` form when 99% / 50% / 1% of the rows match. ### Tests * `test/optimizer/case_to_filter_aggregate.test`: every rewritten form with the rule disabled and enabled (same rows), grouped and ungrouped, empty inputs, groups without a matching row, NULL arguments and NULL conditions, conditions that are always true or always false, DECIMAL / HUGEINT / UBIGINT / FLOAT / DOUBLE arguments, `IF()`, a `THEN` that can throw, the plans (including the scan after a filter was dropped), and the interaction with partial aggregate pushdown. * `test/optimizer/case_to_filter_aggregate_exactness.test`: the shapes in which a naive rewrite is not exact: the grand total of ROLLUP / CUBE / GROUPING SETS over an empty input, `inf` / `nan` / `0.1` / `-0.0` constants, `DOUBLE` columns (also as a constant vector), struct fields and list elements of NOT NULL columns. * `test/optimizer/case_to_filter_aggregate_guards.test`: one negative test per precondition and results on edge inputs. * `test/optimizer/case_to_filter_aggregate_distinct_filter.test`: `DISTINCT` aggregates, aggregates with a `FILTER` and their combination, with the rule and `distinct_aggregate_rewrite` disabled and enabled: expected rows for duplicates, NULL values, NULL / false conditions and filters, constant `THEN` branches, empty inputs, inputs without a matching row, `ROLLUP` and grouping sets, and the plans (the combined filter, the `DISTINCT` count of a constant, the forms that are left alone). * `test/optimizer/case_to_filter_aggregate_statistics.test`: the NOT NULL proof against rows that change after the statistics were read: data-modifying CTEs (`UPDATE`, `INSERT`, upsert), uncommitted inserts / updates / `ALTER TABLE` of the running transaction, a prepared statement executed again after an insert, a transaction that still reads NULLs which another connection has overwritten and checkpointed, table functions, recursive CTEs, outer joins, views and materialized CTEs. The five files pin plans and are added to the skip list of `test/configs/verify_aggregate_state_export.json`, next to the other aggregate rewrite tests. ### Regression check Rule disabled vs enabled in the same binary (`SET disabled_optimizers`), TPC-H and TPC-DS at SF1 and SF10 and ClickBench: 1 of 22, 9 of 99 and 0 of 43 plans change. Results are identical on TPC-H and TPC-DS SF1 with 1 and 32 threads. Measured in user-mode instructions at 1 thread (`perf stat -e instructions:u`, median of 3), no query with a changed plan is above 1.01x; the geometric mean over the changed plans is 0.96 at SF1 and 0.93 at SF10. No expected plan of an existing test changes. This PR is independent of the other optimizer PRs I am opening. Note: this PR was generated by an AI coding agent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enhanced HTTP proxy configuration to automatically extract embedded credentials from proxy URLs and populate the corresponding username/password settings. This allows users to specify proxy credentials in a single URL string rather than managing separate settings.
Key Changes
ApplyHTTPProxyURL()function that parses proxy URLs and extractsuser[:password]@userinfo components, storing them inhttp_proxy_usernameandhttp_proxy_passwordsettings@characters in the authority component as credential separators, ignoring@in path/query/fragment componentsHTTPProxySetting::ResetGlobal()to check environment variables in order:https_proxy,HTTPS_PROXY,http_proxy,HTTP_PROXY(preferring lowercase to avoid CGI httpoxy ambiguity)@characters, colons in passwords), and malformed URLsImplementation Details
@host) or userinfo in query/fragment components are handled gracefully without affecting existing credentialsuser:@host) clears the password while preserving the usernamehttps://claude.ai/code/session_01GC39M53ozHzknQwf3M68BJ