Repository navigation
bubble skill: tree-preserving graft, resumable gated runs, reconciliation audit, and a generated replay log - #8
Merged
Conversation
On top of the merged bubble skill (PR #7) on v2.0. The merged cursor already scopes back-merges to one release (prev-release as 3rd arg), so this only adds what the skill still lacks: - reattach.sh: tree-preserving graft rewrite for step 2 — replace the de-merged merge with the linear reconstruction, re-commit only true descendants reusing trees. The SKILL's rebase --rebase-merges re-merges the upper back-merges and conflicts on generated files (e.g. src/main/config.cpp); the graft does not. - RUN-NOTES.md: the first v1.5 run (de-merged #16 into main-v1.5-variegata, checkpoint 358030a reproduced, 4 commits verbatim, tip tree == main tree), refs/pairing, next-run preview, and the unity-build ccache caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Each run, before rewriting, pushes the pre-de-merge tip to <branch>-<NN> (NN = DEMERGED_SO_FAR, 0-padded) with a create-only push: - preserves the original state that the run's force-push overwrites (numbered, immutable audit trail: -00 = original main, -01 after one de-merge, ...), and - doubles as a concurrency gate: the push fails if the ref already exists, so a second concurrent run on the same step aborts. bubble-cursor.sh emits GATE_BRANCH; SKILL.md step 0 does the gated push. Pushed main-v1.5-variegata-00 (run #1 original input = main-dag, tree==main). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
…d-budget Skill corrections and additions from runs #1–#2: - Gate (step 0): detect creation positively ([new branch] in push output); a same-value push to an existing ref is a no-op success, so don't rely on 'push fails if exists'. Abort if the snapshot branch already existed. - Conflicts (step 3): never -X theirs/ours; every conflict and red herring is reconciled manually by the agent using the merge commit as the oracle. - De-merge (step 2): tree-preserving graft (reattach.sh), not rebase --rebase-merges (which re-merges upper back-merges and conflicts). Base each run on origin/<branch> (the previous result), never main-dag; NEXT_MERGE is that branch's back-merge SHA. - Build budget: drop session-length assumption; use a dedicated build/test worktree separate from the de-merge worktree; note tests run serially (unittest has no parallel option) — shard for speed. - Reconciliation audit (reconcile-audit.sh + section): the reconcile commit's diff is the merge's cross-side resolutions; review non-generated deletions + semantic changes, record a factual note, do not gate (merge tree is oracle). - RUN-NOTES: run #2 record + per-commit build times. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
- Proof of work: distinguish 'perfect' (each replayed PR carried verbatim) from 'faithful' (reproduces the merge's end state in spirit — empties dropped, generated files regenerated, cross-side reconciliation in the reconcile commit). - Invariant 1 / hard rules / checkpoints: the publish gate is tip tree == START_TREE (tree(main) captured at run start), not live main. main only fast-forwards (assumed never force-pushed); new upstream commits are picked up by the next run via step 1, which recomputes START_TREE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Restarts/reclaims kill in-progress runs; re-deriving redoes the manual conflict resolution (risking drift). Add a per-run resume point: - bubble-cursor.sh emits WIP_BRANCH (<branch>-NN-wip) alongside GATE_BRANCH. - Step 2: push RECON (the verified reconstruction, tree == MERGE_TREE) to -NN-wip BEFORE building. - Step 0: if the gate exists (interrupted prior run, runs are sequential) and -NN-wip exists, resume from RECON and skip the de-merge (steps 1-3); else redo. - Step 5: delete -NN-wip on successful publish (gate -NN remains as audit trail). A restart now costs at most a rebuild, never the manual de-merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
- 'Escalate' a hard break means MORE work, not stopping: forward-port the owning fix, or add a transient in-segment measure that's gone at the checkpoint per construction. Inter-segment boundaries (checkpoints) therefore stay clean. - Greenness flows from the merge commit: the reconcile/checkpoint commit reproduces MERGE_TREE and the upstream merge was CI-green, so it inherits that greenness. If still red, split into a green part and a minimal red part — the red is almost always fixed/reverted by mainline soon, so a later run absorbs it. - Use verbose commit messages for all synthesized commits (reconcile, forward-ports, transient measures, splits) incl. expected mainline resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Reinforce in the intro and the relationship section that build-linear-branch is deprecated and kept only for reference; use bubble for new work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
…h run REPLAY-LOG.md lists, oldest-first (git log --reverse), every commit replayed since the v1.5 bifurcation. Each entry: original commit (L1) + its -dag commit and per-run derived commit (L2, with role: replayed / empty-absorbed / reattached) + conflict resolution (L3); synthetic commits (reconcile/checkpoint) inline. All links to github.com/krlmlr/duckdb. Seeded with runs #1-#2. SKILL.md: new 'Replay log (provenance)' section + step 5 now updates the log before publishing, so it grows with every run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
… duckdb/duckdb - gen-replay-log.sh generates REPLAY-LOG.md from branch state: every first-parent commit since the bifurcation (943 PRs + 16 back-merges), grouped by run (de-merge === run). Identity = upstream PR (#N -> duckdb/duckdb) + the -dag commit (-> krlmlr/duckdb); completed runs overlaid with derived sha + role (replayed / empty-absorbed / pending). Regenerated each run (skill step 5). - Drop merge-ordinal naming (#16/#17/#18) in favour of run numbers. - PR references now link to duckdb/duckdb with #N notation; commit SHAs link to krlmlr/duckdb. origin/main (fork main) is not used as a source — it isn't a faithful mirror of duckdb/duckdb (e.g. duckdb#20369 absent); -dag + PR are canonical. - Absorbed commits: noted honestly (change already on the advanced v1.5 base; no single attributable v1.5 PR), not a fabricated link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
… role
- Every completed run now annotates EVERY commit with a krlmlr/duckdb SHA + role:
replayed (de-merged that run), copied (reattached above the de-merge point,
tree-verbatim, new SHA), or empty (absorbed). Built from full per-run result
branches (R{N}ALL) in addition to the de-merged segments (R{N}).
- Absorbed commits: written once as 'from run #N' (absorption persists for all
later runs, so later runs aren't repeated).
- Back-merge checkpoints annotated per run: linearized / copied (merge) / gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Match the normal commit layout: the back-merge's -dag SHA is a sub-bullet under the checkpoint title rather than inline after an em-dash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr
added a commit
that referenced
this pull request
Jul 28, 2026
…tion audit, and a generated replay log (#8) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
krlmlr
added a commit
that referenced
this pull request
Jul 28, 2026
…tion audit, and a generated replay log (#8) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
pull Bot
pushed a commit
that referenced
this pull request
Sep 10, 2026
<details>
<summary>WARNING: ThreadSanitizer: data race; Read of size 8</summary>
```c++
Filters: test/sql/show_select/summarize_subquery.test
[0/1] (0%): test/sql/show_select/summarize_subquery.test==================
WARNING: ThreadSanitizer: data race (pid=80064)
Read of size 8 at 0x00010708e758 by thread T10:
#0 duckdb::ArenaAllocator::AlignNext() <null> (libduckdb.dylib:arm64+0x2a37af0)
#1 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x3184418)
#2 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
#3 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
#4 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
#5 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822 (libduckdb.dylib:arm64+0x318ca64)
#6 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
#7 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a rm64+0x2458dac)
#8 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb. dylib:arm64+0x2457ea0)
#9 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2459624)
#10 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggreg ate.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
#11 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2 448dac)
...
Previous write of size 8 at 0x00010708e758 by thread T7:
#0 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x31844a0)
#1 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
#2 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
#3 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
#4 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822 (libduckdb.dylib:arm64+0x318ca64)
#5 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
#6 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a rm64+0x2458dac)
#7 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb. dylib:arm64+0x2457ea0)
#8 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2459624)
#9 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggrega te.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
#10 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2 448dac)
...
Location is heap block of size 56 at 0x00010708e740 allocated by main thread:
#0 operator new(unsigned long) <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x91650)
#1 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) <null> (libduckdb.dylib:arm64+0x2a37830)
#2 std::__1::vector<duckdb_tdigest::Centroid, duckdb::arena_stl_allocator<duckdb_tdigest::Centroid>>::reserve(unsigned long) vector.h:1100 (libduckdb.dyli b:arm64+0x3180da0)
#3 void duckdb::(anonymous namespace)::ApproxQuantileOperation::Operation<long long, duckdb::(anonymous namespace)::ApproxQuantileState, duckdb::(anonymou s namespace)::ApproxQuantileScalarOperation>(duckdb::(anonymous namespace)::ApproxQuantileState&, long long const&, duckdb::AggregateUnaryInput&) approximate_ quantile.cpp:122 (libduckdb.dylib:arm64+0x318ccd4)
#4 void duckdb::AggregateFunction::UnaryScatterUpdate<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::Approx QuantileScalarOperation>(duckdb::Vector*, duckdb::AggregateInputData&, unsigned long long, duckdb::Vector&, unsigned long long) aggregate_function.hpp:782 (li bduckdb.dylib:arm64+0x318c454)
#5 duckdb::RowOperations::UpdateStates(duckdb::RowOperationsState&, duckdb::AggregateObject&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long, duc kdb::optional_ptr<duckdb::ClusteredAggr const, true>) aggregate_function.hpp (libduckdb.dylib:arm64+0x166ed24)
#6 duckdb::GroupedAggregateHashTable::UpdateAggregates(duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsigned long lon g>> const&, unsigned long long, bool) <null> (libduckdb.dylib:arm64+0x240d234)
#7 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::Vector&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1 ::allocator<unsigned long long>> const&) <null> (libduckdb.dylib:arm64+0x240f000)
#8 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsig ned long long>> const&) <null> (libduckdb.dylib:arm64+0x240cb30)
#9 duckdb::RadixPartitionedHashTable::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&, duckdb::DataChunk&, duckdb::vector<u nsigned long long, false, std::__1::allocator<unsigned long long>> const&) const <null> (libduckdb.dylib:arm64+0x2455190)
#10 duckdb::PhysicalHashAggregate::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const physical_hash_aggregate.cpp:466 ( libduckdb.dylib:arm64+0x212bafc)
SUMMARY: ThreadSanitizer: data race (libduckdb.dylib:arm64+0x2a37af0) in duckdb::ArenaAllocator::AlignNext()+0x2c
==================
```
</details>
`SUMMARIZE` calculates three `approx_quantile` aggregates (`q25`, `q50`,
`q75`). Each aggregate owns a TDigest whose vectors use the hash table’s
`ArenaAllocator`.
Two hash-aggregate worker threads concurrently finalized separate
TDigest states:
- Both called `TDigest::process()` → `updateCumulative()` →
`vector::reserve()`.
- The states shared the same non-thread-safe `ArenaAllocator`.
- One thread read allocator position metadata in
`ArenaAllocator::AlignNext()` while another wrote it.
- The shared arena had originally been allocated during
`approx_quantile` aggregation on the main thread.
Result: a race in arena allocation during parallel `approx_quantile`
finalization, reported at `ArenaAllocator::AlignNext()`.
It was intermittent because `reserve()` only occurs when vector capacity
must grow, and the worker calls had to overlap closely enough.
The fix reserves the cumulative buffer and sufficient centroid merge
capacity when the TDigest is constructed and its allocator is still used
by only one thread. Parallel finalization then operates entirely within
already allocated buffers and no longer mutates the shared arena.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the merged
bubbleskill (PR #7, onv2.0) and folds in everything learned running it for real onv1.5-variegata(runs #1–#2, both published), plus the supporting tooling. Skill/docs only — no product code; basev2.0.What
bubbledoesMaintains a perpetual
main-<release>branch (e.g.main-v1.5-variegata) that mirrorsmain's tree exactly while progressively de-merging the release line's back-merges into a linear, bisectable spine — one back-merge per run (de-merge === run), each checkpointed against the real merge tree, force-pushed only after asserting the tip tree is unchanged.New tooling
reattach.sh— tree-preserving graft for step 2: replace the de-merged merge with the linear reconstruction and re-commit only its true descendants, reusing each tree verbatim. Replacesrebase --rebase-merges, which re-merges the upper back-merges and conflicts on generated files (e.g.src/main/config.cpp).reconcile-audit.sh— classifies a reconcile commit's delta (generated churn vs reviewable deletions/semantic changes). Review is documentation, not a gate.gen-replay-log.sh+REPLAY-LOG.md— generated, oldest-first manifest of all 959 first-parent commits since the bifurcation (943 PRs + 16 back-merges), grouped by run. Per commit: the upstream PR (→duckdb/duckdb) and its-dagcommit (→krlmlr/duckdb); for every completed run a SHA + role — replayed (de-merged that run), copied (reattached above the de-merge point, tree-verbatim), or empty (absorbed) (from run #N). Regenerated each run.bubble-cursor.sh— now also emitsGATE_BRANCHandWIP_BRANCH.SKILL.md refinements (driven by the real runs)
-NNsnapshot (immutable audit trail, claimed by positive[new branch]detection — a same-value push is a no-op success) plus-NN-wipholding the verified reconstruction, so a restart/reclaim resumes without redoing the manual de-merge.-NN-wipis deleted on publish.-X theirs/-X ours; the merge tree is the oracle; generated files are regenerated, not hand-merged.main-dag.START_TREE), not livemain; faithful vs perfect defined.unittesthas no parallel option — shard for speed).build-linear-branchmarked deprecated (superseded bybubble).Evidence (published on
main-v1.5-variegata)358030afreproduced; tip tree ==main.DEMERGED=2, REMAINING=14.-00(originalmain-dag),-01(run avoid copy #1 result).🤖 Generated with Claude Code