8000 bubble skill: tree-preserving graft, resumable gated runs, reconciliation audit, and a generated replay log by krlmlr · Pull Request #8 · krlmlr/duckdb · GitHub
[go: up one dir, main page]

Skip to content

bubble skill: tree-preserving graft, resumable gated runs, reconciliation audit, and a generated replay log - #8

Merged
krlmlr merged 11 commits into
v2.0from
claude/peaceful-hopper-02ofsr
Jun 21, 2026
Merged

krlmlr merged 11 commits into
v2.0from
claude/peaceful-hopper-02ofsr

Conversation

@krlmlr
@krlmlr krlmlr commented Jun 21, 2026 •
Copy link
Copy Markdown
Owner

Builds on the merged bubble skill (PR #7, on v2.0) and folds in everything learned running it for real on v1.5-variegata (runs #1–#2, both published), plus the supporting tooling. Skill/docs only — no product code; base v2.0.

What bubble does

Maintains a perpetual main-<release> branch (e.g. main-v1.5-variegata) that mirrors main's tree exactly while progressively de-merging the release line's back-merges into a linear, bisectable spine — one back-merge per run (de-merge === run), each checkpointed against the real merge tree, force-pushed only after asserting the tip tree is unchanged.

New tooling

  • reattach.sh — tree-preserving graft for step 2: replace the de-merged merge with the linear reconstruction and re-commit only its true descendants, reusing each tree verbatim. Replaces rebase --rebase-merges, which re-merges the upper back-merges and conflicts on generated files (e.g. src/main/config.cpp).
  • reconcile-audit.sh — classifies a reconcile commit's delta (generated churn vs reviewable deletions/semantic changes). Review is documentation, not a gate.
  • gen-replay-log.sh + REPLAY-LOG.md — generated, oldest-first manifest of all 959 first-parent commits since the bifurcation (943 PRs + 16 back-merges), grouped by run. Per commit: the upstream PR (→ duckdb/duckdb) and its -dag commit (→ krlmlr/duckdb); for every completed run a SHA + role — replayed (de-merged that run), copied (reattached above the de-merge point, tree-verbatim), or empty (absorbed) (from run #N). Regenerated each run.
  • bubble-cursor.sh — now also emits GATE_BRANCH and WIP_BRANCH.

SKILL.md refinements (driven by the real runs)

  • Resumable, gated runs: a per-run -NN snapshot (immutable audit trail, claimed by positive [new branch] detection — a same-value push is a no-op success) plus -NN-wip holding the verified reconstruction, so a restart/reclaim resumes without redoing the manual de-merge. -NN-wip is deleted on publish.
  • Manual reconciliation only — never -X theirs/-X ours; the merge tree is the oracle; generated files are regenerated, not hand-merged.
  • Base = the previous run's branch, never main-dag.
  • Escalate by trying harder, not stopping — forward-port the owning fix, or add a transient in-segment measure that vanishes at the checkpoint per construction (boundaries stay clean). Greenness flows from the (CI-green) merge commit; if a cleanup commit is still red, split off a minimal, verbosely documented red part that mainline soon fixes/reverts.
  • Tree ≡ main-at-run-start (START_TREE), not live main; faithful vs perfect defined.
  • Dedicated build/test worktree separate from the de-merge worktree; tests run serially (unittest has no parallel option — shard for speed).
  • build-linear-branch marked deprecated (superseded by bubble).

Evidence (published on main-v1.5-variegata)

🤖 Generated with Claude Code

claude added 11 commits June 21, 2026 10:45
On top of the merged bubble skill (PR #7) on v2.0. The merged cursor already
scopes back-merges to one release (prev-release as 3rd arg), so this only adds
what the skill still lacks:

- reattach.sh: tree-preserving graft rewrite for step 2 — replace the de-merged
  merge with the linear reconstruction, re-commit only true descendants reusing
  trees. The SKILL's rebase --rebase-merges re-merges the upper back-merges and
  conflicts on generated files (e.g. src/main/config.cpp); the graft does not.
- RUN-NOTES.md: the first v1.5 run (de-merged #16 into main-v1.5-variegata,
  checkpoint 358030a reproduced, 4 commits verbatim, tip tree == main tree),
  refs/pairing, next-run preview, and the unity-build ccache caveat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Each run, before rewriting, pushes the pre-de-merge tip to
<branch>-<NN> (NN = DEMERGED_SO_FAR, 0-padded) with a create-only push:
- preserves the original state that the run's force-push overwrites
  (numbered, immutable audit trail: -00 = original main, -01 after one
  de-merge, ...), and
- doubles as a concurrency gate: the push fails if the ref already
  exists, so a second concurrent run on the same step aborts.

bubble-cursor.sh emits GATE_BRANCH; SKILL.md step 0 does the gated push.
Pushed main-v1.5-variegata-00 (run #1 original input = main-dag, tree==main).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
…d-budget

Skill corrections and additions from runs #1–#2:
- Gate (step 0): detect creation positively ([new branch] in push output);
  a same-value push to an existing ref is a no-op success, so don't rely on
  'push fails if exists'. Abort if the snapshot branch already existed.
- Conflicts (step 3): never -X theirs/ours; every conflict and red herring is
  reconciled manually by the agent using the merge commit as the oracle.
- De-merge (step 2): tree-preserving graft (reattach.sh), not
  rebase --rebase-merges (which re-merges upper back-merges and conflicts).
  Base each run on origin/<branch> (the previous result), never main-dag;
  NEXT_MERGE is that branch's back-merge SHA.
- Build budget: drop session-length assumption; use a dedicated build/test
  worktree separate from the de-merge worktree; note tests run serially
  (unittest has no parallel option) — shard for speed.
- Reconciliation audit (reconcile-audit.sh + section): the reconcile commit's
  diff is the merge's cross-side resolutions; review non-generated deletions +
  semantic changes, record a factual note, do not gate (merge tree is oracle).
- RUN-NOTES: run #2 record + per-commit build times.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
- Proof of work: distinguish 'perfect' (each replayed PR carried verbatim) from
  'faithful' (reproduces the merge's end state in spirit — empties dropped,
  generated files regenerated, cross-side reconciliation in the reconcile commit).
- Invariant 1 / hard rules / checkpoints: the publish gate is tip tree ==
  START_TREE (tree(main) captured at run start), not live main. main only
  fast-forwards (assumed never force-pushed); new upstream commits are picked up
  by the next run via step 1, which recomputes START_TREE.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Restarts/reclaims kill in-progress runs; re-deriving redoes the manual
conflict resolution (risking drift). Add a per-run resume point:
- bubble-cursor.sh emits WIP_BRANCH (<branch>-NN-wip) alongside GATE_BRANCH.
- Step 2: push RECON (the verified reconstruction, tree == MERGE_TREE) to
  -NN-wip BEFORE building.
- Step 0: if the gate exists (interrupted prior run, runs are sequential) and
  -NN-wip exists, resume from RECON and skip the de-merge (steps 1-3); else redo.
- Step 5: delete -NN-wip on successful publish (gate -NN remains as audit trail).
A restart now costs at most a rebuild, never the manual de-merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
- 'Escalate' a hard break means MORE work, not stopping: forward-port the owning
  fix, or add a transient in-segment measure that's gone at the checkpoint per
  construction. Inter-segment boundaries (checkpoints) therefore stay clean.
- Greenness flows from the merge commit: the reconcile/checkpoint commit
  reproduces MERGE_TREE and the upstream merge was CI-green, so it inherits that
  greenness. If still red, split into a green part and a minimal red part — the
  red is almost always fixed/reverted by mainline soon, so a later run absorbs it.
- Use verbose commit messages for all synthesized commits (reconcile,
  forward-ports, transient measures, splits) incl. expected mainline resolution.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Reinforce in the intro and the relationship section that build-linear-branch is
deprecated and kept only for reference; use bubble for new work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
…h run

REPLAY-LOG.md lists, oldest-first (git log --reverse), every commit replayed
since the v1.5 bifurcation. Each entry: original commit (L1) + its -dag commit
and per-run derived commit (L2, with role: replayed / empty-absorbed /
reattached) + conflict resolution (L3); synthetic commits (reconcile/checkpoint)
inline. All links to github.com/krlmlr/duckdb. Seeded with runs #1-#2.

SKILL.md: new 'Replay log (provenance)' section + step 5 now updates the log
before publishing, so it grows with every run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
… duckdb/duckdb

- gen-replay-log.sh generates REPLAY-LOG.md from branch state: every first-parent
  commit since the bifurcation (943 PRs + 16 back-merges), grouped by run
  (de-merge === run). Identity = upstream PR (#N -> duckdb/duckdb) + the -dag
  commit (-> krlmlr/duckdb); completed runs overlaid with derived sha + role
  (replayed / empty-absorbed / pending). Regenerated each run (skill step 5).
- Drop merge-ordinal naming (#16/#17/#18) in favour of run numbers.
- PR references now link to duckdb/duckdb with #N notation; commit SHAs link to
  krlmlr/duckdb. origin/main (fork main) is not used as a source — it isn't a
  faithful mirror of duckdb/duckdb (e.g. duckdb#20369 absent); -dag + PR are canonical.
- Absorbed commits: noted honestly (change already on the advanced v1.5 base;
  no single attributable v1.5 PR), not a fabricated link.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
… role

- Every completed run now annotates EVERY commit with a krlmlr/duckdb SHA + role:
  replayed (de-merged that run), copied (reattached above the de-merge point,
  tree-verbatim, new SHA), or empty (absorbed). Built from full per-run result
  branches (R{N}ALL) in addition to the de-merged segments (R{N}).
- Absorbed commits: written once as 'from run #N' (absorption persists for all
  later runs, so later runs aren't repeated).
- Back-merge checkpoints annotated per run: linearized / copied (merge) / gone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
Match the normal commit layout: the back-merge's -dag SHA is a sub-bullet under
the checkpoint title rather than inline after an em-dash.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
@krlmlr krlmlr changed the title bubble: add reattach.sh (tree-preserving graft) + first-run notes bubble skill: tree-preserving graft, resumable gated runs, reconciliation audit, and a generated replay log Jun 21, 2026
@krlmlr
krlmlr merged commit c527838 into v2.0 Jun 21, 2026
3 of 6 checks passed
@krlmlr
krlmlr deleted the claude/peaceful-hopper-02ofsr branch June 21, 2026 15:01
krlmlr added a commit that referenced this pull request Jul 28, 2026
…tion audit, and a generated replay log (#8)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
krlmlr added a commit that referenced this pull request Jul 28, 2026
…tion audit, and a generated replay log (#8)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
pull Bot pushed a commit that referenced this pull request Sep 10, 2026
<details>
<summary>WARNING: ThreadSanitizer: data race; Read of size 8</summary>

```c++
  Filters: test/sql/show_select/summarize_subquery.test

  [0/1] (0%): test/sql/show_select/summarize_subquery.test==================
  WARNING: ThreadSanitizer: data race (pid=80064)
    Read of size 8 at 0x00010708e758 by thread T10:
      #0 duckdb::ArenaAllocator::AlignNext() <null> (libduckdb.dylib:arm64+0x2a37af0)
      #1 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x3184418)
      #2 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
      #3 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
      #4 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano  nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
      #5 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant  ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822   (libduckdb.dylib:arm64+0x318ca64)
      #6 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r  ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
      #7 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a  rm64+0x2458dac)
      #8 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.  dylib:arm64+0x2457ea0)
      #9 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const   <null> (libduckdb.dylib:arm64+0x2459624)
      #10 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggreg  ate.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
      #11 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2  448dac)
...

    Previous write of size 8 at 0x00010708e758 by thread T7:
      #0 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x31844a0)
      #1 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
      #2 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
      #3 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano  nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
      #4 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant  ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822   (libduckdb.dylib:arm64+0x318ca64)
      #5 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r  ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
      #6 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a  rm64+0x2458dac)
      #7 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.  dylib:arm64+0x2457ea0)
      #8 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const   <null> (libduckdb.dylib:arm64+0x2459624)
      #9 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggrega  te.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
      #10 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2  448dac)
...

    Location is heap block of size 56 at 0x00010708e740 allocated by main thread:
      #0 operator new(unsigned long) <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x91650)
      #1 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) <null> (libduckdb.dylib:arm64+0x2a37830)
      #2 std::__1::vector<duckdb_tdigest::Centroid, duckdb::arena_stl_allocator<duckdb_tdigest::Centroid>>::reserve(unsigned long) vector.h:1100 (libduckdb.dyli  b:arm64+0x3180da0)
      #3 void duckdb::(anonymous namespace)::ApproxQuantileOperation::Operation<long long, duckdb::(anonymous namespace)::ApproxQuantileState, duckdb::(anonymou  s namespace)::ApproxQuantileScalarOperation>(duckdb::(anonymous namespace)::ApproxQuantileState&, long long const&, duckdb::AggregateUnaryInput&) approximate_  quantile.cpp:122 (libduckdb.dylib:arm64+0x318ccd4)
      #4 void duckdb::AggregateFunction::UnaryScatterUpdate<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::Approx  QuantileScalarOperation>(duckdb::Vector*, duckdb::AggregateInputData&, unsigned long long, duckdb::Vector&, unsigned long long) aggregate_function.hpp:782 (li  bduckdb.dylib:arm64+0x318c454)
      #5 duckdb::RowOperations::UpdateStates(duckdb::RowOperationsState&, duckdb::AggregateObject&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long, duc  kdb::optional_ptr<duckdb::ClusteredAggr const, true>) aggregate_function.hpp (libduckdb.dylib:arm64+0x166ed24)
      #6 duckdb::GroupedAggregateHashTable::UpdateAggregates(duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsigned long lon  g>> const&, unsigned long long, bool) <null> (libduckdb.dylib:arm64+0x240d234)
      #7 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::Vector&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1  ::allocator<unsigned long long>> const&) <null> (libduckdb.dylib:arm64+0x240f000)
      #8 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsig  ned long long>> const&) <null> (libduckdb.dylib:arm64+0x240cb30)
      #9 duckdb::RadixPartitionedHashTable::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&, duckdb::DataChunk&, duckdb::vector<u  nsigned long long, false, std::__1::allocator<unsigned long long>> const&) const <null> (libduckdb.dylib:arm64+0x2455190)
      #10 duckdb::PhysicalHashAggregate::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const physical_hash_aggregate.cpp:466 (  libduckdb.dylib:arm64+0x212bafc)

  SUMMARY: ThreadSanitizer: data race (libduckdb.dylib:arm64+0x2a37af0) in duckdb::ArenaAllocator::AlignNext()+0x2c
  ==================
```
</details>

`SUMMARIZE` calculates three `approx_quantile` aggregates (`q25`, `q50`,
`q75`). Each aggregate owns a TDigest whose vectors use the hash table’s
`ArenaAllocator`.

Two hash-aggregate worker threads concurrently finalized separate
TDigest states:

- Both called `TDigest::process()` → `updateCumulative()` →
`vector::reserve()`.
- The states shared the same non-thread-safe `ArenaAllocator`.
- One thread read allocator position metadata in
`ArenaAllocator::AlignNext()` while another wrote it.
- The shared arena had originally been allocated during
`approx_quantile` aggregation on the main thread.

Result: a race in arena allocation during parallel `approx_quantile`
finalization, reported at `ArenaAllocator::AlignNext()`.

It was intermittent because `reserve()` only occurs when vector capacity
must grow, and the worker calls had to overlap closely enough.

The fix reserves the cumulative buffer and sufficient centroid merge
capacity when the TDigest is constructed and its allocator is still used
by only one thread. Parallel finalization then operates entirely within
already allocated buffers and no longer mutates the shared arena.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0