8000 bubble skill: incrementally linearise main against a release line (supersedes build-linear-branch) by krlmlr · Pull Request #7 · krlmlr/duckdb · GitHub
[go: up one dir, main page]

Skip to content

bubble skill: incrementally linearise main against a release line (supersedes build-linear-branch) - #7

Merged
krlmlr merged 14 commits into
v2.0from
claude/awesome-wright-qajbai
Jun 21, 2026
Merged

krlmlr merged 14 commits into
v2.0from
claude/awesome-wright-qajbai

Conversation

@krlmlr
@krlmlr krlmlr commented Jun 20, 2026 •
Copy link
Copy Markdown
Owner

Summary

Adds the bubble skill — the validated approach for producing a release-rooted, bisectable, linear vendoring spine for the R package — and marks the earlier build-linear-branch skill outdated (kept for reference). Both live under .claude/skills/.

bubble maintains a perpetual main-<release> branch (e.g. main-v1.5-variegata) whose tree is always identical to main, progressively de-merging the release branch's back-merges. Each self-contained, restartable run advances the bifurcation one back-merge ahead (onto that merge's second parent), reproducing that merge's tree as a deterministic checkpoint, keeping the chain green (release build + fast tests, shared ccache), and force-pushing (--force-with-lease) only after asserting the tip tree is unchanged. One branch per minor release.

Why this supersedes build-linear-branch

build-linear-branch cherry-picks each dev PR onto a fixed release base (flat(<TAG>)). Validation on this branch showed that is the expensive, drift-prone path:

  • It sources from -flat branches, whose merge base is the first-parent bifurcation (Jan) — the back-merge convergence is discarded, inflating apparent divergence from ~1,600 to ~8,500 files.
  • Working from the bifurcation, it must bridge the full v1.5.4→v2.0 divergence one conflicted commit at a time with no re-sync; conflicts recur on roughly every overlapping PR and resolutions can silently drift from the target.
  • The conflicts that matter (external callers, generated files, ABI/type drift) are invisible to cherry-pick and surface only at build/test — they need regeneration or forward-porting, not conflict-marker resolution.

The -dag branches (from #6) preserve the real merge topology and expose the real (June) merge base. bubble advances the bifurcation along those real back-merges, so conflict is confined to the small v1.5-side delta between consecutive checkpoints and drift cannot go undetected.

Key validated findings (folded into the skills)

  • Merge base ≠ bifurcation point. Back-merges advance the merge base to within ~174 files of v1.5.4, while the first-parent bifurcation stays in January; -flat only sees the latter. "Bubbling" onto a merge tree (8k files away) would move the bifurcation — disqualified; using the merge base keeps v1.5.4 as the root.
  • Sequential, not parallelisable. Each segment's PRs assume the accumulated main chain; anchoring batches independently on their 2nd parent drifts by exactly the missing chain — measured 0 → ~3,778 files as the genuine main-only run between back-merges lengthens.
  • Generated files (settings/config.cpp, PEG grammar, serialization, enum_util) must be regenerated, never merged; -X theirs mis-maps their indices and still compiles.
  • Per-file numstat (before vs after) is the proof of work and sharpest drift detector: clean de-merges diff empty; the one observed drift localised to src/main/config.cpp (+2 −1 → +111 −136), the generated settings-index churn, flagged for regeneration.

What's here (all under .claude/skills/)

  • bubble/SKILL.md, bubble/bubble-cursor.sh — the current approach: model + invariants (tree ≡ main, green per commit), the one-merge-per-run de-merge with merge-tree checkpoints, conflict resolution in the main agent (-X theirs provisional → regenerate generated files / forward-port real breaks), commit-optimistically-then-test-until-green, --force-with-lease after the tip-tree assertion, and the per-file numstat proof of work.
  • build-linear-branch/* — OUTDATED: a > [!WARNING] banner and the frontmatter point to bubble; the release-vs-reldebug gate measurements and historical notes are retained for reference.

Depends conceptually on #6 (the -flat/-dag mirror branches and the Upstream-commit trailer convention).

🤖 Generated with Claude Code

@krlmlr
krlmlr changed the base branch from main to v2.0 June 20, 2026 17:54
Tooling to build and resume `v<DEV>-v<TAG>-linear` branches (e.g.
`v2.0-v1.5.4-linear`): a linear, per-commit test-gated replay of a dev
`-flat` branch onto the flat commit for a release tag. Companion to the
`add-flat-branch` skill; reuses its `Upstream-commit` trailer as the restart
cursor.

- `.claude/skills/build-linear-branch/SKILL.md` — the agentic loop:
  cherry-pick + semantic conflict resolution in the main session, then
  build -> prune -> test per commit in disposable worktrees with shared
  ccache, push only when green (CI re-verifies), scrub worktrees. Restartable:
  the branch tip is the state, resumed via the trailer.
- `linear-todo.sh` — resume-aware cursor: derives BASE from a release tag,
  TARGET name, and remaining source commits oldest-first.
- `linear-build-test.sh` — phased green gate; the build phase prunes object
  files after linking so a second worktree can build while this one tests.
- `linear-disk-guard.sh` — admission control modelling one full build plus
  (k-1) pruned trees; caps concurrent worktrees to fit the disk.
- `.gitignore` — keep `.claude/skills/` under version control.

Helper scripts live beside the skill under `.claude/skills/build-linear-branch/`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
@krlmlr
krlmlr force-pushed the claude/awesome-wright-qajbai branch from fadf0ae to 09d224d Compare June 20, 2026 18:04
@krlmlr
krlmlr changed the base branch from v2.0 to main June 20, 2026 18:20
krlmlr and others added 3 commits June 20, 2026 20:24
Measured on v2.0: a release build runs the identical 4859 registered /
4517 executed tests with identical outcomes (0 failures) as reldebug, builds
faster (1016s vs 1508s), and its tree is 0.7 GiB vs 16.1 GiB. So release is the
default gate and disk stops being the binding constraint (concurrency is
CPU-bound); reldebug / relassert remain triage knobs via DUCKDB_BUILD. Also
prune *.a alongside *.o.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
- linear-todo.sh: when the target tip is still at BASE (freshly pushed empty
  branch), treat it as a fresh start and keep all remaining commits, instead of
  trying to match BASE's own (release-tag) Upstream-commit in the dev range.
- SKILL.md: drop a reference to a skill that does not live in this repository.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
@krlmlr
krlmlr force-pushed the claude/awesome-wright-qajbai branch from 353994c to 6cc0e67 Compare June 20, 2026 20:41
claude added 8 commits June 20, 2026 21:09
- Authorship: keep the upstream author on every commit (incl. tweaked ones);
  committer is Claude <noreply@anthropic.com>. These show unverified on GitHub
  (committer != signer) — the accepted trade-off for keeping authorship, as with
  the -flat branches. Never --reset-author.
- Fidelity: the branch content must converge to the dev line (v2.0-flat tip);
  the release base supplies ancestry only. Resolve each commit's touched files
  to the dev line's content (target), covering conflicts and silent auto-merges;
  back-merges are re-sync points; assert `git diff <TARGET> v2.0-flat` empty.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
…nvergence

Encode lessons from replaying v2.0-flat onto v1.5.4:

- Apply strictly in source order and resolve each conflict in place; never
  defer a conflicting commit to a backlog (breaks order and resolution context).
- Resolve a whole commit to one side; partial --ours/--theirs on an atomic PR
  splits declaration from definition and breaks the build. Default mechanism is
  cherry-pick -n -X theirs + commit -C.
- Convergence is a tip/back-merge-boundary property, not per-commit: a region a
  later commit re-touches is transient and may resolve toward base (for build)
  when dev content needs an API the diverged base lacks.
- Release->main back-merges carry only content the v<TAG> base already has; skip.
- On red, prefer re-resolving a transient region toward base before amending;
  escalate only when no small fix greens it. Use the build window for N+1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
Cherry-pick conflict markers are red herrings; -X theirs is the correct
disposition. The real conflicts (external callers, type/ABI drift, behavioural
mismatch) surface only as build/test failures, and the faithful fix is to
forward-port the reconciling change from the later dev commit that supplies it
(possibly much later; ports can chain) -- not to revert the region to base,
which desyncs or guts the commit and often still fails to build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
…iles

Evidence from real conflicts: -X theirs corrupts generated files (settings.hpp/
config.cpp index mapping) while still compiling, and silently drops base-side
edits in cross-cutting refactors (JoinCondition .left -> GetLHS migration).
Classify conflicts by type: generated output -> take the source (settings.json)
and `make generate-files`, never merge the generated files; code -> build+test
gate then forward-port, inspecting for dropped base edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
…elease

A perpetual `main-<release>` branch (e.g. main-v1.5-variegata) whose tree is
always identical to main, with the release branch's back-merges progressively
de-merged from the bottom up. Each self-contained, restartable run advances the
bifurcation one back-merge ahead (onto its second parent), reproducing that
merge's tree as a deterministic checkpoint, keeping the chain green (release
build + fast tests, shared ccache), and force-pushing (--force-with-lease) only
after asserting the tip tree is unchanged.

Encodes the validated findings: the de-merge is strictly sequential (each
segment needs the accumulated main chain; independent per-2nd-parent batches
drift by the missing chain, growing with segment length); conflict resolution
runs in the main agent (-X theirs provisional, generated files regenerated,
real breaks forward-ported); commit optimistically then test, fix until the
whole chain is green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
A faithful de-merge carries each PR's diff verbatim, so comparing every replayed
commit's numstat to the original both proves the work and localises drift.
Validated: clean segments reproduce numstat identically; the one divergence
(`Replace magic number` 1f +2 -1 -> 1f +111 -136) pinpoints the generated
config.cpp index churn from -X theirs, flagged for regeneration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
git show --numstat is per-file; diffing the two sorted per-file lists names the
exact drifted file instead of an aggregate count. Validated: clean segments diff
empty; the magic-number commit's drift is localised to src/main/config.cpp
(+2 -1 -> +111 -136), the generated settings-index churn flagged for regen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
Cherry-picking each PR onto a fixed release base works from the first-parent
bifurcation and must bridge the full ~8500-file v1.5.4->v2.0 divergence one
conflicted commit at a time, with no re-sync (drift-prone). The bubble skill
supersedes it: tree == main, advance the bifurcation one back-merge per run
with merge-tree checkpoints and per-file numstat proof of work. Banner + frontmatter
flag it as reference-only; historical notes retained.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
@krlmlr krlmlr changed the title build-linear-branch skill: test-gated linear vendoring branches bubble skill: incrementally linearise main against a release line (supersedes build-linear-branch) Jun 21, 2026
claude added 2 commits June 21, 2026 10:02
The cursor's "second parent is an ancestor of <release>" filter also matched
predecessor back-merges, because each release contains the older one it forked
from (v1.5-variegata back-merges v1.4-andium). It swept in v1.4->main merges
(31 = 15 v1.4 + 16 v1.5). Add a predecessor-release argument and exclude any
back-merge whose second parent is an ancestor of it, keeping only the release's
own back-merges (16 v1.5->main; next = 7eebd33). SKILL.md documents the scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
Make explicit that a cold release build (~10-17 min) must not be paid per run:
the tree-==-main invariant means each rewritten commit's objects match a main
build, so a persisted/pre-warmed CCACHE_DIR builds them from near-total hits, and
a sequence built in one worktree relinks incrementally (seconds-to-minutes, not
the cold cost). Also scope green-certification to the de-merged segment only
(CURRENT_FORK..merge checkpoint); the history past the merge is unchanged
(trees still equal main), needs no rebuild, and is certified by the run that
linearises it. Updated invariant, run step 4, and hard rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy
@krlmlr
krlmlr changed the base branch from main to v2.0 June 21, 2026 10:41
@krlmlr
krlmlr merged commit 76ae96a into v2.0 Jun 21, 2026
5 of 6 checks passed
@krlmlr
krlmlr deleted the claude/awesome-wright-qajbai branch June 21, 2026 10:41
krlmlr pushed a commit that referenced this pull request Jun 21, 2026
On top of the merged bubble skill (PR #7) on v2.0. The merged cursor already
scopes back-merges to one release (prev-release as 3rd arg), so this only adds
what the skill still lacks:

- reattach.sh: tree-preserving graft rewrite for step 2 — replace the de-merged
  merge with the linear reconstruction, re-commit only true descendants reusing
  trees. The SKILL's rebase --rebase-merges re-merges the upper back-merges and
  conflicts on generated files (e.g. src/main/config.cpp); the graft does not.
- RUN-NOTES.md: the first v1.5 run (de-merged #16 into main-v1.5-variegata,
  checkpoint 358030a reproduced, 4 commits verbatim, tip tree == main tree),
  refs/pairing, next-run preview, and the unity-build ccache caveat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DtisRUvmSNJw5KnvrpbHrM
krlmlr added a commit that referenced this pull request Jul 28, 2026
…persedes build-linear-branch) (#7)

* Add build-linear-branch skill for test-gated linear vendoring branches

Tooling to build and resume `v<DEV>-v<TAG>-linear` branches (e.g.
`v2.0-v1.5.4-linear`): a linear, per-commit test-gated replay of a dev
`-flat` branch onto the flat commit for a release tag. Companion to the
`add-flat-branch` skill; reuses its `Upstream-commit` trailer as the restart
cursor.

- `.claude/skills/build-linear-branch/SKILL.md` — the agentic loop:
  cherry-pick + semantic conflict resolution in the main session, then
  build -> prune -> test per commit in disposable worktrees with shared
  ccache, push only when green (CI re-verifies), scrub worktrees. Restartable:
  the branch tip is the state, resumed via the trailer.
- `linear-todo.sh` — resume-aware cursor: derives BASE from a release tag,
  TARGET name, and remaining source commits oldest-first.
- `linear-build-test.sh` — phased green gate; the build phase prunes object
  files after linking so a second worktree can build while this one tests.
- `linear-disk-guard.sh` — admission control modelling one full build plus
  (k-1) pruned trees; caps concurrent worktrees to fit the disk.
- `.gitignore` — keep `.claude/skills/` under version control.

Helper scripts live beside the skill under `.claude/skills/build-linear-branch/`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* Bump

* build-linear-branch: default the test gate to release

Measured on v2.0: a release build runs the identical 4859 registered /
4517 executed tests with identical outcomes (0 failures) as reldebug, builds
faster (1016s vs 1508s), and its tree is 0.7 GiB vs 16.1 GiB. So release is the
default gate and disk stops being the binding constraint (concurrency is
CPU-bound); reldebug / relassert remain triage knobs via DUCKDB_BUILD. Also
prune *.a alongside *.o.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: fix empty-target resume; scrub cross-repo skill ref

- linear-todo.sh: when the target tip is still at BASE (freshly pushed empty
  branch), treat it as a fresh start and keep all remaining commits, instead of
  trying to match BASE's own (release-tag) Upstream-commit in the dev range.
- SKILL.md: drop a reference to a skill that does not live in this repository.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: preserve upstream authorship; converge to v2.0-flat

- Authorship: keep the upstream author on every commit (incl. tweaked ones);
  committer is Claude <noreply@anthropic.com>. These show unverified on GitHub
  (committer != signer) — the accepted trade-off for keeping authorship, as with
  the -flat branches. Never --reset-author.
- Fidelity: the branch content must converge to the dev line (v2.0-flat tip);
  the release base supplies ancestry only. Resolve each commit's touched files
  to the dev line's content (target), covering conflicts and silent auto-merges;
  back-merges are re-sync points; assert `git diff <TARGET> v2.0-flat` empty.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: ordered in-place resolution, transient-region convergence

Encode lessons from replaying v2.0-flat onto v1.5.4:

- Apply strictly in source order and resolve each conflict in place; never
  defer a conflicting commit to a backlog (breaks order and resolution context).
- Resolve a whole commit to one side; partial --ours/--theirs on an atomic PR
  splits declaration from definition and breaks the build. Default mechanism is
  cherry-pick -n -X theirs + commit -C.
- Convergence is a tip/back-merge-boundary property, not per-commit: a region a
  later commit re-touches is transient and may resolve toward base (for build)
  when dev content needs an API the diverged base lacks.
- Release->main back-merges carry only content the v<TAG> base already has; skip.
- On red, prefer re-resolving a transient region toward base before amending;
  escalate only when no small fix greens it. Use the build window for N+1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: fix remedy to forward-porting, not prefer-base

Cherry-pick conflict markers are red herrings; -X theirs is the correct
disposition. The real conflicts (external callers, type/ABI drift, behavioural
mismatch) surface only as build/test failures, and the faithful fix is to
forward-port the reconciling change from the later dev commit that supplies it
(possibly much later; ports can chain) -- not to revert the region to base,
which desyncs or guts the commit and often still fails to build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: -X theirs is provisional; regenerate generated files

Evidence from real conflicts: -X theirs corrupts generated files (settings.hpp/
config.cpp index mapping) while still compiling, and silently drops base-side
edits in cross-cutting refactors (JoinCondition .left -> GetLHS migration).
Classify conflicts by type: generated output -> take the source (settings.json)
and `make generate-files`, never merge the generated files; code -> build+test
gate then forward-port, inspecting for dropped base edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* Add bubble skill: incremental sequential linearisation of main vs a release

A perpetual `main-<release>` branch (e.g. main-v1.5-variegata) whose tree is
always identical to main, with the release branch's back-merges progressively
de-merged from the bottom up. Each self-contained, restartable run advances the
bifurcation one back-merge ahead (onto its second parent), reproducing that
merge's tree as a deterministic checkpoint, keeping the chain green (release
build + fast tests, shared ccache), and force-pushing (--force-with-lease) only
after asserting the tip tree is unchanged.

Encodes the validated findings: the de-merge is strictly sequential (each
segment needs the accumulated main chain; independent per-2nd-parent batches
drift by the missing chain, growing with segment length); conflict resolution
runs in the main agent (-X theirs provisional, generated files regenerated,
real breaks forward-ported); commit optimistically then test, fix until the
whole chain is green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: require per-commit numstat (before/after) as proof of work

A faithful de-merge carries each PR's diff verbatim, so comparing every replayed
commit's numstat to the original both proves the work and localises drift.
Validated: clean segments reproduce numstat identically; the one divergence
(`Replace magic number` 1f +2 -1 -> 1f +111 -136) pinpoints the generated
config.cpp index churn from -X theirs, flagged for regeneration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: per-file numstat (not shortstat) as the proof-of-work check

git show --numstat is per-file; diffing the two sorted per-file lists names the
exact drifted file instead of an aggregate count. Validated: clean segments diff
empty; the magic-number commit's drift is localised to src/main/config.cpp
(+2 -1 -> +111 -136), the generated settings-index churn flagged for regen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: mark outdated, superseded by bubble

Cherry-picking each PR onto a fixed release base works from the first-parent
bifurcation and must bridge the full ~8500-file v1.5.4->v2.0 divergence one
conflicted commit at a time, with no re-sync (drift-prone). The bubble skill
supersedes it: tree == main, advance the bifurcation one back-merge per run
with merge-tree checkpoints and per-file numstat proof of work. Banner + frontmatter
flag it as reference-only; historical notes retained.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: scope back-merges to the release, excluding the predecessor line

The cursor's "second parent is an ancestor of <release>" filter also matched
predecessor back-merges, because each release contains the older one it forked
from (v1.5-variegata back-merges v1.4-andium). It swept in v1.4->main merges
(31 = 15 v1.4 + 16 v1.5). Add a predecessor-release argument and exclude any
back-merge whose second parent is an ancestor of it, keeping only the release's
own back-merges (16 v1.5->main; next = 7eebd33). SKILL.md documents the scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: warm-ccache build budget; certify only the rewritten segment

Make explicit that a cold release build (~10-17 min) must not be paid per run:
the tree-==-main invariant means each rewritten commit's objects match a main
build, so a persisted/pre-warmed CCACHE_DIR builds them from near-total hits, and
a sequence built in one worktree relinks incrementally (seconds-to-minutes, not
the cold cost). Also scope green-certification to the de-merged segment only
(CURRENT_FORK..merge checkpoint); the history past the merge is unchanged
(trees still equal main), needs no rebuild, and is certified by the run that
linearises it. Updated invariant, run step 4, and hard rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

---------

Co-authored-by: Claude <noreply@anthropic.com>
krlmlr added a commit that referenced this pull request Jul 28, 2026
…persedes build-linear-branch) (#7)

* Add build-linear-branch skill for test-gated linear vendoring branches

Tooling to build and resume `v<DEV>-v<TAG>-linear` branches (e.g.
`v2.0-v1.5.4-linear`): a linear, per-commit test-gated replay of a dev
`-flat` branch onto the flat commit for a release tag. Companion to the
`add-flat-branch` skill; reuses its `Upstream-commit` trailer as the restart
cursor.

- `.claude/skills/build-linear-branch/SKILL.md` — the agentic loop:
  cherry-pick + semantic conflict resolution in the main session, then
  build -> prune -> test per commit in disposable worktrees with shared
  ccache, push only when green (CI re-verifies), scrub worktrees. Restartable:
  the branch tip is the state, resumed via the trailer.
- `linear-todo.sh` — resume-aware cursor: derives BASE from a release tag,
  TARGET name, and remaining source commits oldest-first.
- `linear-build-test.sh` — phased green gate; the build phase prunes object
  files after linking so a second worktree can build while this one tests.
- `linear-disk-guard.sh` — admission control modelling one full build plus
  (k-1) pruned trees; caps concurrent worktrees to fit the disk.
- `.gitignore` — keep `.claude/skills/` under version control.

Helper scripts live beside the skill under `.claude/skills/build-linear-branch/`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* Bump

* build-linear-branch: default the test gate to release

Measured on v2.0: a release build runs the identical 4859 registered /
4517 executed tests with identical outcomes (0 failures) as reldebug, builds
faster (1016s vs 1508s), and its tree is 0.7 GiB vs 16.1 GiB. So release is the
default gate and disk stops being the binding constraint (concurrency is
CPU-bound); reldebug / relassert remain triage knobs via DUCKDB_BUILD. Also
prune *.a alongside *.o.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: fix empty-target resume; scrub cross-repo skill ref

- linear-todo.sh: when the target tip is still at BASE (freshly pushed empty
  branch), treat it as a fresh start and keep all remaining commits, instead of
  trying to match BASE's own (release-tag) Upstream-commit in the dev range.
- SKILL.md: drop a reference to a skill that does not live in this repository.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: preserve upstream authorship; converge to v2.0-flat

- Authorship: keep the upstream author on every commit (incl. tweaked ones);
  committer is Claude <noreply@anthropic.com>. These show unverified on GitHub
  (committer != signer) — the accepted trade-off for keeping authorship, as with
  the -flat branches. Never --reset-author.
- Fidelity: the branch content must converge to the dev line (v2.0-flat tip);
  the release base supplies ancestry only. Resolve each commit's touched files
  to the dev line's content (target), covering conflicts and silent auto-merges;
  back-merges are re-sync points; assert `git diff <TARGET> v2.0-flat` empty.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: ordered in-place resolution, transient-region convergence

Encode lessons from replaying v2.0-flat onto v1.5.4:

- Apply strictly in source order and resolve each conflict in place; never
  defer a conflicting commit to a backlog (breaks order and resolution context).
- Resolve a whole commit to one side; partial --ours/--theirs on an atomic PR
  splits declaration from definition and breaks the build. Default mechanism is
  cherry-pick -n -X theirs + commit -C.
- Convergence is a tip/back-merge-boundary property, not per-commit: a region a
  later commit re-touches is transient and may resolve toward base (for build)
  when dev content needs an API the diverged base lacks.
- Release->main back-merges carry only content the v<TAG> base already has; skip.
- On red, prefer re-resolving a transient region toward base before amending;
  escalate only when no small fix greens it. Use the build window for N+1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: fix remedy to forward-porting, not prefer-base

Cherry-pick conflict markers are red herrings; -X theirs is the correct
disposition. The real conflicts (external callers, type/ABI drift, behavioural
mismatch) surface only as build/test failures, and the faithful fix is to
forward-port the reconciling change from the later dev commit that supplies it
(possibly much later; ports can chain) -- not to revert the region to base,
which desyncs or guts the commit and often still fails to build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: -X theirs is provisional; regenerate generated files

Evidence from real conflicts: -X theirs corrupts generated files (settings.hpp/
config.cpp index mapping) while still compiling, and silently drops base-side
edits in cross-cutting refactors (JoinCondition .left -> GetLHS migration).
Classify conflicts by type: generated output -> take the source (settings.json)
and `make generate-files`, never merge the generated files; code -> build+test
gate then forward-port, inspecting for dropped base edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* Add bubble skill: incremental sequential linearisation of main vs a release

A perpetual `main-<release>` branch (e.g. main-v1.5-variegata) whose tree is
always identical to main, with the release branch's back-merges progressively
de-merged from the bottom up. Each self-contained, restartable run advances the
bifurcation one back-merge ahead (onto its second parent), reproducing that
merge's tree as a deterministic checkpoint, keeping the chain green (release
build + fast tests, shared ccache), and force-pushing (--force-with-lease) only
after asserting the tip tree is unchanged.

Encodes the validated findings: the de-merge is strictly sequential (each
segment needs the accumulated main chain; independent per-2nd-parent batches
drift by the missing chain, growing with segment length); conflict resolution
runs in the main agent (-X theirs provisional, generated files regenerated,
real breaks forward-ported); commit optimistically then test, fix until the
whole chain is green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: require per-commit numstat (before/after) as proof of work

A faithful de-merge carries each PR's diff verbatim, so comparing every replayed
commit's numstat to the original both proves the work and localises drift.
Validated: clean segments reproduce numstat identically; the one divergence
(`Replace magic number` 1f +2 -1 -> 1f +111 -136) pinpoints the generated
config.cpp index churn from -X theirs, flagged for regeneration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: per-file numstat (not shortstat) as the proof-of-work check

git show --numstat is per-file; diffing the two sorted per-file lists names the
exact drifted file instead of an aggregate count. Validated: clean segments diff
empty; the magic-number commit's drift is localised to src/main/config.cpp
(+2 -1 -> +111 -136), the generated settings-index churn flagged for regen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* build-linear-branch: mark outdated, superseded by bubble

Cherry-picking each PR onto a fixed release base works from the first-parent
bifurcation and must bridge the full ~8500-file v1.5.4->v2.0 divergence one
conflicted commit at a time, with no re-sync (drift-prone). The bubble skill
supersedes it: tree == main, advance the bifurcation one back-merge per run
with merge-tree checkpoints and per-file numstat proof of work. Banner + frontmatter
flag it as reference-only; historical notes retained.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: scope back-merges to the release, excluding the predecessor line

The cursor's "second parent is an ancestor of <release>" filter also matched
predecessor back-merges, because each release contains the older one it forked
from (v1.5-variegata back-merges v1.4-andium). It swept in v1.4->main merges
(31 = 15 v1.4 + 16 v1.5). Add a predecessor-release argument and exclude any
back-merge whose second parent is an ancestor of it, keeping only the release's
own back-merges (16 v1.5->main; next = 7eebd33). SKILL.md documents the scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

* bubble: warm-ccache build budget; certify only the rewritten segment

Make explicit that a cold release build (~10-17 min) must not be paid per run:
the tree-==-main invariant means each rewritten commit's objects match a main
build, so a persisted/pre-warmed CCACHE_DIR builds them from near-total hits, and
a sequence built in one worktree relinks incrementally (seconds-to-minutes, not
the cold cost). Also scope green-certification to the de-merged segment only
(CURRENT_FORK..merge checkpoint); the history past the merge is unchanged
(trees still equal main), needs no rebuild, and is certified by the run that
linearises it. Updated invariant, run step 4, and hard rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ1FSGt1fBcpJnNqW3zDxy

---------

Co-authored-by: Claude <noreply@anthropic.com>
pull Bot pushed a commit that referenced this pull request Sep 10, 2026
<details>
<summary>WARNING: ThreadSanitizer: data race; Read of size 8</summary>

```c++
  Filters: test/sql/show_select/summarize_subquery.test

  [0/1] (0%): test/sql/show_select/summarize_subquery.test==================
  WARNING: ThreadSanitizer: data race (pid=80064)
    Read of size 8 at 0x00010708e758 by thread T10:
      #0 duckdb::ArenaAllocator::AlignNext() <null> (libduckdb.dylib:arm64+0x2a37af0)
      #1 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x3184418)
      #2 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
      #3 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
      #4 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano  nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
      #5 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant  ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822   (libduckdb.dylib:arm64+0x318ca64)
      #6 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r  ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
      #7 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a  rm64+0x2458dac)
      #8 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.  dylib:arm64+0x2457ea0)
      #9 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const   <null> (libduckdb.dylib:arm64+0x2459624)
      #10 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggreg  ate.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
      #11 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2  448dac)
...

    Previous write of size 8 at 0x00010708e758 by thread T7:
      #0 std::__1::vector<double, duckdb::arena_stl_allocator<double>>::reserve(unsigned long) vector.h:1100 (libduckdb.dylib:arm64+0x31844a0)
      #1 duckdb_tdigest::TDigest::updateCumulative() t_digest.hpp:538 (libduckdb.dylib:arm64+0x31819cc)
      #2 duckdb_tdigest::TDigest::process() t_digest.hpp:583 (libduckdb.dylib:arm64+0x31816d8)
      #3 void duckdb::(anonymous namespace)::ApproxQuantileScalarOperation::Finalize<long long, duckdb::(anonymous namespace)::ApproxQuantileState>(duckdb::(ano  nymous namespace)::ApproxQuantileState&, long long&, duckdb::AggregateFinalizeData&) approximate_quantile.cpp:162 (libduckdb.dylib:arm64+0x318d540)
      #4 void duckdb::AggregateFunction::StateFinalize<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::ApproxQuant  ileScalarOperation>(duckdb::Vector&, duckdb::AggregateFinalizeInputData&, duckdb::Vector&, unsigned long long, unsigned long long) aggregate_function.hpp:822   (libduckdb.dylib:arm64+0x318ca64)
      #5 duckdb::RowOperations::FinalizeStates(duckdb::RowOperationsState&, duckdb::TupleDataLayout&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long) r  ow_aggregate.cpp:182 (libduckdb.dylib:arm64+0x166f848)
      #6 duckdb::RadixHTLocalSourceState::Scan(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.dylib:a  rm64+0x2458dac)
      #7 duckdb::RadixHTLocalSourceState::ExecuteTask(duckdb::RadixHTGlobalSinkState&, duckdb::RadixHTGlobalSourceState&, duckdb::DataChunk&) <null> (libduckdb.  dylib:arm64+0x2457ea0)
      #8 duckdb::RadixPartitionedHashTable::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::GlobalSinkState&, duckdb::OperatorSourceInput&) const   <null> (libduckdb.dylib:arm64+0x2459624)
      #9 duckdb::PhysicalHashAggregate::GetDataInternal(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const physical_hash_aggrega  te.cpp:978 (libduckdb.dylib:arm64+0x21305cc)
      #10 duckdb::PhysicalOperator::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const <null> (libduckdb.dylib:arm64+0x2  448dac)
...

    Location is heap block of size 56 at 0x00010708e740 allocated by main thread:
      #0 operator new(unsigned long) <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x91650)
      #1 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) <null> (libduckdb.dylib:arm64+0x2a37830)
      #2 std::__1::vector<duckdb_tdigest::Centroid, duckdb::arena_stl_allocator<duckdb_tdigest::Centroid>>::reserve(unsigned long) vector.h:1100 (libduckdb.dyli  b:arm64+0x3180da0)
      #3 void duckdb::(anonymous namespace)::ApproxQuantileOperation::Operation<long long, duckdb::(anonymous namespace)::ApproxQuantileState, duckdb::(anonymou  s namespace)::ApproxQuantileScalarOperation>(duckdb::(anonymous namespace)::ApproxQuantileState&, long long const&, duckdb::AggregateUnaryInput&) approximate_  quantile.cpp:122 (libduckdb.dylib:arm64+0x318ccd4)
      #4 void duckdb::AggregateFunction::UnaryScatterUpdate<duckdb::(anonymous namespace)::ApproxQuantileState, long long, duckdb::(anonymous namespace)::Approx  QuantileScalarOperation>(duckdb::Vector*, duckdb::AggregateInputData&, unsigned long long, duckdb::Vector&, unsigned long long) aggregate_function.hpp:782 (li  bduckdb.dylib:arm64+0x318c454)
      #5 duckdb::RowOperations::UpdateStates(duckdb::RowOperationsState&, duckdb::AggregateObject&, duckdb::Vector&, duckdb::DataChunk&, unsigned long long, duc  kdb::optional_ptr<duckdb::ClusteredAggr const, true>) aggregate_function.hpp (libduckdb.dylib:arm64+0x166ed24)
      #6 duckdb::GroupedAggregateHashTable::UpdateAggregates(duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsigned long lon  g>> const&, unsigned long long, bool) <null> (libduckdb.dylib:arm64+0x240d234)
      #7 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::Vector&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1  ::allocator<unsigned long long>> const&) <null> (libduckdb.dylib:arm64+0x240f000)
      #8 duckdb::GroupedAggregateHashTable::AddChunk(duckdb::DataChunk&, duckdb::DataChunk&, duckdb::vector<unsigned long long, false, std::__1::allocator<unsig  ned long long>> const&) <null> (libduckdb.dylib:arm64+0x240cb30)
      #9 duckdb::RadixPartitionedHashTable::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&, duckdb::DataChunk&, duckdb::vector<u  nsigned long long, false, std::__1::allocator<unsigned long long>> const&) const <null> (libduckdb.dylib:arm64+0x2455190)
      #10 duckdb::PhysicalHashAggregate::Sink(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSinkInput&) const physical_hash_aggregate.cpp:466 (  libduckdb.dylib:arm64+0x212bafc)

  SUMMARY: ThreadSanitizer: data race (libduckdb.dylib:arm64+0x2a37af0) in duckdb::ArenaAllocator::AlignNext()+0x2c
  ==================
```
</details>

`SUMMARIZE` calculates three `approx_quantile` aggregates (`q25`, `q50`,
`q75`). Each aggregate owns a TDigest whose vectors use the hash table’s
`ArenaAllocator`.

Two hash-aggregate worker threads concurrently finalized separate
TDigest states:

- Both called `TDigest::process()` → `updateCumulative()` →
`vector::reserve()`.
- The states shared the same non-thread-safe `ArenaAllocator`.
- One thread read allocator position metadata in
`ArenaAllocator::AlignNext()` while another wrote it.
- The shared arena had originally been allocated during
`approx_quantile` aggregation on the main thread.

Result: a race in arena allocation during parallel `approx_quantile`
finalization, reported at `ArenaAllocator::AlignNext()`.

It was intermittent because `reserve()` only occurs when vector capacity
must grow, and the worker calls had to overlap closely enough.

The fix reserves the cumulative buffer and sufficient centroid merge
capacity when the TDigest is constructed and its allocator is still used
by only one thread. Parallel finalization then operates entirely within
already allocated buffers and no longer mutates the shared arena.
krlmlr pushed a commit that referenced this pull request Oct 17, 2026
krlmlr pushed a commit that referenced this pull request Nov 12, 2026
… rows (duckdb#28315)

Since duckdb#27782, the radix hash table finalize copies the row of a group
that is new to a partition instead of combining it into a fresh state.
For `histogram`, `mode` and `entropy`, the copied state keeps an
`OwningStringMap` that copies non-inlined strings into the arena of the
sink thread that created it. Several finalize threads then combine into
maps bound to that same arena. `ArenaAllocator` is not thread-safe, so
the maps end up with corrupted, duplicated or unsorted keys.

`ThreadSanitizer` report:
```
 WARNING: ThreadSanitizer: data race
   Write of size 8 by thread T95:
     #0 duckdb::ArenaAllocator::Allocate(unsigned long long) arena_allocator.hpp:79
     #2 duckdb::Allocator::AllocateData(unsigned long long) allocator.cpp:131
     #3 duckdb::OwningStringMap<...>::CopyString(duckdb::string_t) owning_string_map.hpp:119
     #7 duckdb::HistogramFunction<...>::Combine<...>(...) histogram.cpp:66
     #12 duckdb::GroupedAggregateHashTable::Combine(...) aggregate_hashtable.cpp:1256
     #13 duckdb::RadixHTLocalSourceState::Finalize(...) radix_partitioned_hashtable.cpp:1154
   Previous write of size 8 by thread T92: (same stack, another finalize thread)
   Location is heap block of size 56 allocated by thread T92:
     #2 duckdb::ArenaAllocator::AllocateNewBlock(unsigned long long) arena_allocator.cpp:86
     #10 duckdb::HistogramUpdateFunction<...>(...) histogram.cpp:112
     #15 duckdb::RadixPartitionedHashTable::Sink(...) radix_partitioned_hashtable.cpp:690
 SUMMARY: ThreadSanitizer: data race arena_allocator.hpp:79 in duckdb::ArenaAllocator::Allocate(unsigned long long)
```

The arena chunk was created by a sink thread (`HistogramUpdateFunction`)
and two finalize threads then allocate from it through
`OwningStringMap::CopyString`.

~~Combine now checks whether the target map copies its strings into the
caller's arena. If not, it rebuilds the map there before inserting and
deletes the old map. The old strings stay valid in their arena.~~

**EDIT** the fix moved one layer up, into
`GroupedAggregateHashTable::Combine`. States no longer change owner: a
new group still copies the source row (the group columns), but its
states are initialized fresh with the target table's allocator and the
source state is combined into it and destroyed, the same as for existing
groups. This covers every aggregate without per-aggregate code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0