8000 fix(sync): harden bidirectional sync, replicate merge commits, and reconcile anchored absorbed histories by joeycumines · Pull Request #9 · grailbio/grit · GitHub
[go: up one dir, main page]

Skip to content

fix(sync): harden bidirectional sync, replicate merge commits, and reconcile anchored absorbed histories - #9

Open
joeycumines wants to merge 43 commits into
grailbio:masterfrom
joeycumines:pr/01-bidirectional-sync-hardening
Open

joeycumines wants to merge 43 commits into
grailbio:masterfrom
joeycumines:pr/01-bidirectional-sync-hardening

Conversation

@joeycumines
@joeycumines joeycumines commented Aug 25, 2026 •
Copy link
Copy Markdown
Contributor

Harden bidirectional syncing, replicate merge commit content, and prevent unrelated histories from replaying when syncing across repository absorption merges.

Motivation

Several edge cases in Grit lead to silent commit drops, broken history replays, or sync crashes:

  1. Skipped side branch commits: git log <anchor>..<branch> --ancestry-path ignored commits merged into the source branch that were not direct descendants of the resume anchor.
  2. Dropped merge commits: The sync loop passed --no-merges, dropping conflict resolutions ("evil merges"), -s ours discards, and octopus merges.
  3. Replaying foreign history across absorption merges: When a repository is absorbed via git merge --allow-unrelated-histories, Grit's range walk traversed the entire foreign history and tried to apply those diffs as individual patches against unrelated destination states, causing corruption and patch failures.
  4. Patch application failures: Pre-existing destination content or cherry-picks failed under plain git am instead of converging against destination HEAD.
  5. State corruption & prefix traps: ResetToRemote used FETCH_HEAD (which gets overwritten by object fetches) instead of the remote tracking branch. Substring prefix matches also matched sibling paths like proj.txt for prefix proj/.

Changes

  • Topological commit traversal: Replaced --ancestry-path with --topo-order so commits apply in order after their parents. Formatted patches include full binary and index metadata.
  • Merge reconciliation: Merge commits are no longer dropped. Instead of replaying merge diffs directly, Grit diffs the merged tree against the destination tree at that point in history, applying only the net changes as a tagged corrective commit (git/merge.go).
  • Skip foreign ancestry on absorption merges: When syncing past an unrelated-histories merge, Grit checks merge parents with git merge-base and excludes commits from branches that don't share history with the resume anchor. Their net content lands cleanly through the merge's own reconciliation.
  • 3-way convergence: Uses git am --3way with source objects pre-seeded into refs/grit/src. Diffs whose target state already matches destination HEAD are pruned so identical commits don't duplicate.
  • Prefix containment & safety: Switched to directory boundary prefix checks (pathInPrefix) and case validation against verbatim git trees.

Known Limitations

  • Initial sync: On a brand-new repository sync without an anchor, unrelated histories replay as individual patches (since there is no baseline anchor to measure against). Conflicting overlapping files pause for manual resolution as usual.
  • Disjoint anchors: If a resume anchor shares no common history with the source branch (e.g. rewritten upstream history), the unrelated-history filter skips with a warning to avoid dropping valid mainline commits.

Testing

Includes regression and unit tests covering:

  • Merge commit replication (evil merges, -s ours discards, octopus joins).
  • Absorption merges with unrelated histories (forward, reversed, and nested parent orders).
  • Bidirectional fixed points and ping-pong sync stability.
  • Path boundary matching, case sensitivity, and paused session recovery.

grit defaults to the master branch and the tests assume it; respect the
user's environment by pinning the initial branch via GIT_CONFIG_* for
every git invocation made by the test helpers.
The resume selection used git log <last>..<branch> --ancestry-path,
which silently drops every source commit that does not descend from
the last synchronized commit. Work merged into the source branch from
side branches after the anchor was synced is exactly of that shape, so
grit reported success while the destination diverged arbitrarily far.

Select every commit in <last>..<branch> instead, ordered topologically
so that patches always apply after their parents' patches regardless
of authorship dates or merge topology, and pass --binary to
format-patch so binary deltas serialize applicably on any git version.
Regression tests reproduce both failure modes end-to-end.
Catch-up replays collide with content that reached the destination
through other routes (cherry-picks, direct merges): identical additions
fail with 'already exists', drifted files fail their context check.

- Prune diffs whose post-image blob already matches the destination;
  commits whose diffs all drop are skipped as already converged.
- Apply with git am --3way, seeding the source objects into the
  destination clone first so pre-image blobs are available: drifted
  files merge instead of failing; only genuine conflicts pause.
- Preserve paused am sessions and unpushed continued sessions across
  grit invocations (Open only resets when HEAD is an ancestor of the
  remote tip), and push previously resolved sessions even when the
  remaining selection is empty. Conflicts print actionable resolution
  instructions.
Selection alone cannot make processing exactly-once: applied sibling
commits of an interrupted run are not ancestors of the run's last
applied commit, so the next run's range re-selects them.

Shipit source ids are the only synchronization state that persists
through grit's push-only model, so selection now also drops every
commit whose digest is recorded as a shipit id anywhere in the
destination history. New tags record the full 40-hex digest (the
existing regex already accepts it, and a full digest doubles as an
exact revision when resolving the resume anchor); legacy abbreviated
ids keep matching by prefix, with the same collision window the anchor
walk has always accepted.
Review findings, addressed in full:

- Prune never matched: format-patch abbreviates index-line blob digests
  to core.abbreviation, so post-images were compared as 7-hex against
  full 40-hex worktree hashes. Pass --full-index to both format-patch
  calls; three-way merges now also reconstruct exact pre-images.
- Tag-set exclusion used a git BRE pattern verbatim under Go regexp,
  where \? is a literal question mark; ids never matched. Use RE2
  syntax, and match only ids standing alone on their own line so prose
  quoting an id cannot silently exclude real source commits.
- Fail loudly when a resume anchor no longer resolves in the source
  instead of risking a version-dependent silent empty selection.
- IsAncestor uses merge-base equality instead of string-matching exit
  statuses; OriginHead's snapshot semantics are documented.
- Corrected the mode-change rationale in the prune comment (pure mode
  changes carry no index line and are kept), and documented dump-mode
  pruning semantics.
- go.mod raised to go 1.17: tests use os.ReadFile, t.Setenv and
  t.TempDir from newer releases than the declared go 1.13.
- Tests: PartialDriftThreeWay now drifts inside the hunk context where
  plain am provably fails; DuplicateAddConvergence asserts the prune
  log line that distinguishes pruning from am --3way duplicate
  handling; new tests pin whole-commit skip accounting and loud failure
  for unborn sources (pre-existing upstream behavior, verified against
  d3b81e6).
- Open preserves unpushed clone state only when HEAD carries a shipit
  id (state grit itself created); foreign commits abort loudly instead
  of being published by the resolved-session push path, which now
  re-checks the same invariant.
- A paused session short-circuits selection and convergence pruning so
  no decisions are made against a conflicted worktree; dump mode stays
  read-only. Pause guidance notes it belongs to this source/destination
  configuration and gains an --abort escape hint.
- Recorded shipit ids shorter than seven characters are ignored for
  exclusion, mirroring strip-commit's minimum; residual ambiguity from
  Log's dedentation of quoted ids is documented rather than overclaimed.
- ncommit counts attempts (am --3way 'No changes' creates no commit);
  documented as benign via self-healing exclusions.
- BlobHash documents symlink and sha256-format behavior; IsAncestor
  documents transient-failure consequences.
- Tests: drift scenario proven to fail plain am; prune assertions
  distinguish pruning from am --3way duplicate handling; dump-mode
  pruning parity; paused-session refusal assertions; non-push pause
  then later publish lifecycle; converged-commit re-examination after
  divergence surfaces loudly; legacy anchor test renamed to match what
  it exercises; gritCloneDir documents its coupling to Open.
- Pre-existing hygiene: strip-commit digests normalized to lowercase;
  unreachable rule-length check after parseRewriteRule removed.
Third-review findings, addressed in full:

- Selection adds --full-history: git's default history simplification
  hid side-branch commits whose prefix effect duplicated a surviving
  merge parent's tree — the same silent-skip signature this fork
  exists to fix, one level deeper. Both selection sites affected.
- The multi-way copy filter now uses own-line shipit id matching (a
  prose-quoted id in a source message can no longer permanently drop
  that commit's changes) and logs each such skip.
- Dump mode previews the unpruned candidate set from the source side:
  convergence pruning is destination-state-dependent and cannot know
  the state a real run will have built up per patch; the comment now
  says exactly that.
- BlobHash hashes symlinks by link text via Lstat/Readlink, matching
  how git stores them: live links now prune correctly instead of
  never matching, and deletions of broken symlinks are no longer
  falsely pruned as converged.
- The clone cache is keyed by url+prefix+branch so configurations
  sharing a URL cannot carry each other's unpushed state.
- Removed the dead unborn-source guard (Open fails loudly first;
  pinned by TestV2UnbornSourceFailsLoudly).
- New tests: merge-hidden sibling visibility; foreign unpushed abort.
Fourth-review findings, addressed in full:

- Message-level shipit state (anchor walk and exclusion set) is read
  with a new LogIgnoringPrefix: a worktree-dependent pathspec silently
  emptied both when the prefix subtree was absent, degrading to full
  replays that the exactly-once exclusion exists to prevent. Log's
  missing-directory fallback now says so out loud.
- The multi-way copy filter skips are logged; prose-quoted ids cannot
  drop source commits (own-line matching), pinned by unit and e2e
  tests in both directions.
- BlobHash symlink hashing is covered end to end: identical re-added
  links prune as converged, and deletions of broken symlinks apply
  instead of being falsely pruned.
- Dump-mode comments state the truth: dump refreshes clones like any
  run, does not apply or push, previews the unpruned candidate set,
  and is subject to the foreign-state preservation gate.
- LFS pointer matching happens in source-relative space so prefixed
  destinations are not silently skipped (pre-existing defect).
- Clone cache keys use 128 digest bits over url+prefix+branch.
Fifth-review findings, addressed in full:

- Convergence pruning required identical content whi
8000
le ignoring the
  diff's declared mode, silently dropping exec-bit flips forever (the
  pruned commits write no tag, so nothing ever re-converged them).
  Pruning now also compares the declared post-image mode against the
  destination file's mode; a pending mode change surfaces as a normal
  resolvable conflict instead of permanent silent divergence. Pinned
  end to end by TestV2ModeFlipIsNotConverged.
- A freshly created clone starts wherever the remote's default branch
  points, which may legitimately be unrelated to the configured
  branch; preservation gating (and its foreign-state abort) now applies
  only to clones that survived from a previous run. Non-default
  spec branches work again on first open.
- The anchor walk's parent-step-back fails with a clear remediation
  message when it reaches a tagged root commit, instead of git's
  ambiguous revision error.
- Rewrite rules never touch diffs carrying git binary sections; their
  limitation regarding NUL-free binary payloads serialized as text is
  documented at the rule site.
Sixth-review findings, addressed in full:

- The preservation and foreign-state gates applied to the source clone
  as well, bricking every run after -linearize left rewritten unpushed
  history in grit's read-through cache. Open now only gates
  destination clones (OpenDestination); source clones always discard
  local state in favor of the remote tip, restoring consecutive
  linearized runs.
- Clone freshness is determined under the flock, so a second process
  that waited for the first no longer attempts to clone into an
  existing directory.
- The rewrite-rule binary guard inspects both Meta (where the parser
  places binary payloads including their marker) and Body, never
  rewrites empty bodies into content, and documents that anchored
  rules are still the operator's responsibility for NUL-free textual
  binaries.
- HeadIsGritAuthored reads message-level state prefix-free; anchor id
  extraction prefers own-line ids with a logged body-wide fallback.
- Deduplicated a test comment sentence.
Seventh-review findings, addressed in full:

- Pruning compared pre-rewrite post-image digests while application
  materializes rewritten content, so rewrite-rule paths could freeze
  un-rewritten content into the destination or never recognize
  converged rewrites. Paths targeted by any rewrite rule are now
  exempt from pruning: divergence surfaces loudly (raw form versus
  rewritten intent is a genuine conflict) and resolution publishes the
  rewritten state. Pinned by TestV2RewrittenPathNeverPruned.
- Clone caches abandoned by the key-derivation change are announced at
  startup so paused sessions left by older versions are not silently
  lost.
Ninth-review findings, addressed in full:

- The workflow pinned Go 1.13 while go.mod requires 1.17 and the new
  tests use post-1.13 APIs, leaving the project's own CI permanently
  red at compile time. CI now matches the module's declared minimum.
- Source clones are unconditionally reset (including past any stray
  paused session), matching their documented read-through-cache
  contract; only destination clones consult paused-session and
  preservation state.
- Windows-hostile new tests skip themselves instead of failing on
  platforms without symlink privileges or executable-bit tracking.
Eleventh-review findings, addressed in full:

- Convergence pruning resolved diff paths through the OS, so on
  case-insensitive filesystems a case-only rename (delete+add under
  --no-renames) had its add half pruned against the differently-cased
  existing file: the destination silently lost the file, and the
  tagged delete-only commit permanently excluded the source commit.
  BlobHash now treats a path as present only when a directory entry
  matches its base name exactly. Pinned by
  TestV2CaseOnlyRenameIsNotPruned.
- compareDirs skips on windows, where no POSIX diff exists.
Twelfth-review findings, addressed in full:

- Convergence pruning verified case-exactness for the leaf entry only:
  on case-insensitive hosts a case-only SUBDIRECTORY rename resolved
  through the still-existing old-case directory and its add half was
  pruned, silently dropping the file while the tagged delete-only
  commit excluded the source commit forever. BlobHash now requires
  every path component to match a directory entry exactly, and
  TestV2CaseOnlySubdirRenameIsNotPruned pins the reviewer's fixture.
- Open releases its flock on every post-acquisition failure (the lock
  lifetime now belongs to a wrapper; success transfers it to Close).
- The unborn-source test asserts a loud failure naming the source
  instead of git's version-specific stderr wording.
- CI runs every package's tests, not just the root's.
- A diff path matching the configured prefix only by letter case now
  aborts loudly instead of being silently dropped (pre-existing silent
  loss on case-insensitive hosts when the prefix directory itself is
  case-renamed).
Thirteenth-review findings, addressed in full:

- The preservation gate inspected HEAD alone; a foreign commit buried
  under a grit-authored HEAD would be preserved and later published.
  Authorship now requires every commit in the unpushed range to carry
  an own-line shipit id (UnpushedCommitsAreGritAuthored).
- A configured source or destination prefix that matches no exact path
  at HEAD but differs from one only by letter case aborts loudly:
  tree-level pathspec matching is case-exact, so such configurations
  silently mirrored nothing. Legitimately absent prefixes (initial
  synchronization) and unborn repositories remain permitted.
- Convergence-pruning comment no longer contradicts the implemented
  content+mode criterion; BlobHash documents that filemode-skew only
  ever over-keeps diffs.
- New coverage: lock release after failed open, cache-key derivation,
  rewrite guards for binary sections and empty bodies, source-clone
  stray-state discard, case-mismatched prefix abort.
Fourteenth-review findings, addressed in full:

- The windows-hostility guard helper used (*exec.Cmd).Environ, added in
  Go 1.19, while the module and CI declare Go 1.17; os.Environ is
  portable across all declared toolchains.
- CheckPrefixCasing validated only the final prefix component: a
  wrong-cased intermediate component returned empty ls-tree output
  under byte-exact tree matching and escaped the guard, reproducing
  the silent zero-selection stall it exists to abort. Every component
  is now checked against its parent tree level.
- CI pinned decommissioned Node-based action releases that GitHub
  runners auto-fail before any step executes; checkout and setup-go
  moved to v4.
- open() derives its clone path through CachePath instead of a
  hand-maintained duplicate of the derivation.
- An interrupted first clone left a non-repository directory that
  wedged the cache entry forever; such leftovers are removed and
  recloned.
- Dead code removed: Patch.Paths lost its last caller to source-
  relative LFS matching; parseHeader was unused even at baseline.
Matches the monorepo's toolchain directive. CI derives its Go version
from go.mod (go-version-file), so the workflow can never drift from the
module's declared minimum again.
Fifteenth-review notes, addressed in full:

- The anchor walk now enforces the same seven-character minimum for
  shipit ids as the exclusion set and strip-commit rules; an anchor
  carrying only shorter ids aborts loudly instead of resolving an
  untrustworthy revision.
- The source-relative LFS path mapping moved into a named helper with
  direct unit coverage covering prefixed and unprefixed destinations.
Sixteenth-review findings, addressed in full:

- Convergence pruning hashed the destination worktree, where untracked
  leftovers (which grit's own pause guidance invites) silenced incoming
  additions: an identical untracked file pruned the diff, the push
  omitted the file permanently, and no tag or error ever recorded it.
  Pruning now consults HEAD's committed tree via ls-tree — inherently
  case-sensitive, symlink- and mode-correct, and immune to worktree
  garbage; collisions with untracked leftovers fail loudly at
  application instead. Replaces the worktree hashing and its
  case-exactness directory walk wholesale.
- The paused-session short-circuit now precedes object seeding and
  prefix-casing validation, and its comment states the actual order.
- Removed a duplicated paragraph left in the pruning comment block by
  earlier edits; strip-commit length errors echo the normalized input.
- New test pins that untracked leftovers fail loudly and that a
  clean-and-rerun converges.
Seventeenth-review findings, addressed in full:

- A plain --single-branch clone checks out the remote's default branch,
  leaving refs/heads/<branch> absent for non-default source branches:
  object seeding then failed loudly on every run, and dump previews
  silently degraded to empty through swallowed pathspec
8000
 errors. Clones
  now request the configured branch itself, and a source branch that
  does not resolve aborts before selection in every mode.
- gritCloneDir is pinned against git.CachePath so derivation drift
  fails loudly instead of mislocating paused sessions in tests.
- CI adds a macOS leg, exercising case-insensitive paths, symlinks and
  executable-bit handling that ubuntu cannot represent.
Eighteenth-review findings, addressed in full:

- HeadEntry disables pathspec globbing via :(literal) and errors on
  ambiguous matches instead of silently taking the first, so
  metacharacters in file names can never alias prune decisions.
- The anchor walk's BRE grep now has a loud environment self-check:
  when the destination provably records shipit ids but the walk found
  no anchor, a git whose regex support lacks the pattern's extensions
  aborts instead of degrading every run into full-history replays.
- Strip-rule applicability was verified consistent across both call
  sites (destination-relative space on both sides); no change needed.
- A test passed -push through the source-spec argument slot; corrected.

The pass's headline major (non-default source branches failing at
object seeding) was refuted empirically against current HEAD: the
configured branch is cloned and mirrored end to end; the reviewer
tested stale state.
…specs

Nineteenth-review findings, addressed in full:

- Diff headers truncated at the first space, wedging synchronization on
  any path containing one (git emits symmetric unquoted headers).
  Parsing now resolves the symmetric form by locating the separator
  whose halves agree, handles git's C-quoted form for special
  characters, and Write re-emits both forms; covered end to end for
  application and convergence pruning.
- Log's absent-prefix fallback detected git's English error text,
  breaking under localized gits; the filesystem decides instead.
- CheckPrefixCasing uses :(literal) pathspecs so glob metacharacters in
  prefix components cannot alias tree entries.
- strip-commit rejects digests longer than forty characters instead of
  accepting silently inert rules.
…ness

Twentieth-review findings, addressed in full:

- parseDiffHeader tried the symmetric form before the C-quoted one,
  whose leading quote could never satisfy the a/ prefix test — the
  quoted branch was unreachable and non-ASCII paths wedged every run
  with 'diff is missing header'. The quoted form is now parsed first,
  decoded with git's C-style grammar, and Patch rewrites C-quoted
  meta lines (--- "a/…" / +++ "b/…") into destination space with
  exact re-quoting on emission. Covered end to end for application
  and convergence pruning of non-ASCII paths.
- The anchor walk is driven by a single prefix-free destination scan
  shared with the exclusion set, using Go's own regex engine: a git
  whose grep lacks the BRE extensions can no longer silently degrade
  runs into full-history replays. A destination whose every tagged
  commit is anchor-ineligible under strip rules is a clean tag-set
  fixed point, not an error (pinned by TestV2AllTagsIneligibleIsFixed
  Point); legacy sub-seven-character anchors resolve exactly as
  upstream did.
- UnpushedCommitsAreGritAuthored requires flush-left fb-prefixed ids —
  grit's exact serialization — so indented prose quotations cannot
  pass the foreign-state gate.
…ed-point test

Twenty-first-review findings, addressed in full:

- The regex-degradation self-check introduced for the pre-refactor
  grep-based walk fired on the legitimate all-tags-ineligible state
  (every recorded tag strip-ineligible), aborting with a factually
  false diagnostic a configuration that converges cleanly through
  initial sync plus tag-set exclusion. Anchor discovery no longer uses
  git's grep at all, so the check is deleted outright.
- Log tolerates exactly upstream's tolerated failure class again; the
  filesystem-based widening swallowed unrelated failures whenever the
  prefix directory was absent. Locale caveat documented at the site.
- TestV2AllTagsIneligibleIsFixedPoint now genuinely exercises the
  all-tags-ineligible fall-through: tightened strip rules make the
  recorded tag ineligible, the walk steps past it, and tag-set
  exclusion drops the mirrored commit to a nothing-to-do fixed point.
Twenty-second-review findings, addressed in full:

- A commit whose diffs partially converged was tagged like any other,
  permanently excluding it even if the destination later lost the
  applied content through a revert. Partially-pruned commits now land
  without a shipit tag: they are re-examined on every run, kept diffs
  no-op through three-way application, pure reverts self-heal, and
  genuine divergence surfaces as a resolvable conflict. Push gating
  counts HEAD movement rather than attempts, so untagged
  re-examinations never spam empty pushes.
- Dead code removed: the commented-out isApplicableCommit block and the
  unused continueHeader identifier.
- New tests: the full partial-prune lifecycle (untagged landing, quiet
  fixed point, revert self-heal) and rejection of indented shipit-id
  quotations by the authorship gate.

The pass's strip-rule path-space note was investigated and refuted:
anchor eligibility evaluates destination commits in destination-native
path space, which is identical to the apply loop's space for every
configuration (the synthetic destination prefix exists only when
mirroring from a differently-shaped repository, where upstream's own
semantics are undefined).
Twenty-third-review findings, addressed in full:

- The stripped-subject override ran after the untagging step and
  unconditionally re-attached the shipit tag, freezing partially-pruned
  commits against re-examination exactly when strip-message rules were
  active. Tag attachment now happens once, after convergence pruning:
  pruned>0 lands the commit untagged (subject and body preserved), and
  the override applies only to fully-tagged commits. Pinned by
  TestV2StripMessagePartialStaysUntagged.
- CheckPrefixCasing's per-component walk and fold-mismatch error are
  covered by TestV2CaseMismatchedPrefixComponentAborts.
Twenty-fourth-review finding, addressed: CheckPrefixCasing compared
prefix components against ls-tree output that core.quotePath escapes
for non-ASCII bytes, so correctly spelled non-ASCII components were
treated as absent and their case-typo'd spellings escaped the
fold-mismatch abort. Listings now use -z (NUL-separated, unquoted) and
compare byte-for-byte; covered by TestV2NonASCIIPrefixComponentCasing.
…file

Twenty-sixth-review findings, addressed in full:

- Convergence pruning deliberately authors untagged commits, but both
  authorship gates required a shipit id on every unpushed commit:
  deferred -push runs aborted on grit's own output, and the documented
  resolve/continue/re-run lifecycle wedged permanently after resolving
  a partial-commit conflict. Untagged commits now carry a machine-
  checkable 'grit-convergence-pruned: pruned/total' marker, and
  UnpushedCommitsAreGritAuthored accepts either serialization per
  commit; foreign commits (including indented prose quotations) still
  abort. Comments updated to the actual invariant.
- main_v2_test.go exceeded the 900-line file threshold under a
  version-numbered name; split into cohesive, descriptively named
  files (harness, sessions, mirroring) within the size guidance.
Completes the file-organization fix: main_mirroring_test.go drops back
under the 900-line threshold with the space/quote/case-rename/prefix-
casing path tests in their own cohesive file.
Twenty-sixth-review major, resolved: convergence pruning deliberately
authors untagged commits, but both preservation gates demanded a
shipit id on every unpushed commit, wedging deferred-push and
resolve/resume lifecycles on partial commits, and misclassifying
grit's own output as foreign after out-of-band destination rewrites.

- Untagged commits now carry a machine-checkable
  'grit-convergence-pruned: pruned/total' marker; the preservation
  scan accepts either that marker (flush-left) or grit's exact
  fb-prefixed tag serialization, and never the dedented-indented form.
- A preserved tail that does not connect to the remote tip is handled
  by composition: an entirely grit-authored tail is stale-but-
  rederivable cache state and is discarded with a loud reclone
  (ErrStaleCloneCache sentinel handled under the still-held lock);
  anything containing non-authored commits aborts loudly as foreign.
- Close tolerates the nil lock held across cache recovery.
Hardening applied while mirroring twenty-three live repositories:

- Convergence pruning now consults HEAD's committed tree via
  'ls-tree :(literal)' instead of the working tree: decisions are
  case-sensitive by construction, immune to untracked leftovers and
  conflict markers, and symlink/mode semantics come straight from the
  tree. An incoming addition colliding with an untracked leftover now
  fails loudly at application instead of being silently pruned.
- A rejected push (destination advanced independently, including
  out-of-band rewrites) discards the diverged clone state and
  instructs a re-run; the next run recomputes against the actual tip.
- Anchor commits that record a source revision without any content
  change (pure resume markers) are accepted as synchronization points.
- strip-commit length errors echo the normalized input.
Directory-boundary containment replaces bare HasPrefix in Patch
routing: an unslashed prefix like "proj" no longer captures sibling
paths such as "project.txt", which TrimPrefix mangled into phantom
destination files. Case-only mismatches now abort loudly instead of
being silently dropped by tree-level pathspec matching.

Patch metadata is rewritten with CRLF fidelity so patches round
tripped through Windows tooling survive intact, and deleted files no
longer report a phantom new mode. Own-line authorship matching
tolerates trailing carriage returns and convergence markers return
clean old/new id pairs.

State preservation now requires ancestry: a grit-authored unpushed
tail that has diverged from the remote tip can never be pushed, so
it is discarded up front and the clone rebuilt rather than deferring
the loss to a doomed push cycle. ResetToRemote and Open reset to the
originHead snapshot captured at open time instead of reading
FETCH_HEAD, which a later object-seeding fetch clobbers; this also
removes the --no-write-fetch-head dependency on newer git versions.

Empty-commit detection switches from format-patch output to
NUL-delimited diff-tree name-only output, so whitespace-only
pathnames are not misclassified and commit messages containing
"diff --git" no longer cause false positives. Prefix checking reads
tree entries directly with path.Base, and processedSourceIDs greps
for the shipit source id instead of walking the full log.
Replicating merge-commit content requires describing a merge the way
grit applies changes: as tree state relative to what the destination
already has. Add that description layer to the git wrapper.

Repo.MergeChangeset computes a merge's in-scope effect as tree entries
plus removals: candidates are paths changed between the merge and any
parent (union across parents), each classified against the merge's own
tree, which is the ground truth of the desired state. A candidate
present in the merge pins its mode and blob; one absent records a
removal. This classifies evil resolutions, "-s ours" discards, and
resurrected files correctly without special-casing any of them. Paths
matching the prefix only by letter case abort loudly instead of being
silently dropped by tree matching.

Repo.DesiredTree builds a tree from HEAD's tree plus such overrides
through a temporary index under GIT_INDEX_FILE, leaving the real index
and working tree untouched. Repo.TreeDiffs diffs HEAD's tree against
an arbitrary tree through the same parsing machinery as format-patch,
so the resulting diffs carry full index lines suitable for three-way
application. Git invocations gain an environment-aware variant so
index-building commands can point GIT_INDEX_FILE at the temporary
index; existing call paths are unchanged.

The main-loop wiring that consumes this plumbing lands in the next
commit.

Tests cover per-parent union semantics including the ours-discard and
resurrection cases, prefix filtering with letter-case aborts exercised
via update-index --cacheinfo so clash paths survive case-insensitive
filesystems, temp-index isolation with no leftover artifacts, the
TreeDiffs round trip through the patch parser, and author-identity
parsing against a pinned GIT_AUTHOR_DATE.
Selection no longer drops merge commits. Each merge is reconciled at
its topological position, after all of its ancestors' changes have
been applied: its net effect, restricted to paths it touched within
the source prefix and relocated under the destination prefix, is
computed against the destination's current tree and serialized as an
ordinary corrective patch tagged with the merge's own digest.

This replicates what format-patch cannot express: hand-resolved
(evil) conflict content, "-s ours" discards, and octopus unions land
byte-exactly in the mirror. Trivial merges converge with destination
state, replicate nothing, and take no tag, so re-runs are no-ops.
Tagging with the source digest makes replication exactly-once and
safe under bidirectional ping-pong. Genuine conflicts still pause the
session loudly through the existing am --3way flow.

Application, outcome accounting, and LFS object copying are shared by
both commit kinds via applyPatch; regular-commit behavior is
unchanged. Dump mode seeds source objects so merge previews serialize
against the destination clone. The tag-set-exclusion oracle narrows
from the generic "skipping converged" substring to the per-path prune
signature, because merges re-selected in a range legitimately report
their own state convergence.

Package documentation gains a "Merge commits" section describing
these semantics (including why -s subtree relocation across the
configured prefix is out of scope), pointed at from the README.

End-to-end tests cover evil merges whose resolved bytes must arrive
exactly, ours-discards that must erase side content without
resurrecting first-parent state, octopus unions, strip-rule
interaction with reconciliation (stripped paths leak neither on the
initial run nor when the rule is later removed), rerun idempotence,
and fixed-point stability across repeated bidirectional runs.
grit's anchored range selection (anchor..branch --topo-order
--full-history) enumerated the entire ancestry of merges whose second
parents root outside the source branch's history, such as whole
repositories absorbed with "merge --allow-unrelated-histories". Those
commits' patches assume parent states the destination never held, so
replaying them corrupted drifted destination files and paused sessions
mid-garbage; their net content is already delivered by each absorption
merge's own reconciliation.

Selection now drops every candidate reachable from a merge's non-first
parent that shares no common ancestor with the resume anchor, via
git.Repo.HasCommonAncestor (merge-base exit status 1 = disjoint) and
RevListExcluding. Initial synchronization is unchanged, and internal
side branches sharing history with the anchor keep being copied
individually. gitIOEnv now wraps exec errors with %w so exit codes
survive errors.As unwrapping.

Red-first coverage: TestGritUnrelatedMergedHistoryIsReconciledNotReplayed
builds an orphan lineage absorbed through an unrelated-histories merge
and pins that its commits land only through reconciliation -- no replayed
subjects, an exact destination commit count, tree equality, and a fixed
point. TestHasCommonAncestorAndRevListExcluding covers disjoint, shared,
and self cases plus rev-list counts.

Verified live against joeycumines/go-utilpkg -> joeycumines/go-eventloop:
all 29 absorption merges excluded their foreign ancestry (selection fell
101 -> 36), the pushed range carried 39 correctly-authored commits, the
source eventloop/ subtree equaled the destination root tree, and a rerun
reported nothing to do.
@joeycumines joeycumines changed the title Fix bidirectional sync and preserve state Fix bidirectional sync, replicate merge commits, and reconcile absorbed histories Aug 25, 2026
git assigns a merge's parent order from the committer's checkout state,
so an unrelated lineage can be the FIRST parent: absorbing via a merge
authored on the foreign branch and fast-forwarding the mainline to the
result left the entire foreign history in the selection, where its
patches were replayed against destination states they never assumed.
A foreign history that internally merges an anchored side branch hid
behind the same blind spot, because the shared second parent masked the
disjoint first one.

Inspect every parent of every merge instead; a related first parent
costs one merge-base call. Reconciliation is parent-symmetric (candidate
paths union over all parents, entries pinned from the merge's own tree),
so excluding a disjoint first parent loses no content.

Also stand the filter down loudly when the resume anchor shares no
history with the source branch: classifying merges against such a
reference -- a rewritten source history whose pre-rewrite digests still
resolve, for instance -- would discard legitimate side branches.

Pin the new regressions: reversed absorption, a disjoint first parent
behind a shared second parent, initial synchronization of an
absorbed-from-birth source, and a disjoint resume anchor; pin the unit
fixture's branch name against init.defaultBranch defaults.
@joeycumines joeycumines changed the title Fix bidirectional sync, replicate merge commits, and reconcile absorbed histories fix(sync): harden bidirectional sync, replicate merge commits, and reconcile anchored absorbed histories Aug 26, 2026
@joeycumines
Copy link
Copy Markdown
Contributor Author

I'll just leave this here. FYI I've made a bunch of changes in my fork. If I got any response I was going to stage up some additional PRs (following this one) so as not to overwhelm. I'm not holding out for this, I've made so many changes that I'm content just using my fork (though I'd also be content contributing back upstream).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0