8000 Wrap groups on the outside by kenhkan · Pull Request #48 · noflo/noflo · GitHub
[go: up one dir, main page]

Skip to content

Wrap groups on the outside - #48

Merged
bergie merged 1 commit into
noflo:masterfrom
kenhkan:group
Nov 2, 2012
Merged

bergie merged 1 commit into
noflo:masterfrom
kenhkan:group

Conversation

@kenhkan
@kenhkan kenhkan commented Oct 30, 2012
Copy link
Copy Markdown
Contributor

Wrap groups on the outside of the incoming connection

bergie added a commit that referenced this pull request Nov 2, 2012
Wrap groups on the outside
8000 @bergie
bergie merged commit 1d9f991 into noflo:master Nov 2, 2012
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 14, 2026
bergie added a commit that referenced this pull request Oct 19, 2026
…ession pin (work document #48)

Distills the noflo-jekyll document leg (work document #35 updates
#14-#22) to its engine shape and runs it through the actual wrapAssembly
machinery: a gate-model render component — standing bracket-framed
config on a scoped:false control port read non-consumingly, per-message
scoped data on two ports gated on both — wrapped as an assembly adapter
(Fork, field extractors, CollectFields), driven with bracket-framed
scoped message envelopes whose arrival order races the config and
stagger with async jitter inside open activations, then driven to EOT
with the closedown cascade asserted.

The negative result this suite pins: on the current engine (post-#1094
control gates, #1092 drain-based closing, #1099 drain pump), roughly
1300 randomized runs across staggered, burst, bounded-edge, multi-
registration-config, and deep-overlap variants delivered every message
and closed down — the intermittent drop class the testbed documented
did not reproduce. Three randomized variants run in CI; the full matrix
rides the WD48_* env knobs.
bergie added a commit that referenced this pull request Oct 19, 2026
…closedown (work document #48)

Pins the engine mechanism behind the noflo-jekyll intermittent drop
(#48 update #2): a gate-model component — standing config on a control
port, scoped posts converted through the component itself, unscoped
pages held head-of-line until every post converted — driven through the
exact interleaving the pipeline races into. All inputs reach EOT while
the last consuming activation is open; its completion runs the
closedown predicate synchronously in deactivate (buffered data at EOT
is 'stranded by definition' per work document #45 and does not block
closing), setting closedDown before the pump offer that completion
schedules can fire. The pump's guard declines on closedDown, the held
resolvable documents never drain, and the shutdown silently discards
them. No error, no residue.

The test asserts the correct contract — every held resolvable document
must emit before closing down — and is RED on the current engine with
the diagnostic in the failure message: emitted posts only,
closedownFired, held=[r1,r2,r3]. It is the acceptance gate for the
engine fix; per-post resume gates make the interleaving exact.
bergie added a commit that referenced this pull request Oct 19, 2026
…before closing (work document #48)

When every input reached EOT while the last consuming activation was
still open, the closedown predicate ran synchronously inside deactivate
— buffered data at EOT is 'stranded by definition' and does not block
closing (#45) — and set closedDown before the pump offer that same
deactivation had scheduled could fire. The pump's guard declined on
closedDown; held documents that had become resolvable before EOT were
silently discarded by the shutdown. The noflo-jekyll testbed r
8000
eproduced
the drop end-to-end at ~10% per build (work document #48 update #2).

The fix: the closedown evaluation defers while a pump offer is pending
— the offer is the one state change that could still drain the buffers
— and the offer re-evaluates the closedown after its pass. An offer
that consumed leaves open activations whose deactivations re-evaluate;
an offer that declined declares the residue stranded with #45's
diagnostics. The deferral is buffer-aware: empty buffers do not defer,
preserving the closedown-per-burst-cycle granularity.

The deterministic pin (spec/wd48-anomaly.js) is green; the randomized
stress suite and the whole engine suite pass. End-to-end acceptance:
the noflo-jekyll stress went from 2-3 drops per 5 builds to 15/15
deterministic builds (987 files, identical tree).
bergie added a commit that referenced this pull request Oct 19, 2026
The closedown-vs-pump delivery race: held resolvable documents drain before closing (work document #48)
bergie added a commit that referenced this pull request Oct 19, 2026
… on their arrivals (work document #48)

A forwarding inport's arrival fired the direct path only — one
activation for the arriving IP's own scope — while the ready-scope walk
that re-fires every other scope holding pending work was skipped. Work
whose gate the arrival completes (a corpus arriving on a
framing-terminated control port completing a held document's hasStream
gate) stranded behind its scope until the shutdown silently discarded
it: the noflo-jekyll testbed's intermittent whole-leg drop, localized
by per-adapter trace attribution to the liquid Template's includes
gate (the corpus arrived, the pages were buffered, the inner never
activated).

The fix runs the ready-scope walk for the remaining scopes after the
direct firing, deduplicating the arriving scope; the forwarding
machinery's bracket ownership is unchanged. The one-firing rule's
coverage is now complete: every arrival on every port re-fires every
scope holding pending work.

The deterministic pin (spec/wd48-anomaly.js) reproduces the race
exactly — the corpus arrives after the message buffers, the pending
scope never re-fires — and is green with the fix. End-to-end
acceptance: the noflo-jekyll stress went from ~10/100 variant builds
across three shapes to 45+ consecutive deterministic builds.
bergie added a commit that referenced this pull request Oct 19, 2026
The one-firing rule's arrival walk fired each ready scope with the
arriving IP as the activation context — for the arriving port's own
scope that IP is the head, but for any other port's pending scope the
context was a lie. The concrete failure (work document #48 update #13,
distilled from the noflo-jekyll verification's null-partition report:
a control port's buffered openBracket arrived while another port held
unscoped work; the bracket became that port's firing context, and
fireProcess's openBracket check engaged auto-ordering for the whole
component. The drain's sends then queued into the ordered output
machinery although no stream ever arrived on the firing port, and the
ordered flush re-sent the activation's accumulated result on every
deactivation — three logical sends emitted nine packets with a
prefix-growing pattern.

The walk now offers readyScope.head as the context — the entry the scan
already captured for the drain pump — in both the unscoped and the
scope-stamped forms. The scan derives the scope from the head IP rather
than the buffer's object key: buffer keys are strings, while the scope
primitive (integer scopes are contract) must be what the process and
its outputs see. The pump's offer path heals identically.

The deterministic pin drives the exact testbed shape: unscoped
documents held head-of-line on a layout chain whose completing template
arrives last in a framed control-stream corpus. It asserts the exact
emission sequence — the strand and the duplication are both regressions
it catches — and the closedown landing after the drain.

Work document #48 update #13.
EOF
)
bergie added a commit that referenced this pull request Oct 19, 2026
…d-error detail

Review findings on #1103 (work document #48 update #15), three nits
resolving together:

The component forwarded only inport stranded reports to its own
stranded surface; outport-side reports (unattached sends, dropped
errors) reached the component surface only through direct dispatches
on nodeInstance — which is wired only inside a Network. A standalone
component's unattached outport fired the port-level event and nothing
on the surface its consumers actually subscribe to. The outport
forwarding listener now matches the inport path, and the direct
node-level dispatches are gone — the component surface is the single
path, so no double-dispatch in networks either.

The dropped-error detail carried the raw Error object; the network's
emit path spreads the detail into the event payload where every other
stranded entry is counts and strings. The error is serialized at the
source (error: String(e)), so the payload survives the spread and the
tracefile unchanged.

The unattached-send port-level payload now carries the port name
itself, so the report is self-describing before any forwarding.
bergie added a commit that referenced this pull request Oct 19, 2026
A component with no error port declared that calls done(err) escalates
through a load-bearing chain: error() throws, fireProcess's catch
answers with its own sendDone(e) which throws again from inside the
catch, the synchronous listener-throw propagation rejects the
delivering edge's write, and the socket's error side-channel turns it
into the network's process-error — or the loud async throw when no
socket error listener exists (standalone). Verified empirically and
held by a network-level pin: the chain is exactly what the
no-silent-loss invariant needs for components without an error leg,
and a refactor of the catch could sever it silently.

Work document #48 update #16.
bergie added a commit that referenced this pull request Oct 19, 2026
The no-silent-loss surface: strands, clobbers, and dropped errors are loud (work document #48)
bergie added a commit that referenced this pull request Oct 19, 2026
…s errors (work document #35 update #23 review)

Two documented-invariant violations the #1106 review verified
by live probes, both in CollectFields' merge loop:

- The named fields and the spread merged together in connection order,
  so a same-key collision resolved by JS object key order — ['meta','*']
  let the spread win while ['*','meta'] let the named field win,
  contradicting the documented order-independence. The merge is now
  three passes — replace, named fields, spread — fixed by mapping kind,
  never by connection order.
- The spread's errors key interacted inconsistently with the error
  accumulation: silently discarded in the spread-only shape (the final
  overwrite), clobbering the whole accumulation in the replace+spread
  shape — a silent loss of diagnostics, the class the #48 no-silent-loss
  surface exists to surface. The spread now never owns errors: the
  errors key is excluded from the merge and the accumulation channels
  own the field, making both config shapes behave identically. The doc
  comment matches.

Plus the review's judgment call: an in-mapping value of '*' is rejected
at wrap time — the spread marker is out-mapping vocabulary, and the
in-side '*' would generate an extractor walking a literal '*' key and
failing at runtime.

Pinned: the connection-order independence (both config orders, same
result), the errors exclusion (both shapes, plus the error-port
append on the replace path under the terminal-event semantics), and the
wrap-time rejection. Suites: assembly-utils 47, assembly 49.
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 20, 2026
… loud (work document noflo#48)

Implements the invariant the three-shape verification re-opened: the
Network must not lose messages silently under any scheduling. Three
surfaces, all engine-level and wire-safe (the 0x0f STRANDED tracefile
encoding stays bracket-shaped; the data-shaped entries ride the JS event
bus pending the freeze discussion):

- The closedown residue census: when the closedown predicate fires, the
  component's undeliverable data residue (per port, per scope, via the
  new InPort.strandedData()) is emitted as a component 'stranded' event
  and logged with named counts — the moment the strand became final,
  not a debug dump at shutdown. Standing IIP buffers are excluded:
  an unconsumed initial is configuration re-delivered per run (noflo#46),
  not stranded work.
- The control-stream replacement clobber: when a control port's
  latest-stream rule discards a buffered stream holding packets, the
  InPort dispatches a stranded report naming the discarded counts —
  the bang-vs-corpus hazard from the noflo-jekyll verification becomes
  diagnosable at the replacement moment instead of as a downstream
  strand.
- The unattached-outport send: a send into an outport with no attached
  sockets vanishes by contract, but the vanish now dispatches a
  stranded report and logs — the noflo#35 update noflo#18 footgun becomes visible.
  The component-side answer for legs that must not drop is
  required: true (escalates in checkRequired).

The Network forwards component stranded events onto its event bus;
bracket-shaped entries record to Flowtrace under the existing 0x0f
encoding, data-shaped entries stay on the JS surface. Three new
deterministic pins in spec/wd48-anomaly.js: the closedown census, the
replacement clobber, and the unattached-fail-leg drop.
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 20, 2026
In ordered mode every activation pushes its context's result object into
outputQ (Component.activate), and a process function that completes its
activation per item — a sendDone-in-a-loop drain — re-activates the same
context on its continuation: the loop's next read, or simply the closing
done(), both call activate on the completed context. The re-activation
re-pushed the same result object whose port arrays still held every
already-sent IP, so each ordered flush re-sent the accumulated history:
three logical sends emitted six-to-nine packets in a prefix-growing
pattern (d1), (d1,d2), (d1,d2,d3) — the duplication the noflo-jekyll
verification probes caught as d1x4/d2x3/d3x2 during the noflo#48
investigation. noflo#1103 fixed that investigation's spurious entry path into
ordered mode (a cross-port context IP engaging auto-ordering for a
component no stream ever arrived on); this closes the genuine
ordered-mode defect on master.

processOutputQueue now empties a result's port arrays as it sends them —
the queue holds unsent work only, by construction. Keys are kept (the
bracket-forwarding machinery walks them); only the sent entries leave.
A re-activated context re-pushing its result is now harmless regardless
of how many times it re-activates: the re-push enqueues an empty result.

The deterministic pin (added red, green at merge) drives a genuinely
auto-ordered component — a real framed stream engaging autoOrdering on
its own inport — whose drain loops sendDone per item, and asserts the
exact emission sequence: each item once, in order. On master it failed
with ["a","a","b","b","c","c"].

Work document noflo#49.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0