Repository navigation
Stream termination - #1092
Merged
Merged
Stream termination#1092
Conversation
Sending no longer ends the transmission: InternalSocket.post auto-connects but never auto-disconnects, and the connection stays open after the send. disconnect() is the sender's explicit decision that nothing more will come from it — the 0.x intent, per work document #40 update #5 — which per-burst framing could never express. The same-tick auto-disconnect was a 1.x timing-compat shim; with every component on the Process API and every graph redesigned for 2.x, the framing it provided is redundant with in-band brackets. Exploratory (stream-termination): the suite is knowingly red on tests whose finalization rode the outport's last auto-disconnect — ordered bracket forwarding, subgraph bracket forwarding, scope isolation, NetworkLifecycle merging. The replacement is EOF propagation (input disconnect = final activation = output finish), landing in the next commits.
An IIP is one packet from a sender that is then done: the runtime performs the sender's disconnect once the packet has been delivered through the edge (drain-then-close, so the packet never overtakes its own EOF). This is the seed of cascading closedown — receiving ports can mark the input at EOF once the transport event reaches them. Exploratory (stream-termination): inert until ports track EOF and components derive closedown from it.
InternalSocket gains terminal-close semantics: disconnect() marks the socket terminally closed (idempotent, emitted even when never connected, so a silent sender still EOFs its downstream), and connect() re-opens a terminal socket — connect is the beginning of transmission. InPort tracks end-of-stream per port, unscoped by definition: EOF means "there will be nothing more coming from this sender", so a component closing down puts every attached edge at EOF in every scope at once. The eof lifecycle event fires on both transitions — entering EOF and a new transmission re-opening the input. Component derives the classical cascading closedown from it: when every input is at EOF, no activation is open, and no buffered deliverable data remains (control ports excluded — standing values are legitimate at quiescence), the component terminally disconnects its outports, and the closedown event fires only after those disconnects have drained. An input re-opening revives a closed-down component. Self-driven components declare generator: true — input EOF must not close down a component that activates from its own clock. Subgraph wrappers close down with their inner networks: the shared-port EOF reflects the deepest sender's state, not the wrapper's completion, so the wrapper's closedown is driven by the inner network's end. Exploratory (stream-termination, work documents #40 update #5, #45).
checkIfFinished becomes a pure predicate: the network ends when every process has closed down AND its post-drain closedown event has fired — closedownEmitted guarantees the terminal disconnects have drained, so in-flight outputs always deliver before the end. The check runs on closedown events only; the deactivation path no longer triggers it, because a deactivated component may still have drains in flight. A network whose boundary inputs never terminate (long-running graphs, remote subgraphs, generators) never ends — end of transmission is owned by the boundary, not inferred from internal quiet. Exploratory (stream-termination, work document #45): the network-end rework exposed the inner-network re-open question — a subgraph whose inner network closed down and whose boundary re-opens must restart cleanly; the remaining suite failures document the class.
… inputs; spec components hold activations asPromise is the boundary for its wrapped run: after each input-map batch the wrapper performs the sender's disconnect, so the wrapped network's inputs reach EOF and the network closes deterministically. Spec components updated to the activation contract: deferred sends resolve their promise with the value (the bridge sends in-activation) — the previous send-then-resolve pattern double-done'd the bridge, re-activating a deactivated context, and out-of-activation sends re-opened a closed-down downstream that could never close again. Subgraph drivers signal per-burst EOF (the driver is the boundary), the two-level trace expectation reflects wrappers closing down with their inner networks, and "should finish" awaits the end event, whose arrival now trails the data flow deterministically.
generator joins the ComponentOptions typedef; the #identifier methods drop their conflicting @Private tags; the InPort detach param types the socket through the namespace member.
…, wrapper EOF, late-attach invariant Four fixes from spinning the branch against the jekyll pipeline (#1092): - The closedown and network-end gates counted buffered bracket framing as pending work: a stranded forwarded closeBracket held a component open forever. Buffered data on an end-of-stream port is stranded by definition — no arrival can complete a waiting context once every input is at EOF — so the gates no longer count buffers at all; the strand surfaces loudly (the trace's last activation, the shutdown diagnostics). - The network-end predicate moves to the network level: all boundary inputs (the graph's inport exports) at EOF plus quiescence (no open activations, no packet in flight). The per-component closedown gate deadlocked on cycles — a feedback edge never EOFs, so mutual closedown never terminates — while a quiescent network with all sources finished is absorbing regardless of cycles. - Subgraph wrappers set their EOF marker with the closedown marking: the outer network's predicate could never pass for a subgraph-containing network without it (the wrapper has no disconnects left to drain — the inner closedowns already terminally closed the shared outports). - An outport socket attaching to a closed-down sender goes terminal immediately: a late-attached edge must not leave its receiving port waiting for an EOF that will never fire. The closing evaluation is debounced (a coalescing window, default 10ms, option closingDebounceMs): state transitions mark the predicate dirty, one evaluation runs per window — scheduling only, the predicate itself is the deterministic end condition. This keeps the 10k-node stress test at ~1.4s (it was 199s under per-event evaluation). New spec: spec/NetworkClosing.js pins the sequencing guarantees — every closedown before the network end, a revived component's second closedown gating the end, and the nested order (inner end, wrapper closedown, outer end).
The drivers are the boundary: every spec that drives a network through raw sockets now signals end-of-input when its inputs are exhausted — scoped bursts in Scoping and NetworkLifecycle, the subgraph per-burst drivers, the two-level trace expectations under the wrapper lifecycle. asPromise performs the sender's disconnect after each input-map batch: the wrapped run's inputs reach EOF and the network closes deterministically, per batch.
A graph that finishes immediately under drain-based closing (quiescence plus boundary EOF) raced the mesh attach: the batch exit fired inside createHost, before the runtime had announced on the mesh. The attach and bind now precede host.start(), so the announce always precedes the run — the boundary owns end-of-transmission, and the shutdown owns teardown.
…overy rns.ready() awaits the AutoInterface's discovery start promise, which on hosts without usable multicast connectivity (CI containers) never settles: the interface-less path dispatches its error without resolving the start. Moving the mesh attach before the network start put that hang on the critical path — the host now gates on the persistor hydration only, and runs whether or not the mesh comes up (work document #45 consumer contracts).
bergie
added a commit
to noflo/noflo-jekyll
that referenced
this pull request
Oct 16, 2026
The wrapped adapters' collector cross-pairing is fixed with unique scopes: ExpandPosts stamps each post (its path), the builder stamps each generated page. On the closedown build (noflo/noflo#1092) the network stays live while a scoped message waits buffered, and the per-scope pairing is immune to async inner completion order. CopyStatic no longer removes directories: the empty-tree rm raced the document pipeline's writes into the same path (baz/ holds both the ignored _posts tree and the rendered baz page), deleting rendered output
Under Deno the node-compat setTimeout returns an object with a real unref, so the closing evaluation's unref'd debounce timer let the event loop drain before the evaluation ran — every network-driven suite failed with "Promise resolution is still pending but the event loop has already resolved". The closing timer is the pending work keeping the loop alive while the network drains: it is intentionally ref'd. Also removes the leftover #evaluateClosing instrumentation whose countBufferedData reference broke under Deno after the method's removal, and pins the revive-cycle contract in spec/NetworkClosing.js: each burst produces exactly one closedown, and the network end fires after the last one.
…view finding 1) An internal edge targeting an exported inport shares the boundary port: its socket can only go terminal when its sender closes down, and the sender waits on the boundary component's closedown — the every-socket-must-be-terminal boundary check deadlocked feedback graphs (A/out -> B/in, B/out -> A/in with A/in exported) exactly like the per-component gate the predicate replaced. The boundary check now requires terminal state only from external sockets: sockets whose source is not one of this network's processes (driver-attached sockets and IIP sockets). Internal edges targeting the same port are skipped — their terminal state is already covered by the quiescence gate, since a quiescent network with all boundary sources finished is absorbing regardless of cycles. The skip is per-network membership: a parent's edge into a subgraph's exported inport has its source in the parent, so it still gates the subgraph's own end. IIP sockets gate (their source is null; they terminal after delivering their single packet). The reviewer's repro is pinned as a NetworkClosing spec: the feedback graph ends instead of sitting quiescent forever.
…view findings 2-3) The [Unreleased] segment documents the behavior-changing series: drain-based network closing, sending no longer disconnecting, the load-bearing activation contract, the generator option, the sendIP signature change, and the new closedown/EOF API surface. Also: the dangling endCheckTimer docblock and the InPort stray blank line are removed, and a rejected terminal disconnect during closedown is logged at debug level instead of being swallowed silently.
bergie
marked this pull request as ready for review
October 16, 2026 23:19
…w question) Self-driven components are protected by the closedown gate's load check alone when they follow the hold-the-context idiom: the start activation stays open, every emission sends within it, and the context completes on the stop packet or teardown. The legacy interval-generator fixture cycled its context per tick (re-activating a completed context) — a load-0 window that made the generator flag look load-bearing — but that pattern is the early-Process-API artifact, not the idiom to protect. The real migrated generator (modernization/noflo-core RunInterval) already holds the context correctly. The flag also guarded an impossible pattern: a component with all inputs at EOF and no open activation can never self-activate (the engine has no input-less firing), so closing it down is correct — and a late input re-opens and revives it. The generator fixtures are rewritten to the hold-the-context idiom (modeled on RunInterval): the start activation held across ticks, the stop packet closing it via the stored deactivate handle, teardown completing the rest. The CHANGELOG's activation-contract entry drops the generator clause.
The boundary concept is end of transmission (EOT), not end-of-stream — it is the sender-side fact the 0.x disconnect carried. The per-layer terms consolidate into one family, each derived from the one above: - socket terminal: the sender finished this transmission; connect re-opens it - port EOT (InPort#isEot): every attached socket terminal - component closedDown: every input at EOT, quiescent, outports terminally disconnected - network ended: every boundary input at EOT and quiescent A flowtrace reads bottom-up: sockets close, ports reach EOT, components close down, the network ends. The internal Eof method names are renamed to match (setEot/refreshEot), and the prose uses end of transmission throughout.
…ingCheck (PR #1092 review round 3) The EOF-to-EOT prose sweep finishes the vocabulary consolidation: EOT is the one term for the input fact, end of transmission for the boundary concept — end-of-stream and end-of-input phrasings are gone from the closing machinery, the specs, and the CHANGELOG. checkIfFinished() is renamed to scheduleClosingCheck(): the 1.x name checked a quiet-window timer; the method now only marks the closing predicate dirty and schedules the one debounced evaluation, so the call sites in subscribeSocket and subscribeNode read as what they do.
bergie
added a commit
to noflo/noflo-jekyll
that referenced
this pull request
Oct 17, 2026
The activation-contract fixes from the noflo/noflo#1092 review: the scan, the static split, the copier, and the document loader returned void and ran their asynchronous work fire-and-forget — the network looked quiescent while packets were still parking in edges, and the corpus emerged into the ended network. The async work now returns its promise, holding the activation until the sends complete in-activation. DocumentBuilder: documents consumed on arrival into a per-scope state store — waiting in the port buffer stranded them behind their scope, because the config and template re-fires are unscoped and a scopeless activation cannot see scoped buffers. Every activation drains every scope's queue head-first; the enrichment runs at consumption time
bergie
added a commit
that referenced
this pull request
Oct 17, 2026
…y (noflo-jekyll review finding)
A data send on a deactivated process context — the process resolved
void ("already sent") and a deferred send fired afterwards — races
drain-based closing and re-opens an ended network that can no longer
trust end of transmission. The guard surfaces it as a process error
naming the component, sent through the error port bypassing the ordered
output queue (the activation is already resolved; a queued error would
never deliver). Error escalations are exempt: reporting a failure after
the activation closed is the engine's own catch path.
Found reviewing the noflo-jekyll corpus race against PR #1092's closing
semantics. The investigation also verified the closing predicate
itself: at every observed premature-looking end the state was genuinely
absorbing (every feeding socket terminal, no in-flight edge data, no
open activations) — the strands were real, and the fix belongs to the
components (hold the activation; consume documents on arrival into
component state rather than waiting in scope-fragmented port buffers).
The guard is what makes the violating pattern diagnosable from the
artifact instead of silent.
Specs: the guard fires for the promise-resolved-then-send pattern and
stays silent for the compliant form (the promise return holds the
activation until the send completes). Two NetworkClosing tests pin the
driver-stream and waiting-context behaviors
bergie
added a commit
that referenced
this pull request
Oct 17, 2026
A synchronous process function return that opened the context (a read) but consumed nothing and sent nothing can never complete: the activation leaks, the component's load never reaches zero, and the network never ends. Found by the noflo-jekyll PR #1092 follow-up, where a control-port corpus read followed by a render gate leaked the inner Template's activation and stalled the whole closedown cascade — the reported EOF-propagation theory was disproven by a minimal repro (the boundary sockets were terminal; the inner network's boundary ledger works), and the state dump showed the open activation. The engine now logs a debug-level warning naming the component at the leak point. It deliberately does not auto-close the activation: the legal deferred-completion patterns consume data or send before their synchronous return (distinguished mechanically by IP ownership transfer, which control reads never do), and auto-closing would break the deferral contract. Peeking into buffered contents with has()/ hasData()/hasStream() validators before reading is the sanctioned precondition mechanism — peeking never activates. Specs: the read-then-decline leak persists (pinned so an auto-close cannot slip in silently), and the legal deferred-completion pattern completes normally
bergie
added a commit
that referenced
this pull request
Oct 17, 2026
…investigation) Instrumented investigation of the PR #1092 review's cycle-graph observations (work document #46): with per-run instrumentation — each network start tagged, every delivery counted per run — the minimal 1-IIP + 1-echo cycle delivers exactly two arrivals per run (the IIP and the echo's return), an explicit restart re-posts the graph's IIPs (the start re-lists nextInitials; each run receives its initial inputs again), and component instance state persists across restarts. The 4-vs-2 observation is explained as two runs of correct per-run delivery. The fan-out miss (a late-attached collector missing its delivery while the edge received extra packets) does not reproduce against current master in any reconstructed shape: fan-out to a collector attached mid-run, attached to an ended network (the source closed down, sockets terminally disconnected), and across restarts all deliver correctly to every attached edge. Specs pin the current semantics while work document #45 gate 3 owns the per-run vs once-per-lifetime re-delivery decision, so it cannot drift silently: the cycle count, the restart re-delivery, and the late-attached fan-out edge
bergie
added a commit
that referenced
this pull request
Oct 19, 2026
…ession pin (work document #48) Distills the noflo-jekyll document leg (work document #35 updates #14-#22) to its engine shape and runs it through the actual wrapAssembly machinery: a gate-model render component — standing bracket-framed config on a scoped:false control port read non-consumingly, per-message scoped data on two ports gated on both — wrapped as an assembly adapter (Fork, field extractors, CollectFields), driven with bracket-framed scoped message envelopes whose arrival order races the config and stagger with async jitter inside open activations, then driven to EOT with the closedown cascade asserted. The negative result this suite pins: on the current engine (post-#1094 control gates, #1092 drain-based closing, #1099 drain pump), roughly 1300 randomized runs across staggered, burst, bounded-edge, multi- registration-config, and deep-overlap variants delivered every message and closed down — the intermittent drop class the testbed documented did not reproduce. Three randomized variants run in CI; the full matrix rides the WD48_* env knobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.