8000 Stream termination by bergie · Pull Request #1092 · noflo/noflo · GitHub
[go: up one dir, main page]

Skip to content

Stream termination - #1092

Merged
bergie merged 16 commits into
masterfrom
stream-termination
Oct 17, 2026
Merged

bergie merged 16 commits into
masterfrom
stream-termination

Conversation

@bergie
@bergie bergie commented Oct 16, 2026
Copy link
Copy Markdown
Member

No description provided.

bergie added 10 commits October 17, 2026 01:28
Sending no longer ends the transmission: InternalSocket.post auto-connects
but never auto-disconnects, and the connection stays open after the send.
disconnect() is the sender's explicit decision that nothing more will come
from it — the 0.x intent, per work document #40 update #5 — which per-burst
framing could never express. The same-tick auto-disconnect was a 1.x
timing-compat shim; with every component on the Process API and every graph
redesigned for 2.x, the framing it provided is redundant with in-band
brackets.

Exploratory (stream-termination): the suite is knowingly red on tests whose
finalization rode the outport's last auto-disconnect — ordered bracket
forwarding, subgraph bracket forwarding, scope isolation, NetworkLifecycle
merging. The replacement is EOF propagation (input disconnect = final
activation = output finish), landing in the next commits.
An IIP is one packet from a sender that is then done: the runtime
performs the sender's disconnect once the packet has been delivered
through the edge (drain-then-close, so the packet never overtakes its
own EOF). This is the seed of cascading closedown — receiving ports can
mark the input at EOF once the transport event reaches them.

Exploratory (stream-termination): inert until ports track EOF and
components derive closedown from it.
InternalSocket gains terminal-close semantics: disconnect() marks the
socket terminally closed (idempotent, emitted even when never connected,
so a silent sender still EOFs its downstream), and connect() re-opens a
terminal socket — connect is the beginning of transmission.

InPort tracks end-of-stream per port, unscoped by definition: EOF means
"there will be nothing more coming from this sender", so a component
closing down puts every attached edge at EOF in every scope at once. The
eof lifecycle event fires on both transitions — entering EOF and a new
transmission re-opening the input.

Component derives the classical cascading closedown from it: when every
input is at EOF, no activation is open, and no buffered deliverable data
remains (control ports excluded — standing values are legitimate at
quiescence), the component terminally disconnects its outports, and the
closedown event fires only after those disconnects have drained. An
input re-opening revives a closed-down component. Self-driven components
declare generator: true — input EOF must not close down a component that
activates from its own clock.

Subgraph wrappers close down with their inner networks: the shared-port
EOF reflects the deepest sender's state, not the wrapper's completion,
so the wrapper's closedown is driven by the inner network's end.

Exploratory (stream-termination, work documents #40 update #5, #45).
checkIfFinished becomes a pure predicate: the network ends when every
process has closed down AND its post-drain closedown event has fired —
closedownEmitted guarantees the terminal disconnects have drained, so
in-flight outputs always deliver before the end. The check runs on
closedown events only; the deactivation path no longer triggers it,
because a deactivated component may still have drains in flight.

A network whose boundary inputs never terminate (long-running graphs,
remote subgraphs, generators) never ends — end of transmission is owned
by the boundary, not inferred from internal quiet.

Exploratory (stream-termination, work document #45): the network-end
rework exposed the inner-network re-open question — a subgraph whose
inner network closed down and whose boundary re-opens must restart
cleanly; the remaining suite failures document the class.
… inputs; spec components hold activations

asPromise is the boundary for its wrapped run: after each input-map
batch the wrapper performs the sender's disconnect, so the wrapped
network's inputs reach EOF and the network closes deterministically.

Spec components updated to the activation contract: deferred sends
resolve their promise with the value (the bridge sends in-activation) —
the previous send-then-resolve pattern double-done'd the bridge,
re-activating a deactivated context, and out-of-activation sends
re-opened a closed-down downstream that could never close again.

Subgraph drivers signal per-burst EOF (the driver is the boundary), the
two-level trace expectation reflects wrappers closing down with their
inner networks, and "should finish" awaits the end event, whose arrival
now trails the data flow deterministically.
generator joins the ComponentOptions typedef; the #identifier methods
drop their conflicting @Private tags; the InPort detach param types the
socket through the namespace member.
…, wrapper EOF, late-attach invariant

Four fixes from spinning the branch against the jekyll pipeline (#1092):

- The closedown and network-end gates counted buffered bracket framing as
  pending work: a stranded forwarded closeBracket held a component open
  forever. Buffered data on an end-of-stream port is stranded by
  definition — no arrival can complete a waiting context once every input
  is at EOF — so the gates no longer count buffers at all; the strand
  surfaces loudly (the trace's last activation, the shutdown
  diagnostics).
- The network-end predicate moves to the network level: all boundary
  inputs (the graph's inport exports) at EOF plus quiescence (no open
  activations, no packet in flight). The per-component closedown gate
  deadlocked on cycles — a feedback edge never EOFs, so mutual closedown
  never terminates — while a quiescent network with all sources finished
  is absorbing regardless of cycles.
- Subgraph wrappers set their EOF marker with the closedown marking: the
  outer network's predicate could never pass for a subgraph-containing
  network without it (the wrapper has no disconnects left to drain — the
  inner closedowns already terminally closed the shared outports).
- An outport socket attaching to a closed-down sender goes terminal
  immediately: a late-attached edge must not leave its receiving port
  waiting for an EOF that will never fire.

The closing evaluation is debounced (a coalescing window, default 10ms,
option closingDebounceMs): state transitions mark the predicate dirty,
one evaluation runs per window — scheduling only, the predicate itself
is the deterministic end condition. This keeps the 10k-node stress test
at ~1.4s (it was 199s under per-event evaluation).

New spec: spec/NetworkClosing.js pins the sequencing guarantees — every
closedown before the network end, a revived component's second closedown
gating the end, and the nested order (inner end, wrapper closedown,
outer end).
The drivers are the boundary: every spec that drives a network through
raw sockets now signals end-of-input when its inputs are exhausted —
scoped bursts in Scoping and NetworkLifecycle, the subgraph per-burst
drivers, the two-level trace expectations under the wrapper lifecycle.

asPromise performs the sender's disconnect after each input-map batch:
the wrapped run's inputs reach EOF and the network closes
deterministically, per batch.
A graph that finishes immediately under drain-based closing (quiescence
plus boundary EOF) raced the mesh attach: the batch exit fired inside
createHost, before the runtime had announced on the mesh. The attach and
bind now precede host.start(), so the announce always precedes the run —
the boundary owns end-of-transmission, and the shutdown owns teardown.
…overy

rns.ready() awaits the AutoInterface's discovery start promise, which on
hosts without usable multicast connectivity (CI containers) never
settles: the interface-less path dispatches its error without resolving
the start. Moving the mesh attach before the network start put that hang
on the critical path — the host now gates on the persistor hydration
only, and runs whether or not the mesh comes up (work document #45
consumer contracts).
bergie added a commit to noflo/noflo-jekyll that referenced this pull request Oct 16, 2026
The wrapped adapters' collector cross-pairing is fixed with unique
scopes: ExpandPosts stamps each post (its path), the builder stamps
each generated page. On the closedown build (noflo/noflo#1092) the
network stays live while a scoped message waits buffered, and the
per-scope pairing is immune to async inner completion order. CopyStatic
no longer removes directories: the empty-tree rm raced the document
pipeline's writes into the same path (baz/ holds both the ignored
_posts tree and the rendered baz page), deleting rendered output
Under Deno the node-compat setTimeout returns an object with a real
unref, so the closing evaluation's unref'd debounce timer let the event
loop drain before the evaluation ran — every network-driven suite failed
with "Promise resolution is still pending but the event loop has already
resolved". The closing timer is the pending work keeping the loop alive
while the network drains: it is intentionally ref'd.

Also removes the leftover #evaluateClosing instrumentation whose
countBufferedData reference broke under Deno after the method's removal,
and pins the revive-cycle contract in spec/NetworkClosing.js: each burst
produces exactly one closedown, and the network end fires after the last
one.
…view finding 1)

An internal edge targeting an exported inport shares the boundary port:
its socket can only go terminal when its sender closes down, and the
sender waits on the boundary component's closedown — the
every-socket-must-be-terminal boundary check deadlocked feedback graphs
(A/out -> B/in, B/out -> A/in with A/in exported) exactly like the
per-component gate the predicate replaced.

The boundary check now requires terminal state only from external
sockets: sockets whose source is not one of this network's processes
(driver-attached sockets and IIP sockets). Internal edges targeting the
same port are skipped — their terminal state is already covered by the
quiescence gate, since a quiescent network with all boundary sources
finished is absorbing regardless of cycles.

The skip is per-network membership: a parent's edge into a subgraph's
exported inport has its source in the parent, so it still gates the
subgraph's own end. IIP sockets gate (their source is null; they
terminal after delivering their single packet).

The reviewer's repro is pinned as a NetworkClosing spec: the feedback
graph ends instead of sitting quiescent forever.
…view findings 2-3)

The [Unreleased] segment documents the behavior-changing series:
drain-based network closing, sending no longer disconnecting, the
load-bearing activation contract, the generator option, the
sendIP signature change, and the new closedown/EOF API surface.

Also: the dangling endCheckTimer docblock and the InPort stray blank
line are removed, and a rejected terminal disconnect during closedown
is logged at debug level instead of being swallowed silently.
@bergie
bergie marked this pull request as ready for review October 16, 2026 23:19
…w question)

Self-driven components are protected by the closedown gate's load check
alone when they follow the hold-the-context idiom: the start activation
stays open, every emission sends within it, and the context completes on
the stop packet or teardown. The legacy interval-generator fixture cycled
its context per tick (re-activating a completed context) — a load-0
window that made the generator flag look load-bearing — but that pattern
is the early-Process-API artifact, not the idiom to protect. The real
migrated generator (modernization/noflo-core RunInterval) already holds
the context correctly.

The flag also guarded an impossible pattern: a component with all inputs
at EOF and no open activation can never self-activate (the engine has no
input-less firing), so closing it down is correct — and a late input
re-opens and revives it.

The generator fixtures are rewritten to the hold-the-context idiom
(modeled on RunInterval): the start activation held across ticks, the
stop packet closing it via the stored deactivate handle, teardown
completing the rest. The CHANGELOG's activation-contract entry drops the
generator clause.
The boundary concept is end of transmission (EOT), not end-of-stream —
it is the sender-side fact the 0.x disconnect carried. The per-layer
terms consolidate into one family, each derived from the one above:

- socket terminal: the sender finished this transmission; connect
  re-opens it
- port EOT (InPort#isEot): every attached socket terminal
- component closedDown: every input at EOT, quiescent, outports
  terminally disconnected
- network ended: every boundary input at EOT and quiescent

A flowtrace reads bottom-up: sockets close, ports reach EOT, components
close down, the network ends. The internal Eof method names are renamed
to match (setEot/refreshEot), and the prose uses end of transmission
throughout.
…ingCheck (PR #1092 review round 3)

The EOF-to-EOT prose sweep finishes the vocabulary consolidation: EOT is
the one term for the input fact, end of transmission for the boundary
concept — end-of-stream and end-of-input phrasings are gone from the
closing machinery, the specs, and the CHANGELOG.

checkIfFinished() is renamed to scheduleClosingCheck(): the 1.x name
checked a quiet-window timer; the method now only marks the closing
predicate dirty and schedules the one debounced evaluation, so the call
sites in subscribeSocket and subscribeNode read as what they do.
@bergie
bergie merged commit c4a37c5 into master Oct 17, 2026
8 checks passed
@bergie
bergie deleted the stream-termination branch October 17, 2026 01:48
bergie added a commit to noflo/noflo-jekyll that referenced this pull request Oct 17, 2026
The activation-contract fixes from the noflo/noflo#1092 review: the
scan, the static split, the copier, and the document loader returned
void and ran their asynchronous work fire-and-forget — the network
looked quiescent while packets were still parking in edges, and the
corpus emerged into the ended network. The async work now returns its
promise, holding the activation until the sends complete in-activation.

DocumentBuilder: documents consumed on arrival into a per-scope state
store — waiting in the port buffer stranded them behind their scope,
because the config and template re-fires are unscoped and a scopeless
activation cannot see scoped buffers. Every activation drains every
scope's queue head-first; the enrichment runs at consumption time
bergie added a commit that referenced this pull request Oct 17, 2026
…y (noflo-jekyll review finding)

A data send on a deactivated process context — the process resolved
void ("already sent") and a deferred send fired afterwards — races
drain-based closing and re-opens an ended network that can no longer
trust end of transmission. The guard surfaces it as a process error
naming the component, sent through the error port bypassing the ordered
output queue (the activation is already resolved; a queued error would
never deliver). Error escalations are exempt: reporting a failure after
the activation closed is the engine's own catch path.

Found reviewing the noflo-jekyll corpus race against PR #1092's closing
semantics. The investigation also verified the closing predicate
itself: at every observed premature-looking end the state was genuinely
absorbing (every feeding socket terminal, no in-flight edge data, no
open activations) — the strands were real, and the fix belongs to the
components (hold the activation; consume documents on arrival into
component state rather than waiting in scope-fragmented port buffers).
The guard is what makes the violating pattern diagnosable from the
artifact instead of silent.

Specs: the guard fires for the promise-resolved-then-send pattern and
stays silent for the compliant form (the promise return holds the
activation until the send completes). Two NetworkClosing tests pin the
driver-stream and waiting-context behaviors
bergie added a commit that referenced this pull request Oct 17, 2026
A synchronous process function return that opened the context (a read)
but consumed nothing and sent nothing can never complete: the
activation leaks, the component's load never reaches zero, and the
network never ends. Found by the noflo-jekyll PR #1092 follow-up, where
a control-port corpus read followed by a render gate leaked the inner
Template's activation and stalled the whole closedown cascade — the
reported EOF-propagation theory was disproven by a minimal repro (the
boundary sockets were terminal; the inner network's boundary ledger
works), and the state dump showed the open activation.

The engine now logs a debug-level warning naming the component at the
leak point. It deliberately does not auto-close the activation: the
legal deferred-completion patterns consume data or send before their
synchronous return (distinguished mechanically by IP ownership
transfer, which control reads never do), and auto-closing would break
the deferral contract. Peeking into buffered contents with has()/
hasData()/hasStream() validators before reading is the sanctioned
precondition mechanism — peeking never activates.

Specs: the read-then-decline leak persists (pinned so an auto-close
cannot slip in silently), and the legal deferred-completion pattern
completes normally
bergie added a commit that referenced this pull request Oct 17, 2026
…investigation)

Instrumented investigation of the PR #1092 review's cycle-graph
observations (work document #46): with per-run instrumentation — each
network start tagged, every delivery counted per run — the minimal
1-IIP + 1-echo cycle delivers exactly two arrivals per run (the IIP and
the echo's return), an explicit restart re-posts the graph's IIPs (the
start re-lists nextInitials; each run receives its initial inputs
again), and component instance state persists across restarts. The
4-vs-2 observation is explained as two runs of correct per-run
delivery.

The fan-out miss (a late-attached collector missing its delivery while
the edge received extra packets) does not reproduce against current
master in any reconstructed shape: fan-out to a collector attached
mid-run, attached to an ended network (the source closed down, sockets
terminally disconnected), and across restarts all deliver correctly to
every attached edge.

Specs pin the current semantics while work document #45 gate 3 owns
the per-run vs once-per-lifetime re-delivery decision, so it cannot
drift silently: the cycle count, the restart re-delivery, and the
late-attached fan-out edge
bergie added a commit that referenced this pull request Oct 19, 2026
…ession pin (work document #48)

Distills the noflo-jekyll document leg (work document #35 updates
#14-#22) to its engine shape and runs it through the actual wrapAssembly
machinery: a gate-model render component — standing bracket-framed
config on a scoped:false control port read non-consumingly, per-message
scoped data on two ports gated on both — wrapped as an assembly adapter
(Fork, field extractors, CollectFields), driven with bracket-framed
scoped message envelopes whose arrival order races the config and
stagger with async jitter inside open activations, then driven to EOT
with the closedown cascade asserted.

The negative result this suite pins: on the current engine (post-#1094
control gates, #1092 drain-based closing, #1099 drain pump), roughly
1300 randomized runs across staggered, burst, bounded-edge, multi-
registration-config, and deep-overlap variants delivered every message
and closed down — the intermittent drop class the testbed documented
did not reproduce. Three randomized variants run in CI; the full matrix
rides the WD48_* env knobs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0