8000 Make sending anything wait until downstream component is ready by kenhkan · Pull Request #40 · noflo/noflo · GitHub
[go: up one dir, main page]

Skip to content

Make sending anything wait until downstream component is ready - #40

Merged
bergie merged 3 commits into
noflo:masterfrom
kenhkan:buffer_until_ready
Oct 1, 2012
Merged

bergie merged 3 commits into
noflo:masterfrom
kenhkan:buffer_until_ready

Conversation

@kenhkan
@kenhkan kenhkan commented Aug 19, 2012
Copy link
Copy Markdown
Contributor

This is handy when you're doing something like:

'Initial' -> IN Merge(Merge) OUT -> PROCESS.IN SomeComponent()

where SomeComponent may not be ready and the initial is already sent to Merge knowing nothing about the downstream components.

@travisbot
Copy link
8000 Copy Markdown

This pull request passes (merged 4df50bb into 5f4fcfa).

@travisbot
Copy link
Copy Markdown

This pull request passes (merged bb02012 into 5f4fcfa).

@kenhkan
kenhkan commented Aug 21, 2012
Copy link
Copy Markdown
Contributor Author

So this turned out to be more of a systemic problem than I thought. Here's a documentation of what I went through.

Currently (i.e. w/o this patch), if you do:

   'graph.fbp' -> GRAPH Graph(Graph)
   'A' -> PROCESS.IN Graph()

This would work, but not this:

   'graph.fbp' -> GRAPH Graph(Graph)
   'A' -> IN Merge(Merge) OUT -> PROCESS.IN Graph()

This is because only sending of initial tests for the readiness of the immediate downstream process. My initial patch solves this.

But this wouldn't work with the patch:

   'graph.fbp' -> GRAPH Graph(Graph)
   'A' -> IN Merge(Merge) OUT -> PROCESS.IN Graph() PROCESS.OUT -> IN Output(Output)

IPs are buffered before PROCESS.IN when Graph is not ready, but as soon as it's ready, a 'ready' event is emitted and the inport connection between the upstream process (i.e. Merge) and the graph is made before it has time to make a connection between PROCESS.OUT and the downstream, Output process.

Because of how 'Graph' is currently implemented, the above is in reality something this:

   'A' -> IN Merge(Merge) OUT -> IN * OUT -> IN PROCESS OUT -> IN * OUT -> IN Output(Output)

The interfacing pseudo-ports are set up during addEdge after everything has set up so I need to make IPs buffer until all addEdges have been called. Since addEdge is called by an event and I have no control over which addEdge is called first, I have to make a stack and flush everything at the end when the stack is emptied.

I know there must be a less hacky way. I still haven't grasped the entire codebase so it's difficult for me to think at the highest level. Please let me know if you have any idea! Thanks! But for now, this works for all intents and purposes.

@kenhkan
kenhkan commented Aug 27, 2012
Copy link
Copy Markdown
Contributor Author

See #42

bergie added a commit that referenced this pull request Oct 1, 2012
Make sending anything wait until downstream component is ready
@bergie
bergie merged commit f7a0704 into noflo:master Oct 1, 2012
kenhkan added a commit to kenhkan/noflo that referenced this pull request May 1, 2013
Checking for downstream readiness was initially a problem as described in noflo#40. That was due to initial packets being sent before the network is ready. It's partially solved in noflo#69 when initial packets are only sent upon the respective network has been loaded. This PR solves the other half which is initial packets being sent in order of the programmer's specification so by the time the network is run, everything should be ready anyway.
bergie added a commit that referenced this pull request Oct 17, 2026
…work documents #40, #45)

The remote-runtime scenario for the boundary-owned end-of-transmission
contract: a client can now drive a stream into an exported inport over
the protocol — hold the boundary open across a packet sequence with back
pressure respected, then signal EOT and let the drain-based closing
cascade end the network.

- Engine: Network#addDriver attaches a persistent external driver socket
  to a resolved inport (the IIP path minus send/disconnect); the
  connection lifecycle of boundary sockets (no in-graph sender —
  driver-attached and IIP) is emitted as a network connection event and
  recorded in flowtraces as 0x0b CONNECTION_OPEN / 0x0c CONNECTION_CLOSE
  with the receiving side identified and the source nil
- Protocol: the 0x49/0x4A opcodes in the execution block with codec
  round-trips, the AsyncAPI channels, and the SPEC §7/§8 entries; the
  tracefile decoder handles the boundary (nil-source) connection payload
  form
- Runtime: the stream registry keyed per plane and port — 0x49 attaches
  and connects the driver, 0x47 routes through an open driver (the IIP
  single-packet path otherwise unchanged), 0x4A disconnects and deletes;
  duplicate opens and stray EOTs answer 0x17 REASON.INVALID_STATE. The
  stream lifecycle records as 0x0b/0x0c telemetry events

Without an open stream the wire behavior is unchanged: each 0x47
injection remains a self-contained single-packet stream. The stream
lifecycle is pinned by specs in all three packages: the engine's
driver-held-open test, the codec round-trips, and the runtime's
open → packets → EOT → network-end sequence with its rejections.
bergie added a commit that referenced this pull request Oct 17, 2026
Declarable framing termination, stranded-framing diagnostics, migration-guide contracts (#40)
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 19, 2026
Sending no longer ends the transmission: InternalSocket.post auto-connects
but never auto-disconnects, and the connection stays open after the send.
disconnect() is the sender's explicit decision that nothing more will come
from it — the 0.x intent, per work document noflo#40 update noflo#5 — which per-burst
framing could never express. The same-tick auto-disconnect was a 1.x
timing-compat shim; with every component on the Process API and every graph
redesigned for 2.x, the framing it provided is redundant with in-band
brackets.

Exploratory (stream-termination): the suite is knowingly red on tests whose
finalization rode the outport's last auto-disconnect — ordered bracket
forwarding, subgraph bracket forwarding, scope isolation, NetworkLifecycle
merging. The replacement is EOF propagation (input disconnect = final
activation = output finish), landing in the next commits.
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 19, 2026
InternalSocket gains terminal-close semantics: disconnect() marks the
socket terminally closed (idempotent, emitted even when never connected,
so a silent sender still EOFs its downstream), and connect() re-opens a
terminal socket — connect is the beginning of transmission.

InPort tracks end-of-stream per port, unscoped by definition: EOF means
"there will be nothing more coming from this sender", so a component
closing down puts every attached edge at EOF in every scope at once. The
eof lifecycle event fires on both transitions — entering EOF and a new
transmission re-opening the input.

Component derives the classical cascading closedown from it: when every
input is at EOF, no activation is open, and no buffered deliverable data
remains (control ports excluded — standing values are legitimate at
quiescence), the component terminally disconnects its outports, and the
closedown event fires only after those disconnects have drained. An
input re-opening revives a closed-down component. Self-driven components
declare generator: true — input EOF must not close down a component that
activates from its own clock.

Subgraph wrappers close down with their inner networks: the shared-port
EOF reflects the deepest sender's state, not the wrapper's completion,
so the wrapper's closedown is driven by the inner network's end.

Exploratory (stream-termination, work documents noflo#40 update noflo#5, noflo#45).
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 19, 2026
…s brackets at arrival

An explicitly-declared empty forwarding rule — c.forwardBrackets =
{ msg: [] } — terminates bracket framing on that port: brackets are
consumed at arrival, they never fire the process, never buffer, and no
output context is mirrored. The component reads pure data.

This is the declarable fix for per-packet readers of bracketed streams,
which otherwise strand the trailing closeBracket: getData silently eats
the leading brackets, and the trailing one arrives after the last data,
matches no data-read pattern, and is declined forever. The declaration
replaces what no amount of pattern analysis can infer — the component
states its intent, and the engine can act on it soundly. The built-in
default's silent adaptation (a default rule whose outports do not exist
is deleted) is unchanged; only an explicitly-declared empty rule gains
the termination semantics.

Work document noflo#40 update noflo#13
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 19, 2026
…last sender disconnects

When an input port's last attached socket terminally disconnects, any
bracket framing still buffered on the port — in any scope or index —
can never pair with arriving data: the future is known. The residue is
surfaced at that moment, per port and per scope: a debug-level log
naming the component, and a 0x0f STRANDED event on the network event
bus and in Flowtrace (EVENT_TYPE.STRANDED, payload [node, port,
scope|null, open, close]).

The residue is reported, never auto-consumed: a component may
legitimately decline a closeBracket now and consume it after its other
input arrives, so silent reaping is unsound — the loudness is the
contract enforcement counterpart. This moves the discovery of the
stranded-bracket class from the shutdown buffer dump to the moment it
became inevitable, and gives the flowtrace artifact the evidence
(work document noflo#40 update noflo#13).

fbp-protocol pins the 0x0f event type and its payload rule (SPEC);
the tracefile round-trip spec covers the flat tuple passing through
the generic payload path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0