8000 Switch to cheerio for scraping. by rybesh · Pull Request #28 · noflo/noflo · GitHub
[go: up one dir, main page]

Skip to content

Switch to cheerio for scraping. - #28

Merged
bergie merged 1 commit into
noflo:masterfrom
rybesh:scrape-html
May 23, 2012
Merged

bergie merged 1 commit into
noflo:masterfrom
rybesh:scrape-html

Conversation

@rybesh
@rybesh rybesh commented May 23, 2012
Copy link
Copy Markdown
Contributor

This is WAY faster than the previous implementation.

bergie added a commit that referenced this pull request May 23, 2012
Switch to cheerio for scraping.
@bergie
bergie merged commit 9d58eed into noflo:master May 23, 2012
bergie added a commit that referenced this pull request May 23, 2012
bergie added a commit that referenced this pull request Oct 9, 2026
Work document #28 (proposed): the isomorphic FBP Protocol 2.0 runtime for
NoFlo over Reticulum, superseding the noflo-runtime-base /
noflo-runtime-websocket lineage — Reticulum-only, protocol-2.0-only,
capabilities as the 0x02 mask instead of protocol: strings.

Isomorphic per the no-build browser contract (work document #18): plain
ESM, no Node builtins in the runtime core; the application supplies its
own Reticulum instance with platform-appropriate interfaces (TCP on
servers, WebSocket/WebRTC in browsers), matching @reticulum/core's
subpath-import design. Dependencies: the 2.x engine (@noflo/noflo,
@noflo/graph), the wire format (@noflo/fbp-protocol), the transport
(@reticulum/core), and @reticulum/lxmf for the offline telemetry drops
(wired behind a subpath export when telemetry lands).

The smoke test pins that the entrypoint and its full dependency graph
import cleanly.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 1 of work document #28: RuntimeServer, the frame-routing heart the
Reticulum binding feeds. It decodes inbound frames through
@noflo/fbp-protocol's decodeFrame dispatch, hands typed decoded frames to
handlers registered per command code, and emits outgoing frames through
injected send/broadcast functions — the core never inspects contexts
(over Reticulum, a context is the link a frame arrived on) and constructs
no transport, keeping it isomorphic.

- capabilitiesMask composes the 0x02 capability mask from names, failing
  loudly on typos; the default advertises the read surface (graph,
  metadata, telemetry, components read-only)
- authorize() sends the auth response unilaterally — the transport calls
  it on link_established with a verified identity
- undecodable frames (garbage, unknown opcodes, malformed payloads) and
  handler failures never throw out of handleFrame: they surface as
  undecodable / error events, since the protocol has no error channel
  for them and a runtime must survive hostile link traffic; commands
  with no handler yet emit unhandledframe
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 2 of work document #28: RegistryProtocol, the Two-Step Cache server
side, driven end-to-end through the runtime core in tests.

The runtime consumes an application-supplied catalog — the same philosophy
as the engine's application-supplied component registry (work documents
#6/#16): signatures are ready at handoff, and the handler derives the wire
state from them. A signature is exactly the 0x24 detail shape (declared
kind plus the full port field set), valid with no implementation behind
it, so manifest/sidecar-sourced stubs flow through unmodified.

- 0x20 syncs answer 0x21 on hash match, 0x22 manifest otherwise — the
  runtime always answers (work document #4 update #18)
- 0x23 details answer with only the requested known components; unknown
  names are omitted, and an empty response is an answer
- 0x25 source writes and 0x27 installs map onto optional catalog hooks
  (source and signature are orthogonal, so a write never relabels a
  component); missing hooks surface as unsupported events
- every catalog change re-derives sig_hash values and the registry_hash
  through the protocol package's canonical serializations

Supporting changes: the runtime core now accepts async handlers (WebCrypto
hashing is async), capturing rejections as error events like synchronous
throws; @noflo/fbp-protocol re-exports its ComponentDetail/PortInfo/
ManifestEntry typedefs from the entrypoint so dependents can type against
them.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 3 of work document #28: GraphProtocol, the 0x10/0x14 server side,
with the live GraphModel as the epoch.

- The epoch id is content-addressed: SHA-256 hex over the graph model's
  canonical serialization, computed lazily and cached until the graph
  changes — epoch comparison needs no clocks to be meaningful
- The sync handshake answers 0x11 on epoch match, and 0x12 with the new
  epoch plus a baseline resource hash otherwise; the baseline bytes come
  from an injected resource provider the transport backs (missing
  provider surfaces as an unsupported event, since stale clients cannot
  resynchronize without one)
- Inbound 0x14 ops apply through the model's mutators and rebroadcast to
  every client except the sender, converging the op log; echo
  suppression keeps the model's own events from re-emitting applied
  operations; tombstones are idempotent; dropped operations (metadata
  for unknown entities) do not propagate, because the runtime is the
  convergence point
- Outbound: model add/remove/change events map to the total op
  projection — inserts carry the entity definition without its id, IIPs
  unwrap from.data to the wire's data field, removals become tombstones,
  metadata changes become UI-metadata ops, graph-level metadata changes
  address nil entity_id — each stamped with the runtime's client id and
  an incrementing logical clock
- Node renames surface as unsupported events: they project at the
  changeset→protocol boundary (work document #4 §5), not in the runtime

The runtime core's broadcast now takes an exclusion context so
convergence broadcasts can skip the originating link.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 4 of work document #28: the 0x30/0x32 streaming telemetry surface,
fed by a real NoFlo network.

NetworkHost owns the network lifecycle: builds the NoFlo network from the
runtime's graph with an injected component loader (platform-neutral, per
work documents #6/#16), re-emits the network's EventTarget events so
protocol handlers observe a network before, after, and across network
recreations, and supports rebuild-on-discard. The engine gap it surfaces
is documented: the 2.x engine consumes the graph at connect() and does
not follow live graph changes, so CRDT operations applied while a network
runs update the wire state but not the running network until the engine
learns to follow graph deltas.

TelemetryProtocol maps host events onto the 0x32 vocabulary:
- data packets become DATA events with raw payloads; brackets become
  BEGIN_GROUP/END_GROUP with the group name
- network start/stop become LIFECYCLE START/STOP codes
- process errors become ERROR events with the exception string — except
  errors in the stub error class (default StubNotImplementedError), which
  become STUB_ERROR: the event type itself is the classification (#4
  update #9)
- payloads the wire cannot carry are frugalized to type-named
  placeholders so serialization can never take down the flush loop

Flush policy is the runtime's physical decision: per-subscription buffers
flush on timers at the effective cadence (the client's requested
interval, bounded below by a configurable floor), with explicit flushing
available for transports that prefer it. The protocol has no unsubscribe
command; subscriptions live with their link, and dropContext tears them
down on link close.

The integration test streams a real network run — a real ComponentLoader,
asComponent factory, graph with IIP — through the full pipeline to codec
round-trips. @noflo/noflo re-exports the Network class, restoring 1.x
surface parity that the host needed to type against.
bergie added a commit that referenced this pull request Oct 9, 2026
Work document #28 (proposed): the isomorphic FBP Protocol 2.0 runtime for
NoFlo over Reticulum, superseding the noflo-runtime-base /
noflo-runtime-websocket lineage — Reticulum-only, protocol-2.0-only,
capabilities as the 0x02 mask instead of protocol: strings.

Isomorphic per the no-build browser contract (work document #18): plain
ESM, no Node builtins in the runtime core; the application supplies its
own Reticulum instance with platform-appropriate interfaces (TCP on
servers, WebSocket/WebRTC in browsers), matching @reticulum/core's
subpath-import design. Dependencies: the 2.x engine (@noflo/noflo,
@noflo/graph), the wire format (@noflo/fbp-protocol), the transport
(@reticulum/core), and @reticulum/lxmf for the offline telemetry drops
(wired behind a subpath export when telemetry lands).

The smoke test pins that the entrypoint and its full dependency graph
import cleanly.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 1 of work document #28: RuntimeServer, the frame-routing heart the
Reticulum binding feeds. It decodes inbound frames through
@noflo/fbp-protocol's decodeFrame dispatch, hands typed decoded frames to
handlers registered per command code, and emits outgoing frames through
injected send/broadcast functions — the core never inspects contexts
(over Reticulum, a context is the link a frame arrived on) and constructs
no transport, keeping it isomorphic.

- capabilitiesMask composes the 0x02 capability mask from names, failing
  loudly on typos; the default advertises the read surface (graph,
  metadata, telemetry, components read-only)
- authorize() sends the auth response unilaterally — the transport calls
  it on link_established with a verified identity
- undecodable frames (garbage, unknown opcodes, malformed payloads) and
  handler failures never throw out of handleFrame: they surface as
  undecodable / error events, since the protocol has no error channel
  for them and a runtime must survive hostile link traffic; commands
  with no handler yet emit unhandledframe
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 2 of work document #28: RegistryProtocol, the Two-Step Cache server
side, driven end-to-end through the runtime core in tests.

The runtime consumes an application-supplied catalog — the same philosophy
as the engine's application-supplied component registry (work documents
#6/#16): signatures are ready at handoff, and the handler derives the wire
state from them. A signature is exactly the 0x24 detail shape (declared
kind plus the full port field set), valid with no implementation behind
it, so manifest/sidecar-sourced stubs flow through unmodified.

- 0x20 syncs answer 0x21 on hash match, 0x22 manifest otherwise — the
  runtime always answers (work document #4 update #18)
- 0x23 details answer with only the requested known components; unknown
  names are omitted, and an empty response is an answer
- 0x25 source writes and 0x27 installs map onto optional catalog hooks
  (source and signature are orthogonal, so a write never relabels a
  component); missing hooks surface as unsupported events
- every catalog change re-derives sig_hash values and the registry_hash
  through the protocol package's canonical serializations

Supporting changes: the runtime core now accepts async handlers (WebCrypto
hashing is async), capturing rejections as error events like synchronous
throws; @noflo/fbp-protocol re-exports its ComponentDetail/PortInfo/
ManifestEntry typedefs from the entrypoint so dependents can type against
them.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 3 of work document #28: GraphProtocol, the 0x10/0x14 server side,
with the live GraphModel as the epoch.

- The epoch id is content-addressed: SHA-256 hex over the graph model's
  canonical serialization, computed lazily and cached until the graph
  changes — epoch comparison needs no clocks to be meaningful
- The sync handshake answers 0x11 on epoch match, and 0x12 with the new
  epoch plus a baseline resource hash otherwise; the baseline bytes come
  from an injected resource provider the transport backs (missing
  provider surfaces as an unsupported event, since stale clients cannot
  resynchronize without one)
- Inbound 0x14 ops apply through the model's mutators and rebroadcast to
  every client except the sender, converging the op log; echo
  suppression keeps the model's own events from re-emitting applied
  operations; tombstones are idempotent; dropped operations (metadata
  for unknown entities) do not propagate, because the runtime is the
  convergence point
- Outbound: model add/remove/change events map to the total op
  projection — inserts carry the entity definition without its id, IIPs
  unwrap from.data to the wire's data field, removals become tombstones,
  metadata changes become UI-metadata ops, graph-level metadata changes
  address nil entity_id — each stamped with the runtime's client id and
  an incrementing logical clock
- Node renames surface as unsupported events: they project at the
  changeset→protocol boundary (work document #4 §5), not in the runtime

The runtime core's broadcast now takes an exclusion context so
convergence broadcasts can skip the originating link.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 4 of work document #28: the 0x30/0x32 streaming telemetry surface,
fed by a real NoFlo network.

NetworkHost owns the network lifecycle: builds the NoFlo network from the
runtime's graph with an injected component loader (platform-neutral, per
work documents #6/#16), re-emits the network's EventTarget events so
protocol handlers observe a network before, after, and across network
recreations, and supports rebuild-on-discard. The engine gap it surfaces
is documented: the 2.x engine consumes the graph at connect() and does
not follow live graph changes, so CRDT operations applied while a network
runs update the wire state but not the running network until the engine
learns to follow graph deltas.

TelemetryProtocol maps host events onto the 0x32 vocabulary:
- data packets become DATA events with raw payloads; brackets become
  BEGIN_GROUP/END_GROUP with the group name
- network start/stop become LIFECYCLE START/STOP codes
- process errors become ERROR events with the exception string — except
  errors in the stub error class (default StubNotImplementedError), which
  become STUB_ERROR: the event type itself is the classification (#4
  update #9)
- payloads the wire cannot carry are frugalized to type-named
  placeholders so serialization can never take down the flush loop

Flush policy is the runtime's physical decision: per-subscription buffers
flush on timers at the effective cadence (the client's requested
interval, bounded below by a configurable floor), with explicit flushing
available for transports that prefer it. The protocol has no unsubscribe
command; subscriptions live with their link, and dropContext tears them
down on link close.

The integration test streams a real network run — a real ComponentLoader,
asComponent factory, graph with IIP — through the full pipeline to codec
round-trips. @noflo/noflo re-exports the Network class, restoring 1.x
surface parity that the host needed to type against.
bergie added a commit that referenced this pull request Oct 9, 2026
Runtime verification (work document #28) surfaced a spec gap: the
LIFECYCLE_CTRL capability names lifecycle control, but the wire had no
command that exercises it — a client could pause a network it could never
start. 0x40 CMD_RUN_CTRL gains START = 0x04 and STOP = 0x05: start builds
and starts the network, stop stops the running network. Additive enum, no
protocol version bump (update #7).

A rejected start — component startup failure, graph error — surfaces as a
0x04 FAILED lifecycle event with the detail in an accompanying 0x04 ERROR
event, the honest signal work document #4 update #1 pinned. SPEC §8
documents the actions; the AsyncAPI enum grows the values.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 5 of work document #28: the 0x40 block server side, honest about
what the engine can do.

Implemented for real:
- START/STOP drive the network lifecycle through the host — the surface
  the wire was missing until work document #4 update #20; a rejected
  start surfaces as a 0x04 FAILED lifecycle event with the detail in an
  accompanying 0x04 ERROR event (update #1), finally exercised by a real
  command path
- HWM_SET records the network-global backpressure default on the host,
  applied when the network is next built; the engine resolves the
  high-water mark at edge construction, so a change while a network runs
  is deferred with an ERROR flowtrace event saying so

Honest about engine gaps (no faked semantics): pause/resume/step, data
breakpoints, and per-process disable have no engine primitive — stop()
tears down sockets and loses queued packets, violating pause semantics.
Those commands surface as unsupported events for the operator and as
ERROR flowtrace events for subscribed clients, never as silence or false
acknowledgments. Recording them as engine work items is the runtime's
verification job.

TelemetryProtocol.record is now public so sibling handlers can feed the
stream; NetworkHost carries pending high-water-mark configuration applied
at build time.
bergie added a commit that referenced this pull request Oct 9, 2026
Phase 6 of work document #28 — the mesh side of the runtime:

ReticulumBinding binds a RuntimeServer to an RNS instance the application
brings (isomorphic: interfaces are the app's choice — TCP on servers,
WebSocket/WebRTC in browsers). It owns the RNS bootstrap:

- a single-purpose IN destination announcing under the stable
  nofo.runtime aspect, its app_data the positional
  msgpack([protocol_version, destination_hash_raw16, node_name]) of
  work document #4 §3, so scanners learn compatibility before spending
  link-establishment airtime and Proof-of-Work
- incoming LINKREQUESTs answered via respondToLinkRequest; on each
  established link, the peer's verified identity (the responder side of
  link.identify()) triggers the unilateral 0x02 auth response, and every
  decrypted DATA packet feeds the frame router with the link as context
- link closure emits linkclosed so context-bound state (telemetry
  subscriptions) drops with the link
- epoch baselines for 0x12 stale-epoch replies are served through the
  link REQUEST/RESPONSE API: the resource token is the truncated SHA-256
  of the canonical graph bytes — content-addressed like the epoch id —
  registered as the request path, so a client fetches with
  link.request(token) using the token the wire carried

The one-call assembleRuntime wires the full stack — server, network host,
registry, graph sync, telemetry, execution control — from the
application's graph, catalog, and optional loader, deriving the registry
state before the first sync; bindReticulum attaches the mesh side and
connects the baseline provider. Per-link DACAR capability narrowing
remains the application's concern; the binding advertises the server's
mask.

Verified against fake RNS plumbing at the binding's seams (destination
factory, respondToLinkRequest, links): announce encoding, the
handshake→auth→frames lifecycle, baseline serving, and the full-stack
assembly answering real protocol frames. A true two-instance RNS
integration test needs an interface with a server side and stays future
work.
bergie added a commit that referenced this pull request Oct 9, 2026
The true two-instance verification (work document #28 phase 6): a real
rnsd relay, one process side running the assembled runtime, the other a
hand-rolled protocol client. Opt-in via RUNTIME_E2E_RNSD=host:port —
skipped otherwise, so CI stays network-free.

The client exercises the whole wire story: announce discovery decodes
through the §3 app_data codec, Destination.recalled hydrates the peer
identity from the announce, the link handshake completes, the client
identifies itself and receives the unilateral 0x02 auth response, a 0x20
sync returns the 0x22 manifest, a stale 0x10 returns 0x12 with the
content-addressed resource token, and link.request(token) fetches the
canonical graph baseline — every frame link-encrypted.

What getting it green surfaced, all real transport bugs:

- the binding never registered the destination with the transport, so
  inbound LINKREQUESTs were dropped as not-addressed-to-us —
  registerDestination is now called (fakes without an RNS instance are
  tolerated)
- the core's send/broadcast were transport-agnostic no-ops: the binding
  now installs link-aware delivery at start(), tracking live links for
  broadcast-with-exclusion and wrapping link.send in error-capturing
  transmit
- leaf fallback requires a default interface: the test marks the rnsd
  uplink with addInterface(iface, true), matching y-reticulum's practice
- interfaces must be explicitly connect()ed — addInterface only binds
AsherBond pushed a commit to AsherBond/noflo that referenced this pull request Oct 14, 2026
The graph protocol handler is plane-aware: 0x10/0x11/0x12/0x14 carry the
plane_id (nil = the running main graph); a sync or op addressed to a
plane this runtime is not hosting is answered with 0x17 CMD_OP_REJECTED
— the explicit no, never silence — until the multi-plane registry lands
(the plane model is specified in work document noflo#4 updates noflo#28-noflo#33; the
registry is the staged next step of this wave).

RuntimeServer broadcasts runtime-initiated ops with the main plane id,
and the 0x02 auth response carries the runtime metadata and the
advertised capability ceiling alongside the granted mask, per the
external review's runtime:runtime and allCapabilities/capabilities
points.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0