Repository navigation
fix(minvmd,switch): the per-box BEP connection cap has a default that one Pulumi preview does not - #3396
Conversation
One Pulumi preview opens dozens of parallel CONNECTs from one box, and the working value of 16 reset them.
minvmd config set --bep-per-box-cap saves the cap to config.toml, and the boot wiring resolves it from MINVMD_BEP_PER_BOX_CAP, then config.toml, then the switch default. minvmd config show prints the effective cap and its source.
The cap is 128, and minvmd config set --bep-per-box-cap overrides it.
Another test thread can cache a never interest for the issue and withdraw callsites before this thread's subscriber counts. Rebuild the cache before each logged call.
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
Spec lint
Advisory. None of this blocks the merge; the review decides. Advisory, and never a merge gate. Linter: |
|
/revise CI No other PR fails this way, and main is green. This branch changes how the installed minvmd starts and reads its config ( |
|
/revise CI This repeats the When that failure was seen, no other PR failed this way and main was green. This branch changes how the installed minvmd starts and reads its config ( |
|
Round 3 of 10 on a /revise note: the coder changed nothing. Revise round 3: no change, on purpose. I did not find a cause on this branch for the test-kvm failure "'min ls --raw' does not list the stock-installed VM session". I also could not reproduce it: this box has no /dev/kvm, so the KVM lane cannot run here. What I checked: (1) The boot wiring in cmd/run.rs keeps the same order. The only change is the cap value it passes to with_per_source_cap. effective_bep_per_box_cap is a pure read of config.toml (ResourceConfig::read, a missing file yields the default) plus an env var. It writes nothing and takes no lock. (2) Nothing in the diff branches on an installed versus a lane-built minvmd. Both resolve the same default (128) when no config.toml and no MINVMD_BEP_PER_BOX_CAP exist. So the lane-pair groups in the same job booted with the same cap and the same code. (3) The cap only sizes the BEP listener pool: 128 slots, two in-memory smoltcp sockets each, per registered box. The pool runs on the BEP peer's own thread. The SocketSet …[truncated] |
|
/revise merge Round 3 found no cause on this branch for the 2026-11-13 Do:
|
|
Round 4 of 10 on a /revise note: the coder changed nothing. every commit this round put on the branch above 183e300 is a merge of the base branch, so it carries no work of its own |
There was a problem hiding this comment.
Read at 602c634c against D3180. DEFAULT_PER_SOURCE_CAP is 128, inside the plan's 64–128. The cap resolves env (MINVMD_BEP_PER_BOX_CAP), then config.toml, then the default (resolve_bep_per_box_cap, pure and tested per layer). A zero is refused by config set and read as unset at resolve. config show prints the cap and its source, and the boot wiring logs both and passes the cap to with_per_source_cap. Spec NET-132 records the tuned value and the knob. Cost: each slot holds two smoltcp sockets with 4 KiB rx and 4 KiB tx (POOL_BUFFER_LEN), so 128 slots is about 2 MiB per registered box, up from 256 KiB at 16. That is acceptable. Outside layers:: bep_attach.rs gets two rebuild_interest_cache() calls in one test, a test-only fix for a log-capture race. Required CI is green on today's main, including the test-kvm lane that failed on 2026-11-13.
|
Approved: every review thread resolved at @norrietaylor resolved the last thread, which releases the approval this review had withheld. Nothing to do. |
…x-bep-connection-cap-has-a-default-tha
Each slot of a box's share holds listening sockets and buffers, so an unbounded value can exhaust memory when a box registers. config set refuses a value above the maximum, and the boot resolver clamps an env or config.toml value to it with a warning in the log.
|
Round 5 of 10 on the runtime's review findings, pushed as
Verify Outside the task's
Converted to a draft: something above needs a person before review. Review threads the coder judged:
After minvmd config set --bep-per-box-cap 128 and a reboot, each box can hold 128 connections to the proxy address, so 60 parallel CONNECTs from one box all get through. minvmd config show prints the cap and its source (env, config or default). This round, minvmd config set --bep-per-box-cap refuses a value above 1024 with the message "--bep-per-box-cap must be at most 1024". An env or config.toml value above 1024 is clamped to 1024 at boot. Notes from the agent: Revise round 5: added MAX_BEP_PER_BOX_CAP = 1024 in cmd/mod.rs. validate_bep_per_box_cap refuses a larger value. resolve_bep_per_box_cap clamps env and config.toml values to it, and effective_bep_per_box_cap logs a warning when it clamps. The --bep-per-box-cap help text now says "(1 to 1024)". I did not run the 60-CONNECT proof on a booted VM on this box. The unit tests cover the resolver and the validation only. The task and spec give no maximum, so 1024 is my choice: it is 8 times the 128 default. |
There was a problem hiding this comment.
arch-review: PASS · the full ruling
The per-box Box Egress Proxy cap now has an upper bound of 1024 and a lower bound of 1. minvmd config set refuses a value outside it, and the boot resolver clamps an env or config.toml value to it. Section 9 line 333 requires per-box connection caps on the BEP plane, and this head meets that requirement. Only the host can set the cap, so the guest cannot reach it (section 7.1 line 245). The change conforms. The NET-132 note does not yet record the 1024 maximum. Add it in a later spec edit.
Ruled at c6a067e against gominimal/arch@327e1b57d16b by claude-opus-5-5.
There was a problem hiding this comment.
Re-read at c6a067ed. Round 5 bounds the knob: MAX_BEP_PER_BOX_CAP = 1024. config set refuses a value above it, and resolve_bep_per_box_cap clamps an env or config.toml value to it, with a warning at boot that names the source. config show reads the same resolver. Tests cover the refusal and the clamp from each layer. The rest is the main merge. bep_attach.rs (test-only, outside layers:) is accepted at this head.
There was a problem hiding this comment.
minimal-review — approved
Verdict APPROVE — every review thread is resolved; @norrietaylor closed the last one. The review of c6a067edf568 withheld this approval on the threads open then. Posted as a comment: the App cannot approve its own pull request; the check and the label carry it.
Defect
Fixes #3385
crates/switch/src/bep_host.rscrates/minvmd/src/config.rscrates/minvmd/src/cmd/mod.rscrates/minvmd/src/cmd/config.rscrates/minvmd/src/main.rscrates/minvmd/src/cmd/run.rscrates/minvmd/src/net/bep_stub.rsdocs/specs/18-spec-box-networking/18-spec-box-networking.mdTask marker
Proof
The default per-box cap on Box Egress Proxy connections is now 128, not 16. So 60 parallel CONNECTs from one box get through with no configuration. An operator can run
minvmd config set --bep-per-box-cap 128. That command saves bep_per_box_cap to config.toml, refuses 0, and printssaved: vcpus=..., ram_mib=..., bep_per_box_cap=128 (takes effect on next boot). At the next boot the BEP wire takes the cap in this order: MINVMD_BEP_PER_BOX_CAP, then config.toml, then the default. The boot logs the cap and its source.minvmd config showprintsbep_per_box_cap = 128 (config), and--jsongives bep_per_box_cap and bep_per_box_cap_source. I did not do the live 60-CONNECT run, because this box has no libkrun or VM.Notes from the agent: 1) The
cargo test -p minvmd bepfailure is not from this change. bep_attach::tests::one_line_per_attachment_issued_and_withdrawn captures logs with a thread-local tracing subscriber (set_default), and sibling tests on other threads hit the same callsites. On the tree without my minvmd changes,cargo test -p minvmd bep_attachfailed 4 times in 8 runs, and that filter runs none of my tests. Before my change, thebepfilter ran 9 tests and happened to pass. It now runs 14, because the new tests containbepin their names as the plan requires. That changes the timing, and the race now loses every time on this 4-core box. The test passes alone and with --test-threads=1 (14 passed). The fix belongs in crates/minvmd/src/bep_attach.rs, which is outside this task's layers. Suggested fix: a process-global capture subscriber, or serialise the log tests. 2) I chose 128, the top of the 64 to 128 range. That gives headroom over the 60 connections in the proof. The cost is about 128 pool slots for each registered box. No switch test assumed 16: the 16s in the refusal-quota test count refusals, not the cap. So only the constant and its doc comment changed in bep_host.rs. 3) The boot wiring in cmd/run.rs is under #[cfg(minvmd_l…[truncated]Tests
cargo test -p switch connections_past_the_per_source_cap_are_resetcargo test -p minvmd effective_bep_per_box_capcargo test -p minvmd round_trips_through_tomlcargo test -p minvmd bepVerify
min task run agent-verifypassed ([tasks.verify]).Out of scope
These files are outside the task's
layers:, so this PR is a draft:crates/minvmd/src/bep_attach.rs