Repository navigation
Tags
Tags
v0.5.3 -- the class-scale grading path, proven end to end
grade class now completes on all three shipped labs and reproduces the
in-place mark exactly: cos461 10.00, advnet 6.00, multicast 4.00.
Four defects on that path made a real cohort's marks wrong, and all
four were invisible when grading a lab in place:
* restore dropped a submission's RPKI authorisations and reported
success
* the reference gate refused to run a class on cos461 at all
* the reset quarantined every submission over the kernel's own gre0
* the convergence wait never waited for label distribution, in the
MPLS course
Fourteen further findings close forwarding-plane grading loopholes and
fault-injection verification gaps. See docs/09_status.md, findings 108
through 125.
Twinet v0.4.0 Twenty-nine more grading defects closed since v0.3.0, all of one family: a check that passes without doing the thing it names. The new shapes were evidence the submission can rewrite (an AS path on the way out, an interface name, a scope the reader filtered on), state held open by a timer that has not expired, a decision the kernel overrules, and a probe whose protocol or port is published and can simply be permitted. Fifty-nine end-to-end mutations now fail if a wrong answer scores.
Twinet v0.3.0 Grading that cannot be fooled by a substitute. Since v0.2.0, twenty-five defects of one shape were found and closed: a check that passed without doing the thing it names. Among them the internet exchange, which had never delivered a route to any member because FRR silently discards a route server's transparent updates unless enforce-first-as is off. Also: a network jail for the RCA benchmark, so an evaluated agent cannot read the answer off the internet; targets drawn at run time; and thirty-four end-to-end mutations that fail if a wrong answer scores.
v0.2.0 — both advanced-course exercises, and an RCA benchmark that me…
…asures something
Since v0.1.0 the platform has gained the half of the course support it claimed
to have, and lost several ways of passing a check without doing the work.
Courses
- The advanced-networks VPN lab is a lab a student can be given. Every AS in
it used to be staff-owned and deployed with LDP, VPNv4 and the route targets
already configured; the provider is now a student system with the interior
given and the VPN to be built.
- The multicast exercise exists: the course's own topology, PIM and IGMP
rendered from one declaration, and four marks -- two for configuration that
can be read back and two for a packet that a host actually received while a
third host, which did not join, heard nothing.
- A manifest declares which half of the configuration Twinet hands out, per
domain, and that declaration is now carried out rather than accepted and
ignored.
- A lab's rubric is found rather than assumed to be COS-461's.
Grading
- Gao-Rexford is checked on every route and every session, including peers
reached across an exchange, which were invisible before.
- A customer's routes must reach the rest of the internet, not merely be
accepted.
- The VPN is graded on a two-label path, not on a ping that static routing
would satisfy.
- The 6in4 tunnel is measured on every host pair, in both directions.
- Private grading adapts the reference to a group's chosen peering addresses,
as class grading already did, and refuses to grade on top of a previous
attempt's harness.
RCA evaluation
- An evaluated agent no longer runs as root with the cluster token and the
directory the ground truth lives in. It gets a filtered copy of the lab, an
unprivileged account, and a credential that can look at one lab and change
nothing. An adversarial test tries every route back to the answer.
- An incident where the agent crashed or printed nothing is an error, not a
silent pass; scenarios may declare a healthy control; multi-fault episodes
are scored against the taxonomy's multiple_faults.
Operations
- A repair no longer starts a second copy of every routing daemon on top of
the running ones, and no longer gives up on routers that recovered.
- A validator session that came back configured and disconnected is repaired,
and the watcher that does it is replaced on deployment instead of the file
being rewritten under a process that will never read it.
Measured on a three-node cluster: the 212-device COS-461 lab scores 10.00 of
10.00 on all eight student systems, the advanced VPN lab 6.00 of 6.00, and the
multicast lab 4.00 of 4.00.
PreviousNext