Repository navigation
node, node-lts, next: rebuild undici's llhttp wasm from source (new wasi-libc) #817
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
7b5ad19
node: rebuild undici's llhttp wasm from source against a new wasi-libc
bryan-minimal f08b493
node: allow V8's two type-section-only test modules in the install ce…
bryan-minimal 029f3df
node-lts: rebuild undici's llhttp wasm from source against wasi-libc
bryan-minimal 021df5a
llhttp-wasm-rebuild: one shared copy of the wasm rebuild script
bryan-minimal 7a5121a
Merge main into bryan/node-wasm-root
bryan-minimal 26aa7e3
next: rebuild the edge runtime's llhttp wasm from source
bryan-minimal f2d7660
Merge main into bryan/node-wasm-root
bryan-minimal 83f9831
wasi-libc, node, node-lts, next: address review
bryan-minimal 8e293eb
wasm-bindgen-cli 0.2.100, amaro-wasm: amaro's swc TypeScript stripper…
bryan-minimal 242af2b
llhttp-wasm-rebuild: python as a runtime dep; next: name og's three w…
bryan-minimal e76fb1e
wasm-bindgen-cli: name the test version_check (version shadowed the b…
bryan-minimal f93022f
amaro-wasm: let bindings named apart from build_args fields
bryan-minimal d5975d8
wasm-bindgen-cli: unpack the .crate in build.sh
bryan-minimal 0bc74cf
amaro-wasm: build from the binding directory; stack size as one rustf…
bryan-minimal d964795
node, node-lts: keep amaro, with its wasm rebuilt from swc
bryan-minimal 8424eab
Merge origin/main into bryan/node-wasm-root (node 26.12.1 / node-lts …
bryan-minimal b0ec739
node, node-lts: re-derive the wasm pins for 26.12.1 / 24.22.1
bryan-minimal fdb5524
Merge origin/main into bryan/node-wasm-root (next: keep both the llht…
b
8000
ryan-minimal 87c64eb
amaro-wasm: rustc 1.99.1, matching main's rust
bryan-minimal e9033a4
next: census the final tree (x64 SWC addon embeds wasm); forward llht…
bryan-minimal 0b373db
Merge origin/main into bryan/node-wasm-root (node, node-lts: system C…
bryan-minimal File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| let { Attrs, BuildSpec, Local, Needs, OutputData, Source, Test, .. } = import "minimal.ncl" in | ||
| let base = import "../base/build.ncl" in | ||
| let llvm = import "../llvm/build.ncl" in | ||
| let rust = import "../rust/build.ncl" in | ||
| let toolchain = import "../toolchain/build.ncl" in | ||
| let wasm-bindgen-cli = import "../wasm-bindgen-cli/build.ncl" in | ||
|
|
||
| # The amaro version node and node-lts bundle (both 1.1.11 since node 26.12.1 / 24.22.1), used in | ||
| # the Source URL, upstream_version and the test, so a bump is one edit here (plus node and | ||
| # node-lts's own `amaro`). Its tag vendors the swc tree the TypeScript stripper is built from | ||
| # (deps/swc, with Cargo.lock). build.sh still loops over amaro_versions, should two ever differ. | ||
| let amaro = "1.1.11" in | ||
| # the rust package's version; build.sh fails if rustc is not this version | ||
| let rustc = "1.99.1" in | ||
| { | ||
| name = "amaro-wasm", | ||
| # amaro is node's built-in TypeScript type stripper (`node x.ts`): swc's | ||
| # binding_typescript_wasm, compiled to wasm32-unknown-unknown and run through wasm-bindgen, | ||
| # base64-inlined into amaro's dist/index.js. This rebuilds that module from the vendored swc | ||
| # source, the way amaro's tools/build-wasm.js does in its container (cargo --release with fat LTO | ||
| # and opt-level z, wasm-bindgen --target nodejs, no wasm-opt), so node and node-lts can splice it | ||
| # over the shipped blob. | ||
| # | ||
| # The rust package ships no wasm32 std and no rust-lld: std is built with -Zbuild-std from the | ||
| # rustc source tarball of the same version, and llvm's wasm-ld links. | ||
| build_deps = [ | ||
| { file = "build.sh" } | Local, | ||
| { | ||
| url = "https://github.com/nodejs/amaro/archive/refs/tags/v%{amaro}.tar.gz", | ||
| sha256 = "f65b4e8ae9ab519e8db1824377eb621d7fce66e21ae1020621d8f54b6ed774fa", | ||
| extract = false, | ||
| } | Source, | ||
| { | ||
| # the rust package's own source | ||
| url = "gs://minimal-staging-archives/rustc-%{rustc}-src.tar.xz", | ||
| sha256 = "6451598a99e77f07571960f6b21d577399d3a30f33e80cbd2e3a4aae6535d265", | ||
| extract = false, | ||
| } | Source, | ||
| base, | ||
| llvm, | ||
| rust, | ||
| toolchain, | ||
| wasm-bindgen-cli, | ||
| ], | ||
|
|
||
| # cargo fetches the crates in swc's Cargo.lock and std's library/Cargo.lock (--locked). | ||
| needs = | ||
| { | ||
| dns = {}, | ||
| internet = {}, | ||
| } | Needs, | ||
|
|
||
| cmd = "./build.sh", | ||
| build_args = { | ||
| amaro_versions = amaro, | ||
| rust_version = rustc, | ||
| }, | ||
|
|
||
| outputs = { | ||
| wasm = { glob = "usr/share/amaro-wasm/**" } | OutputData, | ||
| }, | ||
|
|
||
| attrs = | ||
| { | ||
| upstream_version = amaro, | ||
| license_spdx = "MIT AND Apache-2.0", | ||
| source_provenance = { | ||
| category = 'GithubRepo, | ||
| owner = "nodejs", | ||
| repo = "amaro", | ||
| }, | ||
| } | Attrs, | ||
|
|
||
| tests = { | ||
| modules = | ||
| { | ||
| class = 'Standalone, | ||
| test_deps = [base], | ||
| cmds = [ | ||
| ["/bin/bash", "-c", "for v in %{amaro}; do head -c4 /usr/share/amaro-wasm/$v/wasm_bg.wasm | od -An -c | grep -q 'a s m' || exit 1; done"], | ||
| ], | ||
| } | Test, | ||
| }, | ||
| } | BuildSpec |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| #!/bin/sh | ||
| set -ex | ||
|
|
||
| RV="${MINIMAL_ARG_RUST_VERSION:?build.ncl forwards rust_version}" | ||
| VERSIONS="${MINIMAL_ARG_AMARO_VERSIONS:?build.ncl forwards amaro_versions}" | ||
| TOP=$(pwd) | ||
|
|
||
| # std for wasm32 comes from the rustc source of the exact rustc in use. | ||
| case "$(rustc --version)" in "rustc $RV "*) ;; *) echo "r 8000 ustc is not $RV: update rust_version and its sha" >&2; exit 1 ;; esac | ||
| tar -xof "rustc-$RV-src.tar.xz" "rustc-$RV-src/library" | ||
| LIB="$TOP/rustc-$RV-src/library" | ||
| export __CARGO_TESTS_ONLY_SRC_ROOT="$LIB" | ||
| export RUSTC_BOOTSTRAP=1 | ||
|
|
||
| # Host artifacts (build scripts, proc macros) link with gcc; swc's .cargo/config.toml names | ||
| # cross linkers for the linux targets. | ||
| export CC=gcc | ||
| export CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=gcc | ||
| export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=gcc | ||
| export CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_LINKER=wasm-ld | ||
| # amaro's swc scripts/build.sh | ||
| export CARGO_PROFILE_RELEASE_LTO=fat | ||
| export CARGO_PROFILE_RELEASE_OPT_LEVEL=z | ||
|
|
||
| # build <amaro version> <work dir>: the module amaro's tools/build-wasm.js produces, as | ||
| # <work dir>/pkg/wasm_bg.wasm (before patch.mjs inlines it). | ||
| build() { | ||
| v=$1; w=$2 | ||
| rm -rf "$w"; mkdir -p "$w" | ||
| tar -xof "$TOP/v$v.tar.gz" -C "$w" "amaro-$v/deps/swc" | ||
| # From the binding's directory, as wasm-pack runs it, so its .cargo/config.toml (the 2 MiB | ||
| # stack) applies along with the workspace's; the stack size is repeated in case the env value | ||
| # replaces rather than extends it. | ||
| cd "$w/amaro-$v/deps/swc/bindings/binding_typescript_wasm" | ||
| CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUSTFLAGS="-Clink-arg=-zstack-size=2097152 --remap-path-prefix=$w=/builddir --remap-path-prefix=$LIB=/rustc/library --remap-path-prefix=$HOME/.cargo=/cargo" \ | ||
| cargo build --release --locked -Zbuild-std=std,panic_abort --target wasm32-unknown-unknown | ||
| cd "$w/amaro-$v/deps/swc" | ||
| wasm-bindgen --target nodejs --out-name wasm --out-dir pkg --typescript \ | ||
| target/wasm32-unknown-unknown/release/binding_typescript_wasm.wasm | ||
| cd "$TOP" | ||
| } | ||
|
|
||
| for v in $VERSIONS; do | ||
| # Built twice, at different paths: the module must not depend on where it was built. | ||
| build "$v" "$TOP/a-$v" | ||
| build "$v" "$TOP/b-$v" | ||
| A="$TOP/a-$v/amaro-$v/deps/swc/pkg/wasm_bg.wasm" | ||
| B="$TOP/b-$v/amaro-$v/deps/swc/pkg/wasm_bg.wasm" | ||
| cmp "$A" "$B" || { echo "amaro $v: two builds of wasm_bg.wasm differ" >&2; exit 1; } | ||
| D="$OUTPUT_DIR/usr/share/amaro-wasm/$v" | ||
| mkdir -p "$D" | ||
| cp "$A" "$D/wasm_bg.wasm" | ||
| cp "$TOP/a-$v/amaro-$v/deps/swc/pkg/wasm.js" "$D/wasm.js" | ||
| sha256sum "$D/wasm_bg.wasm" | ||
| done | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| let { BuildSpec, Local, OutputData, Test, .. } = import "minimal.ncl" in | ||
| let base = import "../base/build.ncl" in | ||
| let python = import "../python/build.ncl" in | ||
|
|
||
| # Build-time helper for node and node-lts: rebuilds undici's llhttp wasm from its | ||
| # in-tree C and splices it over the shipped base64 blobs. See the script's docstring. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit · The header says the helper is for node and node-lts, but |
||
| { | ||
| name = "llhttp-wasm-rebuild", | ||
| build_deps = [ | ||
| { file = "build.sh" } | Local, | ||
| { file = "rebuild-llhttp-wasm.py" } | Local, | ||
| base, | ||
| python, # build.sh runs the selftest | ||
| ], | ||
| # node, node-lts and next run the installed script with python3 | ||
| runtime_deps = [python], | ||
| cmd = "./build.sh", | ||
| outputs = { | ||
| script = { glob = "usr/share/llhttp-wasm-rebuild/rebuild-llhttp-wasm.py" } | OutputData, | ||
| }, | ||
| tests = { | ||
| selftest = | ||
| { | ||
| class = 'Standalone, | ||
| test_deps = [python], | ||
| cmds = [["python3", "/usr/share/llhttp-wasm-rebuild/rebuild-llhttp-wasm.py", "selftest"]], | ||
| } | Test, | ||
| }, | ||
| } | BuildSpec | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/bin/sh | ||
| set -ex | ||
|
|
||
| python3 rebuild-llhttp-wasm.py selftest | ||
| install -Dm 0644 rebuild-llhttp-wasm.py "$OUTPUT_DIR/usr/share/llhttp-wasm-rebuild/rebuild-llhttp-wasm.py" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit · Nothing reads the copied
wasm.js.nodeandnode-ltssplice onlywasm_bg.wasmand keep the glue in amaro'sdist/index.js. Themodulestest also reads onlywasm_bg.wasm. Remove this copy unless a planned consumer needs it.