htl/justfile at main · ynishi/htl

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

# Development tasks for htl. `just` on its own lists them.

# The three gates are named for a moment rather than for their contents: `pre-commit`,

# `pre-push`, `pre-publish`. Each is a superset of the one before it, because each of those

# moments is harder to take back than the last — a commit is amended, a push is reverted in

# public, a publish is yanked and never replaced — and a name that says *when* is the one

# thing a person can act on without reading the recipe. The parts they are assembled from

# stay runnable on their own for when you already know which answer you want: `public-refs`,

# `fmt`, `check`, `build`, `e2e`, `e2e-scaffold-packaged`, `bench`. `post-publish` is the one

# recipe that runs *after* a moment rather than before it, and it is not part of any gate.

# These wrap what CONTRIBUTING.md already asks for, so that "did I run everything?" has one

# answer instead of four commands to remember in the right order.

_default:

@just --list

# Everything before a commit: the published set is still legible, format, then

# the green CONTRIBUTING defines. `public-refs` goes first because it is the

# cheapest, so a finding shows before the minutes of test and clippy.

pre-commit: public-refs fmt check

# `check` directly rather than a dependency on `pre-commit`, which would also run `fmt`.

# Reformatting the tree at push time either does nothing, or it edits files the commits

# being pushed do not contain — and then what goes out is not what was reviewed, while the

# reformat itself goes out unrecorded. Formatting belongs at the commit above, where it is

# still part of a commit.

# `build` and `e2e` are here and not there because of what the two moments ask. A commit is

# a local checkpoint, and the question then is whether the workspace still holds together,

# which `check` answers on its own. A push hands the branch to CI and to whoever reads it,

# so the questions worth asking are the ones CI would otherwise ask minutes later: this is

# the same work its two jobs do, and a red run there reproduces here. `build` is not

# redundant beside `check` — run after `cargo test` and clippy have both finished green it

# still compiles, because it is the one of the three that emits each target's artefact

# rather than checking it.

# Everything before a push: the green, a full compile, and every end-to-end case.

pre-push: public-refs check build e2e

# The gate goes last, and not only because it is the slowest thing here. It asks git what

# belongs in a tarball, so it is the one recipe that an uncommitted change can stop — a

# change to a file one of the five crates ships, that is, which it names; a dirty justfile

# or doc does not stop it, because neither would have reached a tarball either way. Running

# it after everything cheaper means its minute is spent on a tree that has already answered

# every question answerable without packaging.

# Everything before a publish: `pre-push`, then the release gate on the five tarballs.

pre-publish: pre-push e2e-scaffold-packaged

# Format in place.

fmt:

cargo fmt --all

# It reads the published set and builds nothing, so each gate runs it first.

public-refs:

python3 scripts/check-public-refs

# `cargo test` and not `cargo test --workspace`, and the two are no longer the same thing:

# `e2e` is a workspace member that `default-members` leaves out, so the bare form runs every

# test in the repository except the four that scaffold a Cargo project and build it from

# nothing. Those are minutes, they are what `just e2e` is for, and putting them here would

# put them in front of every commit and in both toolchains of the check job.

# clippy keeps `--workspace --all-targets`, which is what stops `e2e` rotting: it is

# compiled and linted by this recipe and by CI, and only ever run by `e2e` below.

# Green: the whole workspace, tests and lints. What CI runs.

# The third line is the one build nothing in this workspace asks for: `htl-core` with

# `pkg` off. `htl` turns it on by default and the CLI names it, so an unguarded

# `crate::pkg::` compiles here and fails in the first consumer that wants the macros

# and not the package manager (#288). A `check`, not a test: what it guards is that the

# feature-less crate exists, and its graph is a subset of what `cargo test` just built.

check:

cargo test

cargo clippy --workspace --all-targets

cargo check -p htl-core --no-default-features

# Compile everything, including tests and benches, without running any of it.

build:

cargo build --workspace --all-targets

# What is left here is what `cargo test` will not run on its own: a binary crate embedding

# htl, and four Cargo projects scaffolded from nothing and built. Neither is a claim about

# this workspace, and both are minutes. The CLI's own behaviour on files — check, test, the

# run cache, `cache status`, and the two suites meant to fail — moved to

# `crates/htl-cli/tests/sample_project.rs`, and the four scaffolded projects to `e2e/`,

# where a failure names a file, a line and what it saw instead of reporting that a `grep`

# exited 1. `just check` runs the first of those; this runs the second, by package name.

# The embedding example and every target `--target` offers, end to end.

e2e:

#!/usr/bin/env bash

set -euo pipefail

# The embed example through both the include_tl! and the include_bundle! path, and the

# resolver example, which resolves its `.tl` at run time instead of embedding any. Both

# were workspace members that every `--workspace` command compiled; only `embed` was

# ever run.

cargo run -q -p embed

cargo run -q -p embed -- --bundle

# The async walk: two `async fn` calls awaited side by side, asserted under 400 ms by

# the script itself, and a task left behind whose host future the host sees dropped.

cargo run -q -p embed -- --async

# A host asks Teal: three questions from a thread of its own, each within the kind's

# deadline, and the one Teal answers too late denied by the default taken in one place.

cargo run -q -p embed -- --ask

cargo run -q -p resolver

# `embed`'s `bad` feature adds an `include_tl!` of a module that does not type check,

# and its manifest says so — "demonstrates a Teal type error failing the Rust build".

# Nothing had ever run it, so the demonstration was a claim. Spelled `if …; then exit 1`

# rather than `! cargo build`, because bash exempts a `!`-inverted command from `set -e`:

# that is the form that let a gate in this file report nothing however the manifest

# looked, and it would fail here in the direction that looks like success.

if cargo build -q -p embed --features bad 2>/dev/null; then

echo 'embed --features bad compiled: a Teal type error did not fail the Rust build' >&2

exit 1

# Every target `--target` offers, scaffolded outside this repository and built against this

# checkout. That was 87 lines of bash here; it is now four tests in the `e2e` member

# crate, which `default-members` keeps out of `cargo test` and this line asks for by

# name. The crate finds the binary and the target directory itself, and the binary —

# built from this checkout — pins this checkout in every project it writes, so there

# is nothing to pass and nothing to patch; `e2e-scaffold-packaged` below sets the six

# variables that point the same four tests at a tarball CLI and patch its version pin

# at the extracted trees.

cargo test -p e2e

# The release gate: the CLI suite and the same four host projects, run against the five

# `.crate` files `cargo publish` would upload rather than against this checkout. Everything

# the scaffold case of `e2e` cannot see lives in the difference between the two —

# `include` / `exclude` deciding which files reach the tarball, an `include_str!` target

# that is ignored, the manifest normalisation that strips `[workspace]` and rewrites every

# path dependency to its version key — and each of those breaks a release while leaving

# every check that runs on the checkout green. `htl-core` is the crate with an `include`,

# and it is narrow (`src/**/*`, `lua/**/*`, `vendor/**/*`), so a file added anywhere else

# in that crate and read with `include_str!` is in every checkout and in no tarball.

# Two steps ask that, not one. `cargo package` builds each extracted crate to verify it, so

# a missing `include_str!` target fails there, before anything below it runs; what the

# scaffolding after it adds is the same tarballs compiled the way a consumer compiles them,

# with the features a consumer turns on — `htl` with `ffi` for the C ABI host, which the

# default-feature verification above never builds.

# It is its own recipe rather than a case in `e2e` because it needs a clean worktree and

# several minutes, and a case that refuses to run while a commit is being written does not

# belong in the recipe run while one is. `pre-publish` is what runs it, and it has to be the

# last thing asked before the first `cargo publish`: a version on crates.io is yanked and

# superseded, never replaced, so this is the last step whose answer can still change what

# goes out.

# The release gate: the CLI suite and four scaffolded hosts, asked of the five tarballs a publish would upload.

e2e-scaffold-packaged:

#!/usr/bin/env bash

set -euo pipefail

root="$(pwd)"

# The same target directory `e2e` uses for its own scaffolds: mlua and the graph beneath

# it are compiled once, and what is built again is the htl crates, from their new paths.

target="${CARGO_TARGET_DIR:-$root/target}/e2e-scaffold"

# Four crates in one command, because three of them depend on each other at versions

# nobody has published: while it verifies each tarball by building it, cargo overlays

# the packages it is building on the registry, so `htl`'s requirement on `htl-core`

# 0.6.0 resolves to the `htl-core` 0.6.0 being packaged beside it. No `--allow-dirty`:

# what is under test has to be what git has, or it is not the tarball that would be

# uploaded — and a clean worktree is also what puts .cargo_vcs_info.json inside it.

# Cargo scopes that to the files it is about to ship, which is the right scope and

# narrower than it sounds: an uncommitted README or justfile is not refused here,

# because neither is in any of the six tarballs.

cargo package --target-dir "$target" -p htl-core -p htl-macros -p htl -p htl-std -p htl-cli -p htl-mq

ver="$(cargo pkgid -p htl-core | sed 's/.*[#@]//')"

dir="$(mktemp -d)"

trap 'rm -rf "$dir"' EXIT

# Extracted out of the repository, rather than read where cargo leaves its own unpack,

# for two reasons. The tarball is the artefact and the unpack is a by-product of

# verifying it, so untarring is the step that says what a consumer receives. And that

# unpack sits under this workspace root with `[workspace]` stripped from its manifest,

# which is precisely the arrangement cargo refuses to build: it walks up, finds this

# workspace, and reports a package that believes it is not in one.

for crate in htl-core htl-macros htl htl-std htl-cli htl-mq; do

tar -xzf "$target/package/$crate-$ver.crate" -C "$dir"

done

# The binary that writes the scaffolds below, built from the extracted tree, so that

# the CLI under test is the one being shipped and the thirteen templates it reads with

# `include_str!` are proved to be in the tarball rather than only in the checkout. Its

# four library dependencies are patched at their extracted trees for the same reason

# the projects below are: the version they name is not on crates.io while this runs,

# and this is the run that decides whether it should be. --debug because the question

# is what it builds and what it writes, and a debug build shares the graph the projects

# below compile against instead of compiling a second one. cargo reports the htl-macros

# patch as unused here and is right to: the CLI takes `htl` with default features off,

# so the proc macros are not in its graph. It is passed anyway, because the alternative

# to patching a crate that is not published is resolving it, and if this dependency ever

# arrives the failure should not be a resolution error about crates.io. htl-std is among

# the four — the CLI depends on it directly (`register_libraries`, `src/lib.rs`), unlike

# htl-mq, which is not: the CLI does not depend on it, only the window project it writes

# does — which is what `HTL_PATCH_MQ` below is for.

cargo install --locked --debug --path "$dir/htl-cli-$ver" --root "$dir/cli" \

--target-dir "$target" \

--config "patch.crates-io.htl.path='$dir/htl-$ver'" \

--config "patch.crates-io.htl-core.path='$dir/htl-core-$ver'" \

--config "patch.crates-io.htl-macros.path='$dir/htl-macros-$ver'" \

--config "patch.crates-io.htl-std.path='$dir/htl-std-$ver'"

# Two packages, pointed at the CLI just installed and at the five extracted trees

# instead of at this checkout. `e2e` is the same four tests `e2e` runs; `htl-cli` is

# its whole integration suite — 206 tests across 34 files that spawn `htl_bin()`, which

# is `HTL_TEST_BIN` when it is set. Seven variables are the whole of the difference

# between the two gates, as five positional arguments were when this was a shared bash

# recipe — a release gate that checked less than the loop running on every commit would

# be the wrong way round, and one implementation is how that stays true.

# What the second package adds is not the file set. `cargo package` verifies each

# tarball by building it, so an `include_str!` target left out of one fails above,

# before this line runs, and every file these six crates ship is read that way. What

# is left is everything the shipped binary *does* that compiling it does not ask:

# the bytes `htl new` writes for each host profile, what `fmt` writes, which lints

# fire, what the run cache reports. The four scaffold tests see one corner of that —

# they build what `htl new` wrote, so a template can differ from this checkout's copy

# in any way that still compiles and they stay green.

# Both packages in one `cargo test`, in this workspace's own target directory: nothing

# here sets `CARGO_TARGET_DIR`, so `$target` above is for the packaging and the

# scaffolds only. It is not free the first time even so — `check` builds the six

# default members together, which unifies `htl` to five features, and asking for two

# packages resolves it to (dts, pkg), a different unit — so the first run in a given

# target directory compiles that graph. The two sets then coexist: measured at +35 s

# once and +8 s after, with `cargo test` still finding everything it left behind.

# The variables are set on the command rather than exported: `HTL_TEST_BIN` left in a

# shell is a suite reporting on a binary that stopped matching the source.

HTL_TEST_BIN="$dir/cli/bin/htl" \

HTL_E2E_TARGET="$target" \

HTL_PATCH_HTL="$dir/htl-$ver" \

HTL_PATCH_CORE="$dir/htl-core-$ver" \

HTL_PATCH_MACROS="$dir/htl-macros-$ver" \

HTL_PATCH_MQ="$dir/htl-mq-$ver" \

HTL_PATCH_STD="$dir/htl-std-$ver" \

cargo test -p htl-cli -p e2e

# After a publish, and only then: the CLI a user installs, writing the project a user gets,

# built against the crate a user resolves — `cargo install htl-cli`, `htl new`, `cargo test`,

# nothing redirected. Every other gate here runs before a publish and so patches something:

# `e2e` builds against this checkout (a checkout CLI pins the checkout), and

# `e2e-scaffold-packaged` patches the tarball CLI's version at the extracted tarballs. What

# neither can see is whether the crate that reached crates.io builds what the CLI that

# reached crates.io writes — an `include` that dropped a template, a version that resolved

# to something else — and the one moment that has an answer is after both are up.

# This was `e2e-scaffold-unpatched`, run in CI on every push, back when a checkout CLI

# pinned a release by number and the question "does the published htl read what main's

# scaffold writes" had a subject. A CLI writes for the htl it links now, so on a checkout

# the question has none; the CI step is gone and this is what is left of it, for the

# release flow (`release-plz.yml` publishes on the release PR's merge) to be followed by.

# `cargo test`, not `cargo build`: the library's test is the `require("<mod>")` through

# preload that #244 failed while a build stayed green. The version installed is this

# checkout's, which after a release PR lands is the one just published; run it from `main`.

# It needs the network twice — the install, then each project's resolve.

# After a publish: install the published CLI, scaffold each target, test against the published htl.

post-publish:

#!/usr/bin/env bash

set -euo pipefail

root="$(pwd)"

ver="$(cargo pkgid -p htl-cli | sed 's/.*[#@]//')"

target="${CARGO_TARGET_DIR:-$root/target}/post-publish"

dir="$(mktemp -d)"

trap 'rm -rf "$dir"' EXIT

# The user's install, into a root of its own so the `htl` on PATH is untouched. No

# `--debug`: this is the binary a user gets, release profile included.

cargo install htl-cli --version "$ver" --root "$dir/cli"

"$dir/cli/bin/htl" new "$dir/bin-sample" --target bin

"$dir/cli/bin/htl" new "$dir/lib-sample" --target bin --lib

"$dir/cli/bin/htl" new "$dir/cdylib-sample" --target cdylib --lib

"$dir/cli/bin/htl" new "$dir/window-sample" --target window

# The window project's first build reads `types/htl-mq/mq.d.tl`, which `htl check`

# copies out of the htl-mq this CLI's pin resolves — the published one, which is the

# question this recipe asks — so the check runs here before that project's cargo does.

"$dir/cli/bin/htl" check "$dir/window-sample"

# A published CLI pins a version and nothing else: the bin target's htl line carries

# `features = ["async"]` beside it (its host runs on the executor), the window target's

# is the bare version. Two or three numbers: from 0.7.0 the CLI writes its own three,

# and the 0.6.x CLIs on crates.io wrote two.

pin="$(grep -m1 '^htl = ' "$dir/bin-sample/Cargo.toml")"

if [[ ! "$pin" =~ ^htl\ =\ \{\ version\ =\ \"([0-9]+\.[0-9]+(\.[0-9]+)?)\",\ features\ =\ \[\"async\"\]\ \}$ ]]; then

echo "post-publish: htl-cli $ver pins \`${pin#htl = }\` in bin-sample, not \`{ version = \"<version>\", features = [\"async\"] }\` — a published CLI has to" >&2

exit 1

bare="\"${BASH_REMATCH[1]}\""

echo "post-publish: htl-cli $ver pins $bare"

# The window project pins htl and htl-mq as the same bare version.

win="$(grep -m1 '^htl = ' "$dir/window-sample/Cargo.toml")"

if [[ "${win#htl = }" != "$bare" ]]; then

echo "post-publish: window-sample pins htl as \`${win#htl = }\`, not $bare" >&2

exit 1

mq="$(grep -m1 '^htl-mq = ' "$dir/window-sample/Cargo.toml")"

if [[ "${mq#htl-mq = }" != "$bare" ]]; then

echo "post-publish: window-sample pins htl-mq as \`${mq#htl-mq = }\` beside htl $bare" >&2

exit 1

# bin-sample pins htl-std beside htl too — the asynchronous I/O slice of `std.*` its

# `preload` installs — as the same bare version. window-sample has no htl-std: every

# std.fs / std.proc function is async, and nothing in the window target's run is a

# root the executor runs (`scaffold.rs`'s `WINDOW` doc says why), so it does not

# carry the dependency.

std_bin="$(grep -m1 '^htl-std = ' "$dir/bin-sample/Cargo.toml")"

if [[ "${std_bin#htl-std = }" != "$bare" ]]; then

echo "post-publish: bin-sample pins htl-std as \`${std_bin#htl-std = }\` beside htl $bare" >&2

exit 1

for project in bin-sample lib-sample cdylib-sample window-sample; do

manifest="$(<"$dir/$project/Cargo.toml")"

# The `unpatched_pin` invariant from e2e/tests/scaffold_targets.rs, in the shell: a

# released htl to build against, and nothing redirecting it. Two `if`s over bash's

# own string matching rather than `! grep …`, because `set -e` exempts a `!`-inverted

# command — which is how the gate this pair descends from reported nothing however

# the manifest looked, and it would have failed in the direction that looks like

# success.

if [[ "$manifest" != *$'\nhtl = '* ]]; then

echo "$dir/$project/Cargo.toml: names no htl to build against" >&2

exit 1

if [[ "$manifest" == *patch.crates-io* ]]; then

echo "$dir/$project/Cargo.toml: redirects its own pin, so this is not the manifest a user gets" >&2

exit 1

(cd "$dir/$project" && cargo test --target-dir "$target")

done

# Every benchmark: the figures in the README come from these. Ten samples each; a few minutes.

bench:

cargo bench -p htl-core --bench check

cargo bench -p htl-core --bench fix

cargo bench -p htl-cli --bench cached_check

FilesExpand file tree

justfile

Latest commit

History

justfile

File metadata and controls