Repository navigation
Hosting: SDK crates on crates.io, the application as an image, the service on Fly.io (0.1.1) - #13
Merged
Merged
Conversation
…rvice on Fly.io 0.1.0 went to crates.io as the three SDK crates only (evalhub-schema, evalhub-core, evalhub-query): `cargo publish --workspace` stopped on the server's gitignored `web-dist/`, which cargo's dirty check reports on purpose when an `include` directive packages ignored files (rust-lang/cargo#12294), and the only way past it is `--allow-dirty`. A server that needs a Postgres and an S3 store gains nothing from `cargo install` anyway, so the split is made explicit: the store and the server are `publish = false`, `just package` packages the three SDK crates with no web build and no `--allow-dirty`, and the UI-in-the-binary check stays where it already was, build.rs refusing a release build without `web-dist/`. `cargo package --workspace` could not stay: with the store unpublishable it resolves the server's dependencies against crates.io. The application ships as the container image. Dockerfile builds it in three stages (UI, release binary, slim Debian, non-root, binds 0.0.0.0:8080); the release workflow pushes ghcr.io/ynishi/evalhub on every v* tag. The hosted service runs on Fly.io from fly.toml: built from the Dockerfile, `evalhub migrate` as the release command, healthz as the liveness check, one shared-cpu-1x Machine, Postgres attached and Tigris for attachments, every secret a Fly secret. docs/hosting.md is the runbook. Fly over Cloud Run + Cloud SQL because the smallest Cloud SQL costs more per month than this whole setup, and the question this milestone answers is whether the service is worth running at all. compose.yml is local development only and says so: the hub needs a Postgres and an S3-compatible store to start, and the file exists to close that gap, not to be a deployment. Verified: `just package` (three crates, package-gate ok; run with --allow-dirty once on the uncommitted tree, README being modified); `cargo clippy --workspace --all-targets -D warnings`; `cargo fmt --check`; compose.yml, ci.yml, release.yml and fly.toml parse. `fly deploy` from this tree: image built, the `migrate` release command completed, the Machines came up, `GET /api/v1/healthz` answers 200 at the app URL. Store and server tests not run: no Rust code changed. Refs #12 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
The first stand-up was driven from a script in an ignored working area, and it echoed the output of `fly storage create`, which carries the bucket's access and secret keys, straight to the terminal; the first user was then created by hand over `fly ssh console`, which put the token there too. Three flyctl commands print credentials by design (`postgres create` the superuser password, `postgres attach` the connection URL, `storage create` the keys), and so does `evalhub user create`: a script that shows their output shows secrets. That deployment was torn down and rebuilt from scripts that live in the repository, `deploy/fly/`: - `up.sh` captures each of those commands, hands the values to `fly secrets import` on stdin (never as arguments), and prints only redacted lines. Fields are taken as the whole value after the first colon, since the endpoint URL contains one. The auth keys are generated in the same pipe. The superuser password is deliberately not kept; `fly postgres connect` needs none. A destroyed Tigris bucket name stays unavailable for a while, so the plain name is retried once with a random suffix. `fly deploy --ha=false`: one Machine, not the two Fly adds by default. - `down.sh` destroys the bucket (either name form), the app and the Postgres, in that order, so the bucket's keys die with it. There is no "rebuild": a service has up, down and deploy, and a leaked token is revoked, not wiped with the database. - `user.sh` runs `evalhub user create` on the Machine and writes the token to ~/.config/evalhub/token (mode 600), showing login and scope only. - `smoke.sh` is the acceptance test from outside: three attachments through presigned URLs, an Eval and a Card from the schema fixtures, read back with fingerprints, comparison view, relations and export, then made public and fetched without a token, UI deep link included. The token comes from that file (or EVALHUB_TOKEN, or a silent prompt when stdin is a terminal) and reaches curl through a header file. docs/hosting.md points at the scripts and explains what each step is for instead of listing commands to type; AGENTS.md says the scripts must stay credential-silent. Dockerfile: `useradd` without `--system`, which warned that uid 10001 is above SYS_UID_MAX. Verified: `bash -n` on every script; the redaction filter and the field extraction on synthetic flyctl output; flyctl help for `storage list`, `secrets import --stage`, `deploy --ha`; the image rebuilt locally runs as uid 10001; down.sh, up.sh, user.sh and smoke.sh run end to end against Fly (see the next commit). Refs #12 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
smoke.sh: two checks had their result line swallowed by a trailing `>/dev/null` (the redirect applied to `check` itself), and the 201-or-200 checks ran `||` outside `check`, so a 200 on a repeat run would have counted as a failure. `check` now drops the command output itself and the alternatives are one command. Header points at user.sh. Version 0.1.1 across the workspace and the internal dependency versions, with Cargo.lock refreshed (`cargo update -w`). No Rust code changed since 0.1.0; the release is the hosting: the SDK/application split, the image, fly.toml, the deploy scripts and the runbook. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-schema / -core / -query; smoke.sh twice against https://evalhub.fly.dev after down.sh + up.sh + user.sh (18/18 checks, the second run exercising the idempotent 200 paths). Store and server tests not run: no Rust code changed. Refs #12 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This was referenced Sep 22, 2026
Closed
ynishi
added a commit
that referenced
this pull request
Sep 22, 2026
…w (0.1.2) A release was three remote writes typed by hand after the merge (tag push, `cargo publish` for the SDK crates, and the image through the old release workflow), driven for 0.1.1 by a script pair written during the release and by a long-lived `cargo login` token on one machine. Now the tag push is the only thing a person does. `release.yml` on a `v*` tag: `check` refuses a tag whose version is not the manifest's or whose commit is not on main; `crates` runs `just package`, cargo-semver-checks against the published versions, then publishes whichever of evalhub-schema / -core / -query is not on crates.io at that version, authenticated by crates.io Trusted Publishing (OIDC from this repository and workflow file; no stored token; the first version of each crate had to be published by hand, which 0.1.0 was); `image` is unchanged; `release` creates the GitHub Release with the merged pull request's body as notes. A rerun publishes nothing twice and leaves an existing release alone. `deploy/release/land.sh <pr> <issue>` merges with a merge commit (never squash: the commits carry the Signed-off-by), closes the issue, fast-forwards main, and hands to `deploy/release/tag.sh`, which waits for main's CI, tags once, pushes, and follows the workflow to the end. `just semver-check` is the same API check by hand. release-plz as in dsl-kit was considered and not taken: its version and changelog come from Conventional Commit headers this repository does not use, and its release PR carries a bot commit without a sign-off. Version 0.1.2, so that this change is released by the workflow it adds: the tag will point at a commit that contains it. No Rust code changed. Verified: bash -n on both scripts; release.yml parses; the PR lookup by merge sha returns #13 for the 0.1.1 merge commit; `gh run watch --exit-status` exists; cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test -p evalhub-schema / -core / -query, `just package` at 0.1.2, `just semver-check` against 0.1.1. Trusted Publishing is enabled on crates.io for the three crates. Refs #14 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits: what ships where, the hosted service on Fly.io, and the version.
Hosting (
4470b3d)0.1.0 was published as the three SDK crates only (
evalhub-schema,evalhub-core,evalhub-query).cargo publish --workspacestopped on the server's gitignoredweb-dist/, which cargo's dirty check reports by design when anincludedirective packages ignored files (rust-lang/cargo#12294); the only way past is--allow-dirty, and a server that needs a Postgres and an S3 store gains nothing fromcargo install. So the split is explicit:evalhub-storeandevalhub-serverarepublish = false,just packagepackages the SDK crates with no web build and no--allow-dirty, and the UI-in-the-binary check stays inbuild.rs, which refuses a release build withoutweb-dist/.The application ships as the container image (
Dockerfile, three stages, non-root,0.0.0.0:8080); thereleaseworkflow pushesghcr.io/ynishi/evalhubon everyv*tag. The hosted service runs on Fly.io fromfly.toml: built from the Dockerfile,evalhub migrateas the release command, healthz as the liveness check, one Machine, Postgres attached, Tigris for attachments, every secret a Fly secret.docs/hosting.mdis the runbook.compose.ymlis local development only and says so.Fly scripts (
05b3c98)deploy/fly/:up.sh,down.sh,user.sh,smoke.sh. Three flyctl commands print credentials by design (postgres create,postgres attach,storage create), and so doesevalhub user create; the scripts capture each, pass values tofly secrets importon stdin or write the token to~/.config/evalhub/token(mode 600), and show only redacted lines. The app name is the one line infly.toml; the Postgres, the bucket and the URL derive from it.smoke.shis the acceptance test from outside: attachments through presigned URLs, an Eval and a Card from the schema fixtures, read back with fingerprints, comparison view, relations and export, made public and fetched without a token, UI deep link included.0.1.1 (
10d1a7f)Version bump only; no Rust code changed since 0.1.0.
Verified
just package(three crates,--locked, no--allow-dirty),cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test -p evalhub-schema / -core / -query.down.sh→up.sh→user.sh alice→smoke.shagainst Fly: 18/18 checks, twice (the second run exercising the idempotent 200 paths). Nothing secret on the terminal at any step.Not in this PR
crates.io publish of 0.1.1 and the
v0.1.1tag (hand-typed after merge). A custom domain. Managed Postgres. Terms / content policy / abuse runbooks.Refs #12