8000 Hosting: SDK crates on crates.io, the application as an image, the service on Fly.io (0.1.1) by ynishi · Pull Request #13 · ynishi/evalhub · GitHub
[go: up one dir, main page]

Skip to content

Hosting: SDK crates on crates.io, the application as an image, the service on Fly.io (0.1.1) - #13

Merged
ynishi merged 3 commits into
mainfrom
feat/hosting-fly
Sep 22, 2026
Merged

ynishi merged 3 commits into
mainfrom
feat/hosting-fly

Conversation

@ynishi
@ynishi ynishi commented Sep 22, 2026
Copy link
Copy Markdown
Owner

Three commits: what ships where, the hosted service on Fly.io, and the version.

Hosting (4470b3d)

0.1.0 was published as the three SDK crates only (evalhub-schema, evalhub-core, evalhub-query). cargo publish --workspace stopped on the server's gitignored web-dist/, which cargo's dirty check reports by design when an include directive packages ignored files (rust-lang/cargo#12294); the only way past is --allow-dirty, and a server that needs a Postgres and an S3 store gains nothing from cargo install. So the split is explicit: evalhub-store and evalhub-server are publish = false, just package packages the SDK crates with no web build and no --allow-dirty, and the UI-in-the-binary check stays in build.rs, which refuses a release build without web-dist/.

The application ships as the container image (Dockerfile, three stages, non-root, 0.0.0.0:8080); the release workflow pushes ghcr.io/ynishi/evalhub on every v* tag. The hosted service runs on Fly.io from fly.toml: built from the Dockerfile, evalhub migrate as the release command, healthz as the liveness check, one Machine, Postgres attached, Tigris for attachments, every secret a Fly secret. docs/hosting.md is the runbook. compose.yml is local development only and says so.

Fly scripts (05b3c98)

deploy/fly/: up.sh, down.sh, user.sh, smoke.sh. Three flyctl commands print credentials by design (postgres create, postgres attach, storage create), and so does evalhub user create; the scripts capture each, pass values to fly secrets import on stdin or write the token to ~/.config/evalhub/token (mode 600), and show only redacted lines. The app name is the one line in fly.toml; the Postgres, the bucket and the URL derive from it. smoke.sh is the acceptance test from outside: attachments through presigned URLs, an Eval and a Card from the schema fixtures, read back with fingerprints, comparison view, relations and export, made public and fetched without a token, UI deep link included.

0.1.1 (10d1a7f)

Version bump only; no Rust code changed since 0.1.0.

Verified

  • just package (three crates, --locked, no --allow-dirty), cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo test -p evalhub-schema / -core / -query.
  • down.sh → up.sh → user.sh alice → smoke.sh against Fly: 18/18 checks, twice (the second run exercising the idempotent 200 paths). Nothing secret on the terminal at any step.
  • Store and server tests not run: no Rust code changed.

Not in this PR

crates.io publish of 0.1.1 and the v0.1.1 tag (hand-typed after merge). A custom domain. Managed Postgres. Terms / content policy / abuse runbooks.

Refs #12

…rvice on Fly.io

0.1.0 went to crates.io as the three SDK crates only (evalhub-schema,
evalhub-core, evalhub-query): `cargo publish --workspace` stopped on the
server's gitignored `web-dist/`, which cargo's dirty check reports on
purpose when an `include` directive packages ignored files
(rust-lang/cargo#12294), and the only way past it is `--allow-dirty`. A
server that needs a Postgres and an S3 store gains nothing from `cargo
install` anyway, so the split is made explicit: the store and the server
are `publish = false`, `just package` packages the three SDK crates with
no web build and no `--allow-dirty`, and the UI-in-the-binary check stays
where it already was, build.rs refusing a release build without
`web-dist/`. `cargo package --workspace` could not stay: with the store
unpublishable it resolves the server's dependencies against crates.io.

The application ships as the container image. Dockerfile builds it in
three stages (UI, release binary, slim Debian, non-root, binds
0.0.0.0:8080); the release workflow pushes ghcr.io/ynishi/evalhub on
every v* tag. The hosted service runs on Fly.io from fly.toml: built from
the Dockerfile, `evalhub migrate` as the release command, healthz as the
liveness check, one shared-cpu-1x Machine, Postgres attached and Tigris
for attachments, every secret a Fly secret. docs/hosting.md is the
runbook. Fly over Cloud Run + Cloud SQL because the smallest Cloud SQL
costs more per month than this whole setup, and the question this
milestone answers is whether the service is worth running at all.

compose.yml is local development only and says so: the hub needs a
Postgres and an S3-compatible store to start, and the file exists to
close that gap, not to be a deployment.

Verified: `just package` (three crates, package-gate ok; run with
--allow-dirty once on the uncommitted tree, README being modified);
`cargo clippy --workspace --all-targets -D warnings`; `cargo fmt --check`;
compose.yml, ci.yml, release.yml and fly.toml parse. `fly deploy` from
this tree: image built, the `migrate` release command completed, the
Machines came up, `GET /api/v1/healthz` answers 200 at the app URL.
Store and server tests not run: no Rust code changed.

Refs #12

Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
The first stand-up was driven from a script in an ignored working area,
and it echoed the output of `fly storage create`, which carries the
bucket's access and secret keys, straight to the terminal; the first
user was then created by hand over `fly ssh console`, which put the
token there too. Three flyctl commands print credentials by design
(`postgres create` the superuser password, `postgres attach` the
connection URL, `storage create` the keys), and so does `evalhub user
create`: a script that shows their output shows secrets. That
deployment was torn down and rebuilt from scripts that live in the
repository, `deploy/fly/`:

- `up.sh` captures each of those commands, hands the values to `fly
  secrets import` on stdin (never as arguments), and prints only
  redacted lines. Fields are taken as the whole value after the first
  colon, since the endpoint URL contains one. The auth keys are
  generated in the same pipe. The superuser password is deliberately
  not kept; `fly postgres connect` needs none. A destroyed Tigris
  bucket name stays unavailable for a while, so the plain name is
  retried once with a random suffix. `fly deploy --ha=false`: one
  Machine, not the two Fly adds by default.
- `down.sh` destroys the bucket (either name form), the app and the
  Postgres, in that order, so the bucket's keys die with it. There is
  no "rebuild": a service has up, down and deploy, and a leaked token is
  revoked, not wiped with the database.
- `user.sh` runs `evalhub user create` on the Machine and writes the
  token to ~/.config/evalhub/token (mode 600), showing login and scope
  only.
- `smoke.sh` is the acceptance test from outside: three attachments
  through presigned URLs, an Eval and a Card from the schema fixtures,
  read back with fingerprints, comparison view, relations and export,
  then made public and fetched without a token, UI deep link included.
  The token comes from that file (or EVALHUB_TOKEN, or a silent prompt
  when stdin is a terminal) and reaches curl through a header file.

docs/hosting.md points at the scripts and explains what each step is
for instead of listing commands to type; AGENTS.md says the scripts
must stay credential-silent. Dockerfile: `useradd` without `--system`,
which warned that uid 10001 is above SYS_UID_MAX.

Verified: `bash -n` on every script; the redaction filter and the field
extraction on synthetic flyctl output; flyctl help for `storage list`,
`secrets import --stage`, `deploy --ha`; the image rebuilt locally runs
as uid 10001; down.sh, up.sh, user.sh and smoke.sh run end to end
against Fly (see the next commit).

Refs #12

Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
smoke.sh: two checks had their result line swallowed by a trailing
`>/dev/null` (the redirect applied to `check` itself), and the
201-or-200 checks ran `||` outside `check`, so a 200 on a repeat run
would have counted as a failure. `check` now drops the command output
itself and the alternatives are one command. Header points at user.sh.

Version 0.1.1 across the workspace and the internal dependency
versions, with Cargo.lock refreshed (`cargo update -w`). No Rust code
changed since 0.1.0; the release is the hosting: the SDK/application
split, the image, fly.toml, the deploy scripts and the runbook.

Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D
warnings; cargo test -p evalhub-schema / -core / -query; smoke.sh twice
against https://evalhub.fly.dev after down.sh + up.sh + user.sh (18/18
checks, the second run exercising the idempotent 200 paths). Store and
server tests not run: no Rust code changed.

Refs #12

Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
@ynishi
ynishi merged commit 90a116b into main Sep 22, 2026
4 checks passed
ynishi added a commit that referenced this pull request Sep 22, 2026
…w (0.1.2)

A release was three remote writes typed by hand after the merge (tag
push, `cargo publish` for the SDK crates, and the image through the old
release workflow), driven for 0.1.1 by a script pair written during the
release and by a long-lived `cargo login` token on one machine. Now the
tag push is the only thing a person does.

`release.yml` on a `v*` tag: `check` refuses a tag whose version is not
the manifest's or whose commit is not on main; `crates` runs `just
package`, cargo-semver-checks against the published versions, then
publishes whichever of evalhub-schema / -core / -query is not on
crates.io at that version, authenticated by crates.io Trusted Publishing
(OIDC from this repository and workflow file; no stored token; the first
version of each crate had to be published by hand, which 0.1.0 was);
`image` is unchanged; `release` creates the GitHub Release with the
merged pull request's body as notes. A rerun publishes nothing twice and
leaves an existing release alone.

`deploy/release/land.sh <pr> <issue>` merges with a merge commit (never
squash: the commits carry the Signed-off-by), closes the issue,
fast-forwards main, and hands to `deploy/release/tag.sh`, which waits
for main's CI, tags once, pushes, and follows the workflow to the end.
`just semver-check` is the same API check by hand.

release-plz as in dsl-kit was considered and not taken: its version and
changelog come from Conventional Commit headers this repository does not
use, and its release PR carries a bot commit without a sign-off.

Version 0.1.2, so that this change is released by the workflow it adds:
the tag will point at a commit that contains it. No Rust code changed.

Verified: bash -n on both scripts; release.yml parses; the PR lookup
by merge sha returns #13 for the 0.1.1 merge commit; `gh run watch
--exit-status` exists; cargo fmt --check, cargo clippy --workspace
--all-targets -D warnings, cargo test -p evalhub-schema / -core /
-query, `just package` at 0.1.2, `just semver-check` against 0.1.1.
Trusted Publishing is enabled on crates.io for the three crates.

Refs #14

Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0