8000 Packaging: every crate ships complete, with an end-to-end check and CI by ynishi · Pull Request #11 · ynishi/evalhub · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# The checks CONTRIBUTING.md calls green, run on every push and pull request.
#
# Four jobs, so a failure says which half broke and the slow halves overlap:
# rust fmt, clippy, cargo-deny, the per-crate tests, the sqlx offline check
# web svelte-check and prettier on the SPA
# e2e `just e2e`: the release binary against Postgres, curl and Playwright
# package `just package`: every crate packaged, verified and gated
#
# `cargo test --workspace` is not used, here or anywhere; see CONTRIBUTING.md.

name: ci

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
# corepack would otherwise stop to ask before downloading pnpm.
COREPACK_ENABLE_DOWNLOAD_PROMPT: "0"

jobs:
rust:
name: rust
runs-on: ubuntu-latest
services:
# For `cargo sqlx prepare --check` only; the crate tests start their
# own containers through testcontainers.
postgres:
image: postgres:16-alpine
env:
POSTGRES_PASSWORD: pw
POSTGRES_DB: evalhub
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@v2
with:
tool: cargo-deny,sqlx-cli,just

- name: fmt
run: cargo fmt --check
- name: clippy
run: cargo clippy --workspace --all-targets --locked -- -D warnings
- name: cargo deny
run: cargo deny check

- name: test evalhub-schema
run: cargo test -p evalhub-schema --locked
- name: test evalhub-core
run: cargo test -p evalhub-core --locked
- name: test evalhub-query
run: cargo test -p evalhub-query --locked
- name: test evalhub-store
run: cargo test -p evalhub-store --locked
- name: test evalhub-server
run: cargo test -p evalhub-server --locked

- name: sqlx offline data is current
# Only this step may see DATABASE_URL. With it set, the sqlx macros
# go online and check every query against the database, so clippy
# and the tests would fail against the service before it has been
# migrated; without it they use the committed offline data.
env:
DATABASE_URL: postgres://postgres:pw@127.0.0.1:5432/evalhub
run: |
sqlx migrate run --source crates/evalhub-store/migrations
just sqlx-check

web:
name: web
runs-on: ubuntu-latest
defaults:
run:
working-directory: web
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: corepack enable
- run: corepack pnpm install --frozen-lockfile
- name: svelte-check
run: corepack pnpm check
- name: prettier
run: corepack pnpm lint

e2e:
name: e2e
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@v2
with:
tool: just
- uses: actions/setup-node@v4
with:
node-version: 22
- run: corepack enable
- run: corepack pnpm install --frozen-lockfile
working-directory: web
- name: browser and its system libraries
# The runner has root, so the libraries the headless shell links
# against can be installed here; `just e2e-install` then only has
# to confirm they are present.
run: corepack pnpm exec playwright install --with-deps --only-shell chromium
working-directory: web
- name: just e2e
run: just e2e
- name: playwright report
if: failure()
uses: actions/upload-artifact@v4
with:
name: playwright-report
path: |
web/playwright-report
web/test-results
if-no-files-found: ignore
retention-days: 7

package:
name: package
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@v2
with:
tool: just
- uses: actions/setup-node@v4
with:
node-version: 22
- run: corepack enable
- name: just package
run: just package
14 changes: 10 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,17 @@ copy of anything; follow the links.
- **Design**: the crate docs. `cargo doc --no-deps --open`, start at
`evalhub_server`. The design is not in `docs/`; `docs/` holds runbooks only.
- **User-facing reference**: `README.md`.
- **Verification**: per crate, `cargo test -p <crate>`. Never `cargo test
--workspace` as the routine check. `evalhub-store` and `evalhub-server` tests
need Docker.
- **Verification**: per crate, `cargo test -p <crate>` (`just check` runs
the full list). Never `cargo test --workspace` as the routine check.
`evalhub-store` and `evalhub-server` tests need Docker.
- **E2E**: `just e2e` boots the release binary against Docker Postgres,
fetches the embedded UI and the API (`e2e/smoke.sh`), then runs the
Playwright suite in `web/tests/browser` against it.
- **Web UI**: `just web-build` writes `crates/evalhub-server/web-dist`
(gitignored, shipped via the crate's `include`). A release build of the
server fails without it; `just package` checks the `.crate` carries it.
- **Contracts that are generated and committed**: `crates/evalhub-schema/schemas/`,
`.sqlx/`, the OpenAPI snapshot, the `web/` client. Regenerate in the same
`crates/evalhub-store/.sqlx/`, the OpenAPI snapshot, the `web/` client. Regenerate in the same
commit as the change that invalidated them.
- **Do not**: add a design document under `docs/`; add a dependency without
checking `cargo deny check`; commit anything `.gitignore` excludes.
41 changes: 33 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,23 +61,47 @@ cargo test -p evalhub-core
cargo test -p evalhub-query
cargo test -p evalhub-store # needs Docker
cargo test -p evalhub-server # needs Docker
cargo sqlx prepare --check --workspace
(cd crates/evalhub-store && cargo sqlx prepare --check)
cargo deny check
```

Do not run `cargo test --workspace` as the routine check: linking every test
binary in parallel is what exhausts memory on a shared machine. Run the crates
you touched, then the ones that depend on them.
`just check` runs that list, and CI (`.github/workflows/ci.yml`) runs it,
`just e2e` and `just package` on every push and pull request. Do not run `cargo test --workspace` as the
routine check: linking every test binary in parallel is what exhausts memory
on a shared machine. Run the crates you touched, then the ones that depend on
them.

`just e2e` is the one check that runs the binary a user would: it builds the
UI and a release `evalhub`, starts a throwaway Postgres, migrates, serves
from outside the source tree, and fetches the embedded UI, an immutable
asset, a deep link and the API (`e2e/smoke.sh`), then drives the same server
with Playwright (`web/tests/browser`): the bundle boots, routes on the
client, and paints a list the API filled. Run it after a change to
`embed.rs`, `build.rs`, `main.rs` or the web build; it needs Docker. The
browser is fetched by `just e2e-install` into `~/.cache/ms-playwright`
without root; the recipe checks the host has the shared libraries the
headless shell links against and says what to do if not.

Packaging has its own gate. `just package` builds the web UI into
`crates/evalhub-server/web-dist`, runs `cargo package --workspace`, and then
inspects every `.crate`: LICENSE and README present, the server carrying
`web-dist/index.html`, the store carrying `.sqlx/`, none over the crates.io
size limit. The UI is gitignored and reaches the `.crate` only through the
server crate's `include` list, so that inspection is the one thing standing
between a forgotten `just web-build` and a published server with no UI.
`cargo publish` is not a recipe; it is typed by hand after `just package`.

Three things are contracts and have a dedicated check:

- **JSON Schema**: `crates/evalhub-schema/schemas/*.json` is generated from the
Rust types and committed. A change to a record type is not done until
`cargo insta test -p evalhub-schema` is green and the updated snapshot is in
the same commit.
- **SQL**: `.sqlx/` holds the offline query data. A change to a query is not
done until `cargo sqlx prepare --workspace` has been rerun and the result is
in the same commit.
- **SQL**: `crates/evalhub-store/.sqlx/` holds the offline query data. It
lives in the crate, not at the workspace root, so that `cargo package`
ships it and the crate builds without a database. A change to a query is
not done until `cargo sqlx prepare` has been rerun from
`crates/evalhub-store` and the result is in the same commit.
- **OpenAPI**: `GET /openapi.json` is what clients generate from. A change to a
handler signature is checked by the server's OpenAPI snapshot test, and the
`web/` client is regenerated in the same pull request.
Expand Down Expand Up @@ -145,7 +169,8 @@ Signed-off-by: Your Name <you@example.com>
Origin](https://developercertificate.org/). Use your real name; a pull
request with an unsigned commit is not merged.
- Formatting and clippy fixes go in their own commits.
- Generated files that are contracts (`schemas/*.json`, `.sqlx/`, the OpenAPI
- Generated files that are contracts (`schemas/*.json`,
`crates/evalhub-store/.sqlx/`, the OpenAPI
snapshot, the `web/` client) ride in the same commit as the change that
invalidated them.
- Never commit anything `.gitignore` excludes: local working areas, agent
Expand Down
6 changes: 5 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,13 @@ edition = "2024"
# sqlx 0.9 declares rust-version 1.94.0; the workspace cannot claim less.
rust-version = "1.94"
license = "AGPL-3.0-only"
authors = ["Yutaka Nishimura"]
authors = ["Yutaka Nishimura <ytk.nishimura@gmail.com>"]
repository = "https://github.com/ynishi/evalhub"
homepage = "https://github.com/ynishi/evalhub"
# Project-level pointer. Each crate overrides it with its own docs.rs page.
documentation = "https://github.com/ynishi/evalhub#readme"
# Relative to the workspace root; cargo copies it into each package.
readme = "README.md"
description = "A hosting service for named, versioned LLM evaluation results and their materials."
keywords = ["llm", "evaluation", "benchmark", "hub"]
categories = ["web-programming::http-server", "science"]
Expand Down
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,14 +173,17 @@ for everyone else. It is compiled into the binary, so a self-host is still
one file and there is no second origin to configure.

```bash
cd web && pnpm install && pnpm build # fills web/build
just web-build # pnpm, into crates/evalhub-server/web-dist
cargo build --release -p evalhub-server
```

A checkout without `web/build` builds and runs: the asset set is empty and
any browser path gets a placeholder that names `/openapi.json`. A debug
build reads the files from disk at request time, so `pnpm build` shows up on
reload; a release build bakes them in.
A debug build of a checkout without `web-dist/` still builds and runs: the
asset set is empty and any browser path gets a placeholder that names
`/openapi.json`. A debug build reads the files from disk at request time, so
`just web-build` shows up on reload. A release build bakes them in, and
refuses to build without them unless `EVALHUB_ALLOW_MISSING_UI=1` says the
omission is deliberate. `just package` builds the UI, packages every crate
and checks that the server's `.crate` carries it.

The UI does not hold a token in JavaScript. `POST /api/v1/session` takes one
and returns a cookie — private, `HttpOnly`, `SameSite=Strict`, and `Secure`
Expand Down
2 changes: 2 additions & 0 deletions crates/evalhub-core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ license.workspace = true
authors.workspace = true
repository.workspace = true
homepage.workspace = true
documentation = "https://docs.rs/evalhub-core"
readme.workspace = true
description = "evalhub domain rules: canonical form, fingerprints, validation, badges. No I/O."
keywords.workspace = true
categories.workspace = true
Expand Down
Loading
Loading
0