8000 Hosting: SDK crates on crates.io, the application as an image, the service on Fly.io (0.1.1) by ynishi · Pull Request #13 · ynishi/evalhub · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Only what the Dockerfile reads: the workspace manifests, the crates, and
# the web source. Build output is produced inside the image.
*
!Cargo.toml
!Cargo.lock
!crates
!web
crates/evalhub-server/web-dist
web/node_modules
web/.svelte-kit
web/test-results
web/playwright-report
**/target
6 changes: 1 addition & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# rust fmt, clippy, cargo-deny, the per-crate tests, the sqlx offline check
# web svelte-check and prettier on the SPA
# e2e `just e2e`: the release binary against Postgres, curl and Playwright
# package `just package`: every crate packaged, verified and gated
# package `just package`: the SDK crates packaged, verified and gated
#
# `cargo test --workspace` is not used, here or anywhere; see CONTRIBUTING.md.

Expand Down Expand Up @@ -147,9 +147,5 @@ jobs:
- uses: taiki-e/install-action@v2
with:
tool: just
- uses: actions/setup-node@v4
with:
node-version: 22
- run: corepack enable
- name: just package
run: just package
46 changes: 46 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# On a version tag: build the container image from the tagged commit and
# push it to GitHub Container Registry as ghcr.io/ynishi/evalhub:<version>
# and :latest. That image is how the application is distributed; the
# hosted service itself is deployed with `fly deploy` from the checkout
# (docs/hosting.md). crates.io publishing of the three SDK crates is not
# done here; it is a hand-typed `cargo publish` (see CONTRIBUTING.md), and
# the tag that triggers this workflow is pushed as part of that same
# release script.

name: release

on:
push:
tags: ["v*"]

permissions:
contents: read
packages: write

jobs:
image:
name: container image
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=semver,pattern={{version}}
type=raw,value=latest
- uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
8 changes: 7 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,13 @@ copy of anything; follow the links.
- **Conventions** (issues, branches, verification, commits, DCO): `CONTRIBUTING.md`.
- **Design**: the crate docs. `cargo doc --no-deps --open`, start at
`evalhub_server`. The design is not in `docs/`; `docs/` holds runbooks only.
- **User-facing reference**: `README.md`.
- **User-facing reference**: `README.md`. Local development with Docker:
`compose.yml`. The hosted service: `Dockerfile`, `fly.toml`,
`docs/hosting.md`, and the scripts in `deploy/fly/` (`up.sh` /
`down.sh` / `smoke.sh`), which never print a credential; keep it that
way when editing them. Only the three SDK crates (`evalhub-schema`,
`-core`, `-query`) are published to crates.io; the store and the server
are `publish = false` and ship as the image.
- **Verification**: per crate, `cargo test -p <crate>` (`just check` runs
the full list). Never `cargo test --workspace` as the routine check.
`evalhub-store` and `evalhub-server` tests need Docker.
Expand Down
30 changes: 18 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,20 @@ browser is fetched by `just e2e-install` into `~/.cache/ms-playwright`
without root; the recipe checks the host has the shared libraries the
headless shell links against and says what to do if not.

Packaging has its own gate. `just package` builds the web UI into
`crates/evalhub-server/web-dist`, runs `cargo package --workspace`, and then
inspects every `.crate`: LICENSE and README present, the server carrying
`web-dist/index.html`, the store carrying `.sqlx/`, none over the crates.io
size limit. The UI is gitignored and reaches the `.crate` only through the
server crate's `include` list, so that inspection is the one thing standing
between a forgotten `just web-build` and a published server with no UI.
`cargo publish` is not a recipe; it is typed by hand after `just package`.
Packaging has its own gate, and it covers the SDK only. Three crates are
published to crates.io: `evalhub-schema`, `evalhub-core` and
`evalhub-query`, the ones a client or a converter links against.
`evalhub-store` and `evalhub-server` are the application; they are
`publish = false` and ship as the container image (`Dockerfile`), the way a
web application is not uploaded to npm. `just package` packages the three,
then inspects each `.crate`: LICENSE and README present, none over the
crates.io size limit. `cargo publish` is not a recipe; it is typed by hand
after `just package`, naming the three crates.

The UI reaches the binary through `build.rs`, which refuses a release build
without `crates/evalhub-server/web-dist`; the image build and `just e2e`
are what exercise that, so a server with no UI cannot be built for
release, let alone shipped.

Three things are contracts and have a dedicated check:

Expand Down Expand Up @@ -141,11 +147,11 @@ issue or a chat disagree, the code wins, then the comment.
Beyond doc comments there are two places, and nothing else:

- README.md: the user-facing reference (running the server, configuration
keys, the API in one table, self-hosting). A new flag, endpoint or config key
is not done until it is in there.
keys, the API in one table, where the image and the crates come from). A
new flag, endpoint or config key is not done until it is in there.
- `docs/`: guides and runbooks for things that are done by hand (releasing,
migrating a database, standing up MinIO for local development). Not design,
not architecture, not decision records.
migrating a database, operating the hosted service). Not design, not
architecture, not decision records.

Write a rule once, where the thing it constrains is defined, and link to it
from anywhere else; a second copy is the one nobody updates.
Expand Down
10 changes: 5 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/*"]

[workspace.package]
version = "0.1.0"
version = "0.1.1"
edition = "2024"
# sqlx 0.9 declares rust-version 1.94.0; the workspace cannot claim less.
rust-version = "1.94"
Expand All @@ -21,10 +21,10 @@ categories = ["web-programming::http-server", "science"]

[workspace.dependencies]
# internal
evalhub-schema = { path = "crates/evalhub-schema", version = "0.1.0" }
evalhub-core = { path = "crates/evalhub-core", version = "0.1.0" }
evalhub-query = { path = "crates/evalhub-query", version = "0.1.0" }
evalhub-store = { path = "crates/evalhub-store", version = "0.1.0" }
evalhub-schema = { path = "crates/evalhub-schema", version = "0.1.1" }
evalhub-core = { path = "crates/evalhub-core", version = "0.1.1" }
evalhub-query = { path = "crates/evalhub-query", version = "0.1.1" }
evalhub-store = { path = "crates/evalhub-store", version = "0.1.1" }

# serde
serde = { version = "1", features = ["derive"] }
Expand Down
44 changes: 44 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# The evalhub image: one static-ish binary with the web UI inside it.
#
# Three stages. `web` builds the SvelteKit SPA into the server crate, which
# is where `build.rs` insists on finding it for a release build; `build`
# compiles the release binary; the final stage is a slim Debian with CA
# certificates (the hub talks TLS to Postgres and S3) and nothing else.
#
# docker build -t evalhub .
# docker run --rm -p 8080:8080 -e EVALHUB_DATABASE__URL=postgres://... evalhub
#
# `rust-toolchain.toml` is deliberately not copied in: the image's own
# stable toolchain is used, so the build does not download another.

FROM node:22-bookworm-slim AS web
ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0
WORKDIR /src/web
COPY web/package.json web/pnpm-lock.yaml ./
RUN corepack enable && corepack pnpm install --frozen-lockfile
COPY web/ ./
# svelte.config.js writes to ../crates/evalhub-server/web-dist.
RUN mkdir -p /src/crates/evalhub-server && corepack pnpm build

FROM rust:1-bookworm AS build
WORKDIR /src
COPY Cargo.toml Cargo.lock ./
COPY crates ./crates
COPY --from=web /src/crates/evalhub-server/web-dist ./crates/evalhub-server/web-dist
# No BuildKit cache mounts: the file then builds with the classic builder
# too, and CI caches layers instead.
RUN cargo build --release --locked -p evalhub-server \
&& cp target/release/evalhub /evalhub

FROM debian:bookworm-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --uid 10001 --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin evalhub
COPY --from=build /evalhub /usr/local/bin/evalhub
USER evalhub
# Inside a container the loopback default would be unreachable.
ENV EVALHUB_BIND=0.0.0.0:8080
EXPOSE 8080
ENTRYPOINT ["evalhub"]
CMD ["serve"]
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ cargo run -p evalhub-server -- serve --bind 127.0.0.1:8080
cargo run -p evalhub-server -- config show --origin
```

`compose.yml` starts the same thing with Docker for local development
(Postgres, MinIO and the hub, nothing tuned or secured). The hosted service
is deployed from `Dockerfile` and `fly.toml`; `docs/hosting.md` is its
runbook. The application is distributed as that image, not through
crates.io: the published crates (`evalhub-schema`, `evalhub-core`,
`evalhub-query`) are the SDK a client or a converter links against.

Configuration is layered: defaults, then a TOML file (`--config` or
`EVALHUB_CONFIG`, or `evalhub.toml` in the working directory), then
`EVALHUB_*` environment variables (nested keys joined with `__`, e.g.
Expand Down
90 changes: 90 additions & 0 deletions compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Local development only. The hub needs a Postgres and an S3-compatible
# store to start, which is the one thing that makes it harder to run on a
# laptop than a binary with an embedded database would be; this file
# closes that gap and nothing more. It is not a deployment and is not kept
# deployable: the hosted service is deployed from `fly.toml`
# (docs/hosting.md), and nothing here is tuned, secured or backed up.
#
# docker compose up --build
# docker compose run --rm evalhub user create alice # prints alice's token
#
# The UI is at http://localhost:8080. Presigned attachment URLs point at
# http://localhost:9000, so the browser must reach MinIO on that address.
# Auth keys are left unset on purpose: every UI session ends when the
# container restarts, which does not matter here.

services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: evalhub
POSTGRES_PASSWORD: evalhub
POSTGRES_DB: evalhub
volumes:
- postgres:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U evalhub -d evalhub"]
interval: 5s
timeout: 5s
retries: 12

minio:
image: quay.io/minio/minio
command: server /data --console-address :9001
environment:
MINIO_ROOT_USER: evalhub
MINIO_ROOT_PASSWORD: evalhub-minio
ports:
- "127.0.0.1:9000:9000"
- "127.0.0.1:9001:9001"
volumes:
- minio:/data

# The hub does not create its bucket; this does, once, then exits.
minio-init:
image: quay.io/minio/mc
depends_on:
- minio
entrypoint: >
/bin/sh -c "
until mc alias set local http://minio:9000 evalhub evalhub-minio >/dev/null 2>&1; do sleep 1; done;
mc mb --ignore-existing local/evalhub
"

# Brings the schema current, then exits. `evalhub serve` refuses to start
# with pending migrations, so this runs first.
migrate:
build: .
command: migrate
environment:
EVALHUB_DATABASE__URL: postgres://evalhub:evalhub@postgres/evalhub
depends_on:
postgres:
condition: service_healthy

evalhub:
build: .
ports:
- "127.0.0.1:8080:8080"
environment:
EVALHUB_DATABASE__URL: postgres://evalhub:evalhub@postgres/evalhub
EVALHUB_S3__ENDPOINT: http://minio:9000
EVALHUB_S3__PUBLIC_ENDPOINT: http://localhost:9000
EVALHUB_S3__BUCKET: evalhub
EVALHUB_S3__ACCESS_KEY: evalhub
EVALHUB_S3__SECRET_KEY: evalhub-minio
EVALHUB_S3__PATH_STYLE: "true"
EVALHUB_S3__ALLOW_HTTP: "true"
# Unset, so every session ends when the container restarts (see the
# header). Set both to keep sessions across restarts while developing.
# EVALHUB_AUTH__COOKIE_KEY:
# EVALHUB_AUTH__CURSOR_KEY:
depends_on:
migrate:
condition: service_completed_successfully
minio-init:
condition: service_completed_successfully

volumes:
postgres:
minio:
9 changes: 7 additions & 2 deletions crates/evalhub-server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,15 @@ description = "evalhub HTTP API, auth, OpenAPI, embedded web UI, and the `evalhu
keywords.workspace = true
categories.workspace = true
default-run = "evalhub"
# The application, not the SDK. crates.io carries evalhub-schema,
# evalhub-core and evalhub-query; the server ships as the container image
# (`Dockerfile`), the way a web application is not uploaded to npm.
publish = false
# `include` is exhaustive and switches the gitignore filter off for
# packaging. It is here for one reason: `web-dist/` (the built UI) is
# gitignored, and this is the only way it reaches the `.crate`. Anything
# not listed does not ship, so a new top-level file needs a line here.
# gitignored, and this is the only way it reaches the `.crate` that
# `just package` inspects. Anything not listed does not ship, so a new
# top-level file needs a line here.
include = [
"/src",
"/tests",
Expand Down
4 changes: 4 additions & 0 deletions crates/evalhub-store/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ readme.workspace = true
description = "evalhub persistence: Postgres repositories, object storage, query IR to SQL."
keywords.workspace = true
categories.workspace = true
# Part of the application, not the SDK. crates.io carries evalhub-schema,
# evalhub-core and evalhub-query, the crates a client or a converter links
# against; the store and the server ship as the container image.
publish = false

[lints]
workspace = true
Expand Down
Loading
Loading
0