8000 feat: add a DuckDB dialect by colinhacks · Pull Request #142 · colinhacks/reylo · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8000
15 changes: 8 additions & 7 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
#
# Everything the release needs is given at dispatch: the commit is the one dispatched, the version comes from its `package.json` (every package moves at reylo's version, which `scripts/build.ts` enforces), and the release notes are the required `notes` input. A run's inputs are fixed for its whole life, so the notes survive the approval wait and any re-run of a job.
#
# One run, two release jobs. `stage` puts the four versions in npm's staged queue: the three engines, then reylo, which names them as peers at its own version. `finish` is a job on the `release-approval` environment, whose required reviewer is a maintainer: GitHub holds the run there, with no runner, for up to 30 days. The npm approval is the gate; the environment approval only resumes the run, and `finish` checks the registry before it tags or releases anything. `stager`, run by a maintainer with one authenticator code, does both, approving on npm and then resuming the run.
# One run, two release jobs. `stage` puts the five versions in npm's staged queue: the four engines, then reylo, which names them as peers at its own version. `finish` is a job on the `release-approval` environment, whose required reviewer is a maintainer: GitHub holds the run there, with no runner, for up to 30 days. The npm approval is the gate; the environment approval only resumes the run, and `finish` checks the registry before it tags or releases anything. `stager`, run by a maintainer with one authenticator code, does both, approving on npm and then resuming the run.
#
# Only `finish` can create a tag. The "Release tags" ruleset refuses tag creation, moves and deletions to everyone, admins and workflow tokens included, except deploy keys; the one deploy key is the `RELEASE_TAG_KEY` secret on `release-approval`, so a tag exists only after a maintainer approved the run.
#
Expand Down Expand Up @@ -95,7 +95,7 @@ jobs:
# The docs site's toolchain, which builds reylo play's page into the reylo package.
- run: nub install --frozen-lockfile
working-directory: site
# dist/reylo, dist/postgres, dist/sqlite and dist/mysql; the build refuses an engine whose version is not reylo's
# dist/reylo, dist/postgres, dist/sqlite, dist/mysql and dist/duckdb; the build refuses an engine whose version is not reylo's
- run: nub run build

- name: Resolve the version
Expand All @@ -107,7 +107,7 @@ jobs:
env:
VERSION: ${{ steps.version.outputs.version }}
run: |
for dir in dist/postgres dist/sqlite dist/mysql dist/reylo; do
for dir in dist/postgres dist/sqlite dist/mysql dist/duckdb dist/reylo; do
bash .github/scripts/npm-stage-publish.sh "$dir" --provenance --access public
done

Expand All @@ -117,7 +117,7 @@ jobs:
VERSION: ${{ steps.version.outputs.version }}
run: |
served=true
for name in reylo @reylosql%2fpostgres @reylosql%2fsqlite @reylosql%2fmysql; do
for name in reylo @reylosql%2fpostgres @reylosql%2fsqlite @reylosql%2fmysql @reylosql%2fduckdb; do
curl -sf "https://registry.npmjs.org/$name/$VERSION" > /dev/null || served=false
done
echo "served=$served" >> "$GITHUB_OUTPUT"
Expand All @@ -134,16 +134,16 @@ jobs:
{
echo "## Staged on npm, waiting for approval"
echo
for name in @reylosql/postgres @reylosql/sqlite @reylosql/mysql reylo; do echo "- \`$name@$VERSION\`"; done
for name in @reylosql/postgres @reylosql/sqlite @reylosql/mysql @reylosql/duckdb reylo; do echo "- \`$name@$VERSION\`"; done
echo
echo 'No version is installable until a maintainer approves it with 2FA. This run is now held on the `release-approval` environment, with no runner, for up to 30 days. It resumes when that deployment is approved, and `finish` then waits for npm to serve every version before it tags and releases.'
echo
echo '```'
echo "stager # in your own terminal: a, Enter, then ONE authenticator code approves the four versions and resumes this run"
echo "stager # in your own terminal: a, Enter, then ONE authenticator code approves the five versions and resumes this run"
echo "stager reject 'reylo@$VERSION' '@reylosql/*@$VERSION' # drop the staged versions and fail this run"
echo '```'
echo
echo "Without stager: read the stage ids with \`npm stage list reylo\` and \`npm stage list @reylosql/postgres\` (and sqlite, mysql), then approve each with \`npm stage approve <id>\`, engines first; \`pnpm stage approve\` takes them in one interactive picker under a single one-time password, as does the Staged Packages tab on npmjs.com. Then approve the pending \`release-approval\` deployment on $RUN_URL."
echo "Without stager: read the stage ids with \`npm stage list reylo\` and \`npm stage list @reylosql/postgres\` (and sqlite, mysql, duckdb), then approve each with \`npm stage approve <id>\`, engines first; \`pnpm stage approve\` takes them in one interactive picker under a single one-time password, as does the Staged Packages tab on npmjs.com. Then approve the pending \`release-approval\` deployment on $RUN_URL."
echo
echo "## Release notes" 10BC8
echo
Expand Down Expand Up @@ -183,6 +183,7 @@ jobs:
wait_for "@reylosql/postgres@$VERSION" "https://registry.npmjs.org/@reylosql%2fpostgres/$VERSION"
wait_for "@reylosql/sqlite@$VERSION" "https://registry.npmjs.org/@reylosql%2fsqlite/$VERSION"
wait_for "@reylosql/mysql@$VERSION" "https://registry.npmjs.org/@reylosql%2fmysql/$VERSION"
wait_for "@reylosql/duckdb@$VERSION" "https://registry.npmjs.org/@reylosql%2fduckdb/$VERSION"
wait_for "reylo@$VERSION" "https://registry.npmjs.org/reylo/$VERSION"

# the public repository answers `git ls-remote` without a credential, so none is kept on disk
Expand Down
23 changes: 14 additions & 9 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ jobs:
# Every database, or the ones the change reaches.
everything() {
echo engine=true
echo 'tests=-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine -p reylo-engine-native -p reylo-oracle'
echo 'shards=["postgres","sqlite","mysql-1","mysql-2","mysql-slow"]'
echo 'tests=-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine-duckdb -p reylo-engine -p reylo-engine-native -p reylo-oracle'
echo 'shards=["postgres","sqlite","duckdb","mysql-1","mysql-2","mysql-slow"]'
}
if [ "$EVENT" != pull_request ]; then everything >> "$GITHUB_OUTPUT"; exit 0; fi
if ! git rev-parse -q --verify HEAD^2 > /dev/null; then everything >> "$GITHUB_OUTPUT"; exit 0; fi
Expand All @@ -64,18 +64,20 @@ jobs:
everything >> "$GITHUB_OUTPUT"
exit 0
fi
postgres=false mysql=false sqlite=false
postgres=false mysql=false sqlite=false duckdb=false
while read -r file; do
case "$file" in
engine/postgres/*) postgres=true ;;
engine/mysql/* | oracle/cases/mysql_*) mysql=true ;;
engine/sqlite/* | oracle/cases/sqlite_*) sqlite=true ;;
engine/duckdb/* | oracle/cases/duckdb_*) duckdb=true ;;
oracle/cases/*) postgres=true ;;
esac
done <<<"$files"
tests='' shards=''
$postgres && { tests="$tests -p reylo-engine-postgres"; shards="$shards,\"postgres\""; }
$sqlite && { tests="$tests -p reylo-engine-sqlite"; shards="$shards,\"sqlite\""; }
$duckdb && { tests="$tests -p reylo-engine-duckdb"; shards="$shards,\"duckdb\""; }
$mysql && { tests="$tests -p reylo-engine-mysql"; shards="$shards,\"mysql-1\",\"mysql-2\",\"mysql-slow\""; }
if [ -z "$shards" ]; then
{ echo engine=false; echo tests=; echo 'shards=[]'; } >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -186,8 +188,9 @@ jobs:

# The soundness oracle (oracle/): every statement of oracle/cases is analyzed by the engine's sources and
# run on live servers over seeded edge values, and a value that contradicts its inferred type or row count
# fails the case. The cases run in shards side by side: Postgres's, SQLite's, and MySQL's in three, one of
# them mysql_types alone, which takes as long as half the others together. Postgres 18 runs on the
# fails the case. The cases run in shards side by side: Postgres's, SQLite's, DuckDB's, and MySQL's in three, one of
# them mysql_types alone, which takes as long as half the others together. DuckDB's shard builds the oracle
# with DuckDB compiled from its sources (`--features duckdb`), which no other build of the workspace does. Postgres 18 runs on the
# runner rather than in a container, because the json_schema_* cases call five validator extensions built
# into it; its packages are cached, since Ubuntu's mirror has taken half an hour to send them.
oracle:
Expand Down Expand Up @@ -292,14 +295,14 @@ jobs:
env:
SHARD: ${{ matrix.shard }}
run: |
# A case's database is its name's prefix (`mysql_`, `sqlite_`, Postgres's none), as its `dialect` file says.
# A case's database is its name's prefix (`mysql_`, `sqlite_`, `duckdb_`, Postgres's none), as its `dialect` file says.
cases='' n=0
for dir in oracle/cases/*/; do
name=${dir%/}
name=${name##*/}
case "$name" in mysql_*) dialect=mysql ;; sqlite_*) dialect=sqlite ;; *) dialect=postgres ;; esac
case "$name" in mysql_*) dialect=mysql ;; sqlite_*) dialect=sqlite ;; duckdb_*) dialect=duckdb ;; *) dialect=postgres ;; esac
case "$SHARD" in
postgres | sqlite) [ "$dialect" != "$SHARD" ] || cases="$cases ${dir%/}" ;;
postgres | sqlite | duckdb) [ "$dialect" != "$SHARD" ] || cases="$cases ${dir%/}" ;;
mysql-slow) [ "$name" != mysql_types ] || cases="$cases ${dir%/}" ;;
mysql-1 | mysql-2)
if [ "$dialect" = mysql ] && [ "$name" != mysql_types ]; then
Expand All @@ -309,12 +312,14 @@ jobs:
;;
esac
done
features=''
case "$SHARD" in
postgres) servers="--url postgres://postgres@localhost:55432/postgres" ;;
sqlite) servers='' ;;
duckdb) servers='' features='--features duckdb' ;;
*) servers="--mysql-url mysql://root@127.0.0.1:53306" ;;
esac
# shellcheck disable=SC2086
cargo run -q -p reylo-oracle -- $servers --strict-coverage $cases
cargo run -q -p reylo-oracle $features -- $servers --strict-coverage $cases
- if: always()
run: sccache --show-stats
Loading
Loading
0