8000 fix(mapper): read .reylo/ offline whatever the database url, or none by colinhacks · Pull Request #9 · colinhacks/reylo · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions examples/esbuild/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,13 @@ A container or CI build that cannot reach the database builds from `.reylo/`, th
DATABASE_URL=postgres://db.internal:5432/app npx reylo generate # nubx reylo generate here
```

Carry `.reylo/` into the build with the sources (it ignores itself in git, so copy it, or run `reylo generate` in the same CI job), then build with the same `DATABASE_URL`, even though it cannot connect: Reylo keys each analysis by the scheme, user, host, port and database of the URL, not the password.
Carry `.reylo/` into the build with the sources (it ignores itself in git, so copy it, or run `reylo generate` in the same CI job), then build. The build needs no `DATABASE_URL`: one it cannot reach, or none, reads the analyses in `.reylo/`, whichever database they were read from.

```sh
DATABASE_URL=postgres://db.internal:5432/app npm run build
npm run build
```

The type check and the plugin both read `.reylo/`, so both pass without the database, and the dev dependencies (esbuild, TypeScript and `@reylosql/postgres`) must be installed for the build. The image that runs the server needs only `node` and `dist/server.mjs`. A query that `.reylo/` lacks, such as one edited since `reylo generate` ran, fails the type check and the build at that file: `.reylo has no analysis of this statement, so run reylo generate where the database can be reached`. So does a build with another URL, or none.
The type check and the plugin both read `.reylo/`, so both pass without the database, and the dev dependencies (esbuild, TypeScript and `@reylosql/postgres`) must be installed for the build. The image that runs the server needs only `node` and `dist/server.mjs`. A query that `.reylo/` lacks, such as one edited since `reylo generate` ran, fails the type check and the build at that file: with a `DATABASE_URL` it cannot reach, `.reylo has no analysis of this statement, so run reylo generate where the database can be reached`, and with none, `No database URL`.

## In your own project

Expand Down
2 changes: 1 addition & 1 deletion examples/nub/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,5 +45,5 @@ Each run of `start` adds a post; `nub run setup` starts over. The tests roll the
## Notes

- A `NODE_COMPAT=1` in the environment makes nub run as plain Node, which skips `nub.jsonc`: a `.sql` import then fails with `ERR_UNKNOWN_FILE_EXTENSION`. A process started under nub's `node` shim passes it on, so unset it if a `.sql` import stops loading.
- The preload analyzes each `.sql` file as it loads, so `nub src/main.ts` needs `DATABASE_URL`, in `.env` or the environment, as the type check does; when that database is out of reach, it reads the analyses in `.reylo/`.
- The preload analyzes each `.sql` file as it loads, against `DATABASE_URL` in `.env` or the environment, as the type check does; when that database is out of reach, or no URL is set, it reads the analyses in `.reylo/`.
- SQLite infers no union from a `check` constraint, so `role` is a `string` here, where Postgres's enum gives `"admin" | "member"`.
4 changes: 2 additions & 2 deletions site/content/sections/03-inference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@ Every step that loads a `.sql` file (the type check, a plugin, the preload) keep

- The `.reylo/` directory is a cache with its own `.gitignore`, so git skips it.
- When the database answers, an entry made against another schema is analyzed again. An editor or a dev server checks the schema again at most every 5 seconds.
- When the database is out of reach, the entries are used as they are, and a statement that `.reylo/` lacks is an error.
- When the database is out of reach, or no database URL is set, the entries are used as they are, and a statement that `.reylo/` lacks is an error.

A build that can't reach the database, such as a container build, reads the `.reylo/` a type check filled. Set the same database URL in the build, even though it can't connect: the entries are keyed by it.
A build that can't reach the database, such as a container build, reads the `.reylo/` a type check filled, with any database URL or none: where it has no analysis of its own for a statement, it takes the newest one, whichever database it was read from.

</Details>

Expand Down
96 changes: 72 additions & 24 deletions src/analyze.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
// the runtime hook and `reylo generate` — and one cache they share: `.reylo/` beside the project
// file, one entry per statement. The cache is only for speed. An entry records the schema it was
// analyzed against; while the database answers, an entry for another schema is analyzed again,
// and while it does not, the entries are used as they are, so a build that cannot reach the
// database runs on what `reylo generate` or an earlier type check wrote.
import { existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
// and while it does not, or no URL names it, the entries are used as they are, so a build that
// cannot reach the database runs on what `reylo generate` or an earlier type check wrote.
import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { createHash } from "node:crypto";
import { join } from "node:path";
import { anyEngine, engine, reason, AnswerCache, MissingEngine, Unreachable, type Analysis, type Engine, type EngineIo } from "./engine.ts";
Expand All @@ -14,7 +14,14 @@ import { NotMysql, openMysqlDescriber } from "./mysql-describe.ts";
import { openSqliteDescriber, sqlitePath } from "./sqlite-describe.ts";

/** What `.reylo/` holds for one statement. */
export type Entry = { version: 1; dialect: ResolvedProject["dialect"]; schema: string; analysis: Analysis };
export type Entry = {
version: 1;
dialect: ResolvedProject["dialect"];
schema: string;
analysis: Analysis;
/** When it was written, in milliseconds since 1970: a copy of `.reylo/` keeps it where it may not keep the file's time. */
at: number;
};

export type Outcome =
| { ok: true; analysis: Analysis; source: "cache" | "database"; entry: string }
Expand Down Expand Up @@ -95,11 +102,42 @@ export class Session {
return this.#engine.lex(sql, this.project.dialect);
}

/** Where the entry for a statement lives, whether or not it exists yet. */
/**
* Where this session's entry for a statement lives, whether or not it exists yet: named by the
* statement, then by the server. Two databases keep an entry each, since a Postgres fingerprint
* holds object ids, so a dev and a test database built by the same migrations still differ. A
* session that cannot reach its database, or has no URL, reads another server's entry for the
* statement where it has none of its own (`#offlineEntry`).
*/
entryFile(sql: string): string {
const server = this.#server();
const key = createHash("sha256").update(JSON.stringify([ENTRY_VERSION, this.#engineFingerprint, this.project.dialect, server, sql])).digest("hex");
return join(this.root, ".reylo", "queries", `${key.slice(0, 32)}.json`);
return join(this.root, ".reylo", "queries", `${this.#statementKey(sql)}-${sha256(this.#server()).slice(0, 16)}.json`);
}

#statementKey(sql: string): string {
return sha256(JSON.stringify([ENTRY_VERSION, this.#engineFingerprint, this.project.dialect, sql])).slice(0, 32);
}

/** The statement's entry from this session's server, else the newest from any other: for a session that cannot reach its database, or has no URL. */
#offlineEntry(sql: string): { file: string; entry: Entry } | undefined {
const own = this.entryFile(sql);
const entry = readEntry(own);
if (entry) return { file: own, entry };
const dir = join(this.root, ".reylo", "queries");
const prefix = `${this.#statementKey(sql)}-`;
let names: string[];
try {
names = readdirSync(dir).filter((n) => n.startsWith(prefix) && n.endsWith(".json"));
} catch {
return undefined;
}
// Newest by the time the entry records, since a copy (`cp -r`, `fs.cpSync`, a CI artifact) may
// reset the file's.
let newest: { file: string; entry: Entry } | undefined;
for (const file of names.map((n) => join(dir, n))) {
const other = readEntry(file);
if (other && (!newest || (other.at ?? -Infinity) > (newest.entry.at ?? -Infinity))) newest = { file, entry: other };
}
return newest;
}

/** The statement's analysis: from `.reylo/` when the entry matches the schema, else from the database. */
Expand All @@ -115,18 +153,25 @@ export class Session {

async #analyze(sql: string, options: { force?: boolean }): Promise<Outcome> {
if (this.project.error) return { ok: false, message: this.project.error, code: 11 };
if (!this.project.url) return { ok: false, message: this.project.missingUrl, code: 4 };
// With no URL, `.reylo/` answers as it does where the database cannot be reached (the dialect
// is the installed engine's), and a statement it lacks needs a URL.
if (!this.project.url) {
const hit = this.#engine instanceof MissingEngine || options.force ? undefined : this.#offlineEntry(sql);
return hit ? { ok: true, analysis: hit.entry.analysis, source: "cache", entry: hit.file } : { ok: false, message: this.project.missingUrl, code: 4 };
}
// The URL names the database, whose engine package the project has to have installed.
const engine = this.#engine;
if (engine instanceof MissingEngine) return { ok: false, message: engine.message, code: 11 };
const file = this.entryFile(sql);
const entry = options.force ? undefined : readEntry(file);
const schema = await this.schema();
if (schema.fatal) return { ok: false, message: schema.error!, code: 5 };
if (entry && (schema.value === null || entry.schema === schema.value)) return { ok: true, analysis: entry.analysis, source: "cache", entry: file };
if (schema.value === null) {
const hit = options.force ? undefined : this.#offlineEntry(sql);
if (hit) return { ok: true, analysis: hit.entry.analysis, source: "cache", entry: hit.file };
return { ok: false, message: `${this.cannotConnect(schema.error)}; .reylo has no analysis of this statement, so run reylo generate where the database can be reached.`, code: 5 };
}
const entry = options.force ? undefined : readEntry(file);
if (entry && entry.schema === schema.value) return { ok: true, analysis: entry.analysis, source: "cache", entry: file };
let analysis: Analysis;
try {
analysis = await engine.analyze(sql, this.project.dialect, await this.#connection(), this.#answers);
Expand All @@ -144,7 +189,7 @@ export class Session {
}
return { ok: false, message: (e as Error).message, code: 5 };
}
writeEntry(this.root, file, { version: ENTRY_VERSION, dialect: this.project.dialect, schema: schema.value, analysis });
writeEntry(this.root, file, { version: ENTRY_VERSION, dialect: this.project.dialect, schema: schema.value, analysis, at: Date.now() });
return { ok: true, analysis, source: "database", entry: file };
}

Expand Down Expand Up @@ -201,6 +246,19 @@ export class Session {
return `Cannot type-check against the database at ${this.where()} (${this.project.urlSource}): it is ${e.server}, and Reylo type-checks MySQL statements against MySQL. Point the type check's URL at a MySQL server, 8.4 or later, with the same schema.`;
}

/** The server, without credentials, that names this session's entries: none with no URL, which writes none. */
#server(): string {
const { dialect, url, root } = this.project;
if (!url) return "";
if (dialect === "sqlite") return sqlitePath(url, root);
try {
const u = new URL(url);
return `${u.protocol}//${u.username}@${u.host}${u.pathname}`;
} catch {
return url;
}
}

/** The database as a person names it: host, port and database, or the SQLite file. */
where(): string {
const { dialect, url, root } = this.project;
Expand Down Expand Up @@ -231,18 +289,6 @@ export class Session {
if (io) await io.then((i) => i.close()).catch(() => {});
}

/** The server, without credentials: entries for another database are other entries. */
#server(): string {
const { dialect, url, root } = this.project;
if (dialect === "sqlite") return sqlitePath(url ?? "", root);
try {
const u = new URL(url ?? "");
return `${u.protocol}//${u.username}@${u.host}${u.pathname}`;
} catch {
return url ?? "";
}
}

#connection(): Promise<Io> {
const { dialect, url, root } = this.project;
this.#io ??= (async (): Promise<Io> => {
Expand Down Expand Up @@ -273,6 +319,8 @@ export async function releaseConnections(): Promise<void> {
await Promise.all([...sessions.values()].map((session) => session.release()));
}

const sha256 = (text: string) => createHash("sha256").update(text).digest("hex");

function readEntry(file: string): Entry | undefined {
if (!existsSync(file)) return undefined;
try {
Expand Down
29 changes: 26 additions & 3 deletions src/project.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
// `direct_url`, else `${DATABASE_URL}`, and never `url`, which the file keeps for another use
// (wiki/project-config.md section 4). A file that sets `url` without `direct_url` is told so by
// `reylo generate`, once, and by the error a project with no connection gets; never by an error of
// its own, since TypeScript makes every content mapper diagnostic one.
// its own, since TypeScript makes every content mapper diagnostic one. With no URL, the dialect
// is that of the one engine the project installed.
import { existsSync, readFileSync } from "node:fs";
import { createRequire } from "node:module";
import { dirname, isAbsolute, join, relative, resolve } from "node:path";
import { type Engine, MissingEngine } from "./engine.ts";
import { type Engine, MissingEngine, ENGINE_PACKAGES, enginePath } from "./engine.ts";

export type Dialect = "postgres" | "sqlite" | "mysql";

Expand Down Expand Up @@ -71,6 +73,23 @@ export function findProjectFile(from: string, env: Record<string, string | undef
return PROJECT_FILES.map((name) => join(dir, name)).find((path) => existsSync(path));
}

/** Whether the project itself installed `dialect`'s engine, as against reylo's own install. */
function installedIn(root: string, dialect: Dialect): boolean {
try {
createRequire(join(root, "package.json")).resolve(`${ENGINE_PACKAGES[dialect]}/engine.wasm`);
return true;
} catch {
return false;
}
}

/** The databases whose engine the project installed, else those reylo's own install has. */
function installedEngines(root: string): Dialect[] {
const dialects = Object.keys(ENGINE_PACKAGES) as Dialect[];
const own = dialects.filter((d) => installedIn(root, d));
return own.length ? own : dialects.filter((d) => enginePath(d, root));
}

/** The dialect a URL's scheme names, or undefined for a scheme reylo does not read. */
export function dialectOf(url: string): Dialect | undefined {
const scheme = /^([a-z][a-z0-9+.-]*):/i.exec(url)?.[1]?.toLowerCase();
Expand Down Expand Up @@ -175,10 +194,14 @@ export function resolveProject(engine: Engine | MissingEngine, projectDir: strin
url = hit?.value;
if (hit) urlSource = hit.source;
}
const dialect = url ? dialectOf(url) : "postgres";
let dialect = url ? dialectOf(url) : "postgres";
if (url && !dialect) error ??= `Cannot tell the database from the URL (${urlSource}): reylo reads postgres://, postgresql://, mysql://, file: and sqlite: URLs.`;
// Either name, created or removed, changes the project.
const watched = [...new Set([...(file ? [file] : []), ...both]), ...envFiles];
// No URL names the database: the engine the project installed does, when it installed one, for
// `.reylo/`, which a build with no URL reads (src/analyze.ts).
const installed = url ? [] : installedEngines(root);
if (installed.length === 1) dialect = installed[0];
return { file, root, dialect: dialect ?? "postgres", strictParams: config.strict_params ?? true, cli: config.cli ?? {}, url, urlSource, watched, error, notice, missingUrl };
}

57 changes: 56 additions & 1 deletion tests/analyze.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, renameSync, symlinkSync, writeFileSync } from "node:fs";
import { cpSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, renameSync, rmSync, symlinkSync, utimesSync, writeFileSync } from "node:fs";
import { createRequire } from "node:module";
import { DatabaseSync } from "node:sqlite";
import { tmpdir } from "node:os";
Expand Down Expand Up @@ -85,6 +85,61 @@ test("without the database, .reylo/ answers as it is, and a statement it lacks i
assert.ok(!forced.ok);
});

test("two databases the project uses in turn keep an entry each, so neither analyzes the other's statements again", async () => {
const { root, env } = project();
const db = new DatabaseSync(join(root, "test.db"));
db.exec("create table users (id integer primary key, name text not null, email text)");
db.close();
const test = { DATABASE_URL: "file:./test.db" };
const sources = [];
for (const e of [env, test, env, test]) {
const outcome = await analyze(root, e, "select * from users");
assert.ok(outcome.ok);
sources.push(`${outcome.source} ${outcome.analysis.columns.length}`);
}
assert.deepEqual(sources, ["database 2", "database 3", "cache 2", "cache 3"]);
// Where neither can be reached, a URL with no entry of its own reads the one written last, by
// the time it records, which a copy that resets the files' times (cp -r, fs.cpSync) keeps.
for (const f of ["app.db", "test.db"]) renameSync(join(root, f), join(root, `${f}.gone`));
const queries = join(root, ".reylo", "queries");
const [older, newer] = readdirSync(queries).map((n) => join(queries, n)).sort((a, b) => JSON.parse(readFileSync(a, "utf8")).at - JSON.parse(readFileSync(b, "utf8")).at);
utimesSync(newer!, new Date(2000, 0, 1), new Date(2000, 0, 1));
utimesSync(older!, new Date(), new Date());
const offline = await analyze(root, { DATABASE_URL: "file:./elsewhere.db" }, "select * from users");
assert.ok(offline.ok);
assert.deepEqual([offline.source, offline.entry, offline.analysis.columns.length], ["cache", newer, 3]);
});

test("a build that cannot reach the database reads .reylo/ with another URL, with none, and in a copy of the project elsewhere", async () => {
const { root, env } = project();
// The project installs its database's engine, which names the dialect where no URL does.
mkdirSync(join(root, "node_modules", "@reylosql"), { recursive: true });
symlinkSync(fileURLToPath(new URL("../packages/sqlite", import.meta.url)), join(root, "node_modules", "@reylosql", "sqlite"));
const typed = await analyze(root, env, SELECT);
assert.ok(typed.ok);
assert.equal(typed.source, "database");
// No URL: the entries are used as they are, and a statement they lack needs a URL.
const bare = await analyze(root, {}, SELECT);
assert.ok(bare.ok);
assert.deepEqual([bare.source, bare.entry], ["cache", typed.entry]);
const lacking = await analyze(root, {}, "select name from users");
assert.ok(!lacking.ok);
assert.deepEqual([lacking.code, lacking.message], [4, 'No database URL: set DATABASE_URL in the environment, .env.local or .env, or "direct_url" in .reylo.yaml.']);
assert.ok(!(await analyze(root, {}, SELECT, { force: true })).ok);
// Another URL, to a database the build cannot open, reads the same entry.
const other = await analyze(root, { DATABASE_URL: "file:./elsewhere.db" }, SELECT);
assert.ok(other.ok);
assert.deepEqual([other.source, other.entry], ["cache", typed.entry]);
// A copy of the project at another path, as a container build has it, without its database.
const copy = mkdtempSync(join(tmpdir(), "reylo-analyze-copy-"));
cpSync(root, copy, { recursive: true, verbatimSymlinks: true });
rmSync(join(copy, "app.db"));
const moved = await analyze(copy, env, SELECT);
assert.ok(moved.ok);
assert.equal(moved.source, "cache");
assert.ok(moved.entry.startsWith(join(copy, ".reylo")));
});

test("the module a build step emits carries the analysis and reads nothing at run time", async () => {
const { root, env } = project();
const file = join(root, "user.sql");
Expand Down
Loading
Loading
0