Repository navigation
Add UAW bootstrap source snapshot and audit evidence #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,55 @@ | ||
| name: UAW bootstrap snapshot checks | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - '**' | ||
| pull_request: | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| source-snapshot: | ||
| name: Syntax and evidence sanity checks | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Check out repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '22' | ||
|
|
||
| - name: Check JavaScript syntax | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| while IFS= read -r file; do | ||
| node --check "$file" | ||
| done < <(find bootstrap/src bootstrap/scripts bootstrap/test -type f -name '*.mjs' -print | sort) | ||
|
|
||
| - name: Validate JSON metadata | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| node --input-type=module <<'NODE' | ||
| import { readFile } from 'node:fs/promises'; | ||
| for (const path of [ | ||
| 'bootstrap/integration-registry.json', | ||
| ]) { | ||
| 8000 | JSON.parse(await readFile(path, 'utf8')); | |
| console.log('valid JSON:', path); | ||
| } | ||
| NODE | ||
|
|
||
| - name: Verify captured test evidence is labelled | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| grep -F '109' bootstrap/test-results.txt >/dev/null | ||
| grep -F 'PostgreSQL' bootstrap/SNAPSHOT-README.md >/dev/null | ||
| git diff --check | ||
| echo 'Captured local test evidence and snapshot limits are documented.' | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: UAW CI | ||
|
|
||
| on: | ||
| push: | ||
| pull_request: | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: uaw-ci-${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| postgres-and-contract-tests: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 15 | ||
| services: | ||
| postgres: | ||
| image: postgres:16 | ||
| env: | ||
| POSTGRES_USER: postgres | ||
| POSTGRES_PASSWORD: postgres | ||
| POSTGRES_DB: uaw_test | ||
| ports: | ||
| - 5432:5432 | ||
| options: >- | ||
| --health-cmd "pg_isready -U postgres -d uaw_test" | ||
| --health-interval 5s | ||
| --health-timeout 5s | ||
| --health-retries 20 | ||
| env: | ||
| UAW_TEST_DATABASE_URL: postgres://postgres:postgres@127.0.0.1:5432/uaw_test | ||
| UAW_TEST_RUNTIME_DATABASE_URL: postgres://uaw_runtime:uaw_runtime@127.0.0.1:5432/uaw_test | ||
| steps: | ||
| - name: Check out source | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 22 | ||
|
|
||
| - name: Install optional PostgreSQL test client | ||
| run: npm install --no-save --no-package-lock --no-audit --no-fund pg@8.23.1 | ||
|
|
||
| - name: Create restricted runtime role | ||
| run: | | ||
| node --input-type=module <<'NODE' | ||
| import pg from 'pg'; | ||
| const client = new pg.Client({ connectionString: process.env.UAW_TEST_DATABASE_URL }); | ||
| try { | ||
| await client.connect(); | ||
| await client.query("CREATE ROLE uaw_runtime LOGIN PASSWORD 'uaw_runtime'"); | ||
| } finally { | ||
| await client.end(); | ||
| } | ||
| NODE | ||
|
|
||
| - name: Run full test suite and integration registry validation | ||
| run: npm run check | ||
|
|
||
| - name: Run webhook-to-worker end-to-end contract | ||
| run: npm run test:e2e |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # UAW bootstrap source snapshot | ||
|
|
||
| This directory preserves the reviewable source snapshot for UAW event-ingress and durable-worker work completed on 2026-10-20. | ||
|
|
||
| ## Included | ||
|
|
||
| - PostgreSQL queue and worker implementations, DDL, and worker lease tests. | ||
| - HTTP webhook ingress, runtime configuration, and GitHub/Linear/Calendly signature profiles. | ||
| - Worker/ingress entrypoints and documentation. | ||
| - Integration registry, local test/validation logs, and the UAW audit report under `docs/audit/`. | ||
|
|
||
| ## Verification evidence | ||
|
|
||
| The complete local package's captured checks reported: | ||
| - `npm run check`: 109 passed, 0 failed, 1 skipped. | ||
| - Registry: 18 integrations — 9 connected, 5 partial, 4 unavailable. | ||
| - The skipped test is the opt-in live PostgreSQL integration suite; no successful live PostgreSQL run is claimed. | ||
| - Contract E2E passed against an in-memory queue double; this is not DB-backed production E2E. | ||
|
|
||
| The repository-root workflow `.github/workflows/uaw-bootstrap-check.yml` checks JavaScript syntax and JSON evidence for this source snapshot. It does **not** resolve all imports, run the complete original npm suite, or validate a live PostgreSQL database. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Correct the repository-root workflow path. This paragraph points to Proposed correction-The repository-root workflow `.github/workflows/uaw-bootstrap-check.yml` checks JavaScript syntax and JSON evidence for this source snapshot.
+The repository-root workflow `.github/workflows/main.yml` checks JavaScript syntax and JSON evidence for this source snapshot.🧰 Tools🪛 LanguageTool[uncategorized] ~20-~20: The official name of this software platform is spelled with a capital “H”. (GITHUB) 🤖 Prompt for AI Agents |
||
|
|
||
| ## Snapshot boundary | ||
|
|
||
| This is a reviewable supplemental source snapshot within the existing Python UAW repository, not a claim that every file from the generated bootstrap ZIP has been mirrored here or that `bootstrap/` is a standalone runnable package. The canonical package manifest, every adapter, the full test tree, and the original package's checksum manifest are not all replicated here yet. The preserved `bootstrap/.github/workflows/ci.yml` is a reference file; GitHub Actions does not discover workflows nested below the repository root workflow directory. | ||
|
|
||
| ## Not production-ready | ||
|
|
||
| - No live database, public webhook endpoint, or long-running worker was deployed. | ||
| - Provider secrets/subscriptions have not been installed. | ||
| - A policy-checked host executor must be explicitly configured. | ||
| - Workflow `demo-001` remains Blocked until real DB-backed verification and provider access are available. | ||
|
|
||
| Do not infer production readiness from local unit/contract tests alone. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| # PostgreSQL event queue adapter | ||
|
|
||
| `src/postgres-event-queue.mjs` implements a durable multi-process event inbox against the `node-postgres` Pool contract. It is a provider-neutral queue core, not an HTTP server and not a finished provider webhook receiver. | ||
|
|
||
| ## Guarantees implemented by the adapter | ||
|
|
||
| - Database uniqueness on `(provider, provider_event_id)` and a stable, payload-independent event key. | ||
| - HMAC-SHA256 validation against the exact raw request bytes before a webhook write is issued. The receiver must pass the untouched body bytes and a signature already extracted into the `sha256=<hex>` form. | ||
| - Atomic claim transactions using `FOR UPDATE SKIP LOCKED`, per-attempt lease tokens (fencing token), and an attempt-history table. | ||
| - Stale lease reclamation; attempts are recorded as `lease_expired` before the lease is replaced. | ||
| - `extendLease()` heartbeat for long-running work. It only renews a still-live owner/token lease, never shortens the current deadline, and updates the attempt-history deadline in the same transaction. | ||
| - Completion requires a verified evidence reference and a still-valid owner/token lease. Failures use bounded exponential backoff with redacted error text. | ||
| - Provider pause/resume, dead-letter listing and explicit dead-letter redrive. Redrive preserves lifetime attempt history rather than reusing old attempt numbers. | ||
|
|
||
| The queue stores the decoded JSON event body but deliberately does not store the webhook signature or signing secret. Provider-specific signature header decoding must be done at the HTTP boundary; providers whose signature format is not plain `sha256=<hex>` need a provider-specific verifier before enabling ingress. | ||
|
|
||
| ## Schema setup | ||
|
|
||
| Apply `schemas/postgres-event-inbox.sql` with a database migration role before starting workers. Alternatively, `queue.initialize()` runs the same `CREATE TABLE/INDEX IF NOT EXISTS` batch and should only be called by a role permitted to create schema objects. For normal runtime operations, provision a least-privilege database role separately; avoid granting schema ownership to the application process. | ||
|
|
||
| PostgreSQL's shared row locks and unique constraints are the cross-process coordination mechanism. The JSON file inbox remains useful for local demos but is **not** a distributed queue. | ||
|
|
||
| ## Minimal wiring example | ||
|
|
||
| ```js | ||
| import pg from 'pg'; | ||
| import { createPostgresEventQueue } from './src/postgres-event-queue.mjs'; | ||
|
|
||
| const pool = new pg.Pool({ connectionString: process.env.UAW_DATABASE_URL }); | ||
| const queue = createPostgresEventQueue({ pool }); | ||
|
|
||
| // Run migrations separately in production. `initialize()` is shown only for a | ||
| // disposable development database whose database role is allowed to create tables. | ||
| // await queue.initialize(); | ||
|
|
||
| // The HTTP framework must preserve request bytes before JSON parsing. | ||
| // The provider's header has to be decoded into `sha256=<hex>` before this call. | ||
| const accepted = await queue.ingestWebhook({ | ||
| provider: 'github', | ||
| eventId: request.headers['x-github-delivery'], | ||
| rawBody: request.rawBody, | ||
| signature: `sha256=${request.headers['x-uaw-signature']}`, | ||
| secret: secretStore.get('github-webhook-secret') | ||
| }); | ||
|
|
||
| if (accepted.duplicate) return { status: 202, duplicate: true }; | ||
| return { status: 202, eventKey: accepted.event.eventKey }; | ||
| ``` | ||
|
|
||
| The sample assumes the HTTP layer has validated the provider event ID and transformed the provider's documented signature format into the queue's `sha256=<hex>` contract. Do not copy that transformation for a different provider without checking its official signing scheme. | ||
|
|
||
| ## Worker lifecycle | ||
|
|
||
| 1. `claimNext({ workerId, leaseMs })` obtains a transactionally claimed event and a new lease token. | ||
| 2. Execute a bounded, policy-checked UAW workflow. For work that may outlast its lease, periodically call `extendLease({ eventKey, workerId, leaseToken, leaseMs })` before expiration. Renewal requires the same live lease and extends from the database's current time without shortening a longer existing deadline. | ||
| 3. If renewal fails, stop starting further work and treat the result of any already-issued external mutation as uncertain; reconcile it before attempting a replay. Queue fencing prevents stale completion/settlement but does not cancel an external API request already in flight. | ||
| 4. On independently verified success, call `complete({ eventKey, workerId, leaseToken, resultEvidence })`. | ||
| 5. On known failure, call `fail({ ..., retryable, maxAttempts })`. A stale or expired lease is rejected. Inspect exhausted events with `listDeadLetters()` and redrive only after an operator/policy decision. | ||
| 6. `pauseSource`/`resumeSource` stop or restore dispatch for a provider; ingress may continue persisting new messages while dispatch is paused. | ||
|
|
||
| ## Validation limits | ||
|
|
||
| The local unit/contract suite uses a mock `Pool` and validates SQL call sequencing, rollback paths, lease fencing/renewal, deduplication and retry policies. The current run reports **67 passed, 0 failed and one skipped**. The skipped test is an opt-in real-database suite enabled with `UAW_TEST_DATABASE_URL` and the optional `pg` package; it creates a temporary schema, exercises concurrent claims/retry/redrive, then removes that schema. | ||
|
|
||
| For a meaningful least-privilege test, set `UAW_TEST_DATABASE_URL` to a disposable migration/admin role that can create/drop a temporary schema and grant privileges, and `UAW_TEST_RUNTIME_DATABASE_URL` to a separate login in the **same database**. The harness creates the schema as the migration role, applies the DDL, and grants the runtime role schema `USAGE`, table `SELECT/INSERT/UPDATE`, and sequence `USAGE/SELECT`. When the roles differ, the test asserts that the runtime role cannot create schema objects, delete events or truncate the inbox. If `UAW_TEST_RUNTIME_DATABASE_URL` is omitted, the suite uses the admin URL for both and does **not** verify separation of runtime privileges. | ||
|
|
||
| This environment has no `psql`/PostgreSQL server and did not execute that integration suite. Treat the target PostgreSQL version, privileges, connection pooling and concurrency behavior as deployment gates until the opt-in suite passes in the target environment. | ||
|
|
||
| ## Reference worker runner | ||
|
|
||
| `src/postgres-worker.mjs` adds a cooperative sequential runner around the queue contract. It is deliberately not a policy engine and does not make provider calls itself. The injected `execute` function must run the normal UAW policy executor and return evidence produced by an independent read-after-write/health check; the runner refuses a success result that lacks `resultEvidence.verified === true` and a non-empty `resultEvidence.evidenceRef`. | ||
|
|
||
| ```js | ||
| import pg from 'pg'; | ||
| import { createPostgresEventQueue } from './src/postgres-event-queue.mjs'; | ||
| import { createPostgresWorker } from './src/postgres-worker.mjs'; | ||
|
|
||
| const pool = new pg.Pool({ connectionString: process.env.UAW_DATABASE_URL }); | ||
| const queue = createPostgresEventQueue({ pool }); | ||
| const worker = createPostgresWorker({ | ||
| queue, | ||
| workerId: process.env.UAW_WORKER_ID, | ||
| leaseMs: 30_000, | ||
| heartbeatMs: 10_000, | ||
| execute: async (event, { signal }) => { | ||
| // Replace with your policy-checked executor; it must support AbortSignal and reconciliation. | ||
| const verified = await runAllowedWorkflowAndVerify(event, { signal }); | ||
| return { resultEvidence: verified.evidence, value: verified.output }; | ||
| }, | ||
| onError: ({ error, stage, eventKey }) => console.error({ stage, eventKey, message: error.message }) | ||
| }); | ||
|
|
||
| process.once('SIGTERM', () => { void worker.stop(); }); | ||
| process.once('SIGINT', () => { void worker.stop(); }); | ||
| try { | ||
| await worker.start(); | ||
| } finally { | ||
| await pool.end(); | ||
| } | ||
| ``` | ||
|
|
||
| Operational behavior: | ||
|
|
||
| - One worker processes one event at a time; horizontal scale comes from multiple processes sharing the PostgreSQL queue. | ||
| - The runner refreshes leases serially and aborts the injected operation if renewal fails. Work that ignores the signal may still have an external request in flight; lease fencing cannot undo that request. | ||
| - An execution exception is passed to `queue.fail()` with bounded retry settings. Configure `isRetryableError` so permanent policy/validation failures do not retry. | ||
| - Missing independent evidence is a non-retryable failure. A connection error during `complete()` or `fail()` is returned as `settlement_error`; the runner does not attempt a second settlement because the first transaction may already have committed. | ||
| - Graceful `stop()` stops polling and wakes an idle worker, but lets the current event finish while heartbeats continue. Ensure service-manager termination grace periods exceed the longest allowed workflow or implement a separate bounded shutdown protocol. | ||
| - The sample intentionally omits provider credentials, HTTP ingress, workflow routing and any production action implementation. Do not use the illustrative `runAllowedWorkflowAndVerify` identifier as if it were a function included in this package. | ||
|
|
||
| The worker runner is covered by mock-contract tests for successful heartbeat, lease loss/abort, retry classification, missing evidence, ambiguous completion and graceful stop. These tests do not replace the opt-in live PostgreSQL suite. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Provider-specific webhook signatures | ||
|
|
||
| Status checked: 2026-10-20. This is implementation guidance for the bundled reference receiver; no webhook subscriptions, public endpoints, or live provider connections were created. | ||
|
|
||
| ## Built-in route factories | ||
|
|
||
| `src/webhooks/provider-routes.mjs` exports three route factories for `createWebhookServer()`: | ||
|
|
||
| ```js | ||
| import { createWebhookServer } from '../src/webhook-server.mjs'; | ||
| import { | ||
| createGitHubWebhookRoute, | ||
| createLinearWebhookRoute, | ||
| createCalendlyWebhookRoute | ||
| } from '../src/webhooks/provider-routes.mjs'; | ||
|
|
||
| const routes = [ | ||
| createGitHubWebhookRoute({ getSecret: () => secretStore.read('uaw/github/webhook-secret') }), | ||
| createLinearWebhookRoute({ getSecret: () => secretStore.read('uaw/linear/webhook-secret') }), | ||
| createCalendlyWebhookRoute({ getSecret: () => secretStore.read('uaw/calendly/webhook-secret') }) | ||
| ]; | ||
| const server = createWebhookServer({ queue, routes }); | ||
| ``` | ||
|
|
||
| Use distinct secrets for separate webhook registrations where the provider supports it. The example calls are inert until mounted by a host process; bind/listen, TLS termination, rate limits, firewall rules, logging, secret store and database pool are host responsibilities. | ||
|
|
||
| ### GitHub | ||
|
|
||
| - Delivery ID: `X-GitHub-Delivery`. | ||
| - Signature header: `X-Hub-Signature-256`, format `sha256=<64 hex chars>`. | ||
| - Signed bytes: exact raw HTTP request body, HMAC-SHA256 with the webhook secret. | ||
| - There is no timestamp in this signature protocol. Deduplicate using the provider's delivery ID and the durable `(provider,event_id)` uniqueness key. | ||
| - The included test verifies GitHub's documented HMAC test vector. | ||
|
|
||
| Official docs: https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries | ||
|
|
||
| ### Linear | ||
|
|
||
| - Delivery ID: `Linear-Delivery`. | ||
| - Signature header: `Linear-Signature`, a hex HMAC-SHA256 of the exact raw request body. | ||
| - Freshness headers/fields: `Linear-Timestamp` in Unix milliseconds and signed JSON `webhookTimestamp`; this route requires both to be fresh (default absolute skew <= 60 seconds) and within the configured skew of each other. | ||
| - The timestamp header itself is not incorporated into the HMAC input. Treat timestamp validation as an additional replay guard, not a replacement for signature verification and durable delivery-ID deduplication. | ||
|
|
||
| Official docs: https://linear.app/developers/webhooks | ||
|
|
||
| ### Calendly | ||
|
|
||
| - Signature header: `Calendly-Webhook-Signature`, with `t=<Unix seconds>,v1=<hex HMAC>`. | ||
| - Signed bytes: UTF-8 timestamp, a literal `.` byte, then the exact raw HTTP request body. | ||
| - Timestamp freshness defaults to an absolute skew of <= 180 seconds. | ||
| - The bundled route derives the event key from SHA-256 of the exact raw request body because a universal delivery-ID header is not documented in the signature instructions. Exact-byte retries deduplicate; this is not a universal logical-event idempotency guarantee. If a trusted provider event identifier becomes available in a subscribed event envelope, prefer it via a reviewed route-specific `eventIdFromPayload`. | ||
|
|
||
| Official docs: https://developer.calendly.com/api-docs/overview/webhooks/webhook-signatures | ||
|
|
||
| ## Why no universal provider route | ||
|
|
||
| Provider signature schemes are not interchangeable. The generic receiver accepts HMAC-SHA256 over the raw body. The provider-specific adapter only normalizes the official signature string and signed bytes; the queue still independently verifies HMAC before persisting the event. Timestamped providers must validate freshness before handing normalized signed bytes to the queue. Unknown schemes should fail closed and must not be approximated by signing re-serialized JSON. | ||
|
|
||
| Notion and the other catalogued integrations do not have a built-in webhook route in this iteration. Do not configure them through one of these three factories based on name similarity; first verify the applicable official protocol and add its own tests. | ||
|
|
||
| ## Verification limits | ||
|
|
||
| The route tests are local HTTP contract tests around a fake queue. They validate signatures, timestamp rejection and idempotent retries without needing a provider account. They do not prove webhook subscriptions, real provider deliveries, TLS, reverse-proxy behavior, DB durability, or deployment-level clock synchronization. The PostgreSQL integration suite must run against the actual target database before production use. | ||
|
|
||
| ## Runnable server configuration | ||
|
|
||
| `npm run start:webhooks` loads `scripts/serve-webhooks.mjs`. Required configuration: `UAW_DATABASE_URL`; `UAW_WEBHOOK_PROVIDERS=github,linear,calendly` (select only the needed providers); and the matching secret variable for each selected provider (`UAW_GITHUB_WEBHOOK_SECRET`, `UAW_LINEAR_WEBHOOK_SECRET`, `UAW_CALENDLY_WEBHOOK_SECRET`). `UAW_PORT` defaults to `8080`, `UAW_HOST` defaults to `127.0.0.1`; non-loopback bind is rejected unless `UAW_ALLOW_PUBLIC_BIND=1`. The runtime verifies that the queue schema is already installed and does not execute migrations. It needs the optional `pg` npm package. `/healthz` is process liveness; `/readyz` checks table presence and a bounded database query. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Check committed changes for whitespace errors.
In a clean Actions checkout,
git diff --checkinspects no changes. Whitespace errors in the pushed commit or pull request therefore pass this step. Compare the appropriate base and head commits for each trigger before runninggit diff --check.🤖 Prompt for AI Agents