Repository navigation
Harden security, operations, and add R2 shadow delivery #174
Annotations
2 errors
|
verify
Process completed with exit code 1.
|
|
error: expect(received).toMatch(expected):
apps/api-ground-codes#L1
Expected substring or pattern: /Pages deployment ID and commit[\s\S]*Web response[\s\S]*full production smoke/i
(1 - 0.999))`. A 30-day month has 1,440 expected\nslots, so at most 1 failed slot still meets 99.9%. This synthetic calculation\ndoes not claim that a failed slot represents 30 minutes of actual downtime.\n\nCalculate latency compliance separately for `earth.english.encode` and\n`earth.english.search`. An expected slot complies only when the exact check ID\npasses and its recorded `durationMs` < 2,000 ms. Monthly latency compliance is\n`passing-and-fast expected slots / all expected slots`; every missing, errored,\nor slow check is a latency miss. Do not assign a numeric duration to a check\nthat did not complete.\n\n## Evaluation and escalation\n\nUse the [Production Smoke workflow][smoke] as the synthetic source of record.\nPreserve the expected-slot list, matched workflow run, exact check ID, outcome,\nand elapsed time with the monthly calculation. The daily and post-deploy full\nprofiles provide broader regression evidence.\n\nOpen an incident immediately when readiness or the Web root fails, when a\npost-deploy full smoke fails, or when the remaining monthly error budget is\nexhausted. Record every latency miss; open an incident after two consecutive\nlatency misses or immediately when user impact is visible. Follow the\n[incident runbook](./incident-runbook.md) for triage, rollback, evidence\ncapture, and closure.\n\n[smoke]: https://github.com/hmmhmmhm/ground.codes/actions/workflows/production-smoke.yml\n\n# Production Incident Runbook\n\nUse this runbook for a production-smoke failure, SLO alert, elevated API error\nrate, or a bad Worker/Pages deployment.\n\n## 0. Reproduce release artifacts without publishing\n\nProduction workflows and the root `.nvmrc` select Node.js 22. The remaining\nrelease tools are pnpm 9.0.0, Bun 1.3.1, next-on-pages 1.13.16, Vercel CLI\n47.0.4, and the repository-local Wrangler 4.110.0. Select Node.js 22 with your\nversion manager, then fail closed if the release toolchain does not match those\npins:\n\n```sh\nset -eu\ntest \"$(tr -d '\\n' < .nvmrc)\" = \"22\"\ntest \"$(node -p 'process.versions.node.split(\".\")[0]')\" = \"22\"\ntest \"$(pnpm --version)\" = \"9.0.0\"\ntest \"$(bun --version)\" = \"1.3.1\"\ntest \"$(pnpm --filter web exec next-on-pages --version)\" = \"1.13.16\"\ntest \"$(pnpm --filter web exec vercel --version)\" = \"47.0.4\"\ntest \"$(pnpm --filter grok-spiral exec vercel --version)\" = \"47.0.4\"\ntest \"$(pnpm exec wrangler --version)\" = \"4.110.0\"\n```\n\nFrom a clean checkout, install twice and confirm both status commands print\nnothing. This detects any manifest or lockfile mutation while proving the\nsecond install is reproducible:\n\n```sh\nset -eu\ngit status --porcelain -- package.json pnpm-lock.yaml pnpm-workspace.yaml \\\n ':(glob)**/package.json'\npnpm install --frozen-lockfile\npnpm install --frozen-lockfile\ngit status --porcelain -- package.json pnpm-lock.yaml pnpm-workspace.yaml \\\n ':(glob)**/package.json'\n```\n\nBuild the same release targets with lockfile-resolved tools. The Pages commands\nmust report `Vercel CLI 47.0.4` and `Completed pnpm exec vercel build`. Do not\nsubstitute a global tool or use `npx`, `pnpm dlx`, or an unpinned `latest` tool:\n\n```sh\nset -eu\npnpm --filter web pages:build\npnpm --filter grok-spiral pages:build\npnpm --filter api-ground-codes build\nWORKER_DRY_RUN_DIR=\"$(mktemp -d)\"\ntrap 'rm -rf -- \"$WORKER_DRY_RUN_DIR\"' EXIT\npnpm exec wrangler deploy --dry-run \\\n --config apps/api-ground-codes/wrangler.toml \\\n --outdir \"$WORKER_DRY_RUN_DIR\"\nrm -rf -- \"$WORKER_DRY_RUN_DIR\"\ntrap - EXIT\n```\n\nThe Worker dry-run is local and does not require Cloudflare credentials. The\n`--dry-run` flag on the first physical command line is the publication safety\nboundary even when Wrangler authentication is configured. Its temporary output\ndirectory is unique to the command and is removed on both success and failure.\n\nFor an additional lockfile proof, repeat the Pages builds with package fetching\ndisab
|