Repository navigation
Tags
Tags
feat(platform): move onboarding paywall after personalization (Signif… …icant-Gravitas#15196) ### Why / What / How New users currently encounter the card-required trial/plan checkout before they personalize their account. Move checkout after the role, pain-point/brain-dump and optional expert-hiring steps so users finish personalization before choosing a plan. The hosted wizard proceeds past checkout only after confirming an active trial or paid plan. Save progress asynchronously to the authenticated account and keep a user-scoped local outbox for navigation/offline recovery. Resume uses semantic steps instead of persisted numeric positions. Revision checks reject stale-tab draft writes, and an expected account ID prevents draft writes to a different signed-in account. Trial and paid checkout flush pending progress before leaving; final completion also waits for a successful profile save. ### Changes 🏗️ - Put the paywall after personalization/hiring while preserving self-hosted flows, existing expert creation, and completion/credit behavior. No new experiments or A/B tests. - Add `wizardProgress` and `wizardRevision` to `UserOnboarding` with an additive migration; extend the authenticated onboarding GET/PATCH API and generated OpenAPI contract. - Autosave step/answer changes in the background with a short debounce, durable local outbox, retries, and explicit conflict/retry UI. Navigation does not wait on saves; checkout and final completion do. - Restore confirmed drafts without writing them again. Pending changes and corrected resume steps still sync, and uncertain saves already acknowledged by the server become clean locally. - Wait for every layout flag before freezing the wizard's step order, using the existing bounded fallback and a visible loading status. - Share pending profile saves with completion, show retryable failures, and ignore requests from previous account sessions. Drafts missing a required role resume at Role for correction. - Bound new unpaid cloud personalization to 10 attempts per UTC hour and 30 per UTC day, shared across voice, typed and profile processing. Atomic Redis counters run before costly processing; ordinary draft saves/status checks, paid users and self-hosted deployments do not consume the budget. Existing duplicate checks reuse available results. - Return retryable errors on limits or unavailable protection without discarding recording data. This is an account-level cost guard; broader security hardening remains separate. Deploy the database migration before the updated API and frontend. No environment variables or Docker configuration changes are needed. ### Agents and large language models used OpenAI Codex with GPT-6, including three collaborating agents. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [ ] I have tested my changes according to the test plan: - [x] Backend focused tests: 151 passed covering durable drafts, validation, account ownership/CAS, HTTP contract, quota boundaries/outages, duplicate admission, recording recovery, profile handling and existing brain-dump behavior. Backend code is unchanged in the review fixes. - [x] Frontend format, lint and types. All 130 onboarding tests pass with coverage; changed hooks have 92–100% line coverage. - [x] Full frontend CI suite with coverage on `692b2d3`: **9,120 tests across 846 files passed**, with zero failed/skipped tests across all four shards. [Workflow](https://github.com/Significant-Gravitas/AutoGPT/actions/runs/37531571488). The local broad run did not complete; these full-suite results are from CI. - [x] Regression tests reproduced delayed-flag, failed/pending-profile-save, empty-role recovery, and redundant-restore-save failures before the fixes; targeted suites pass afterward. - [x] Repository pre-commit checks on the review fixes, including secret scanning, formatting and TypeScript. - [ ] Staging smoke test with a real test-mode Stripe checkout, reload/new-session resume, cancellation and delayed webhook confirmation. Live Stripe, Redis and Postgres integration was not exercised locally. #### For configuration changes: - [x] `.env.default` is already compatible with these changes - [x] `docker-compose.yml` is already compatible with these changes - [x] Configuration/migration changes are listed above --------- Co-authored-by: GPT-6 (Codex) <agent@example.invalid> Co-authored-by: Nicholas Tindle <nick@ntindle.com> Co-authored-by: Claude Opus 5.5 (Claude Code) <noreply@anthropic.com>
feat(platform): signup consent line, marketing opt-out, and MailerLit… …e gate (Significant-Gravitas#15152) ### Why / What / How **Why:** [SECRT-2783](https://linear.app/autogpt/issue/SECRT-2783). Signing up with Google skipped the mandatory terms checkbox, so two thirds of new accounts have no recorded agreement. Nothing stored when anyone agreed, or to which version. The form also gave people no way to refuse marketing email, which the UK soft opt-in needs before we run the win-back and MailerLite lifecycle emails. **What:** The signup page now matches the final mockup on the ticket: - **Layout:** "Already a member? Log in" sits under the title. The checkbox is gone. One 14px line sits under the buttons: *"By continuing you agree to our Terms of Use and Privacy Policy. We may email you product updates and offers; opt out."* - **Opt-out:** "opt out" swaps the second sentence in place to *"You won't get marketing emails. Undo."* - **Recording:** both signup paths (email and Google) write `termsAcceptedAt`, `termsVersion` and, if the person opted out, `marketingOptOutAt` / `marketingOptOutSource = signup` on the new user. A returning account that opts out on `/signup` before continuing with Google has the refusal recorded too. - **MailerLite:** - An opted-out account is never written there, by any live path or backfill. The audience consumer re-checks the refusal right before each write. - A new refusal unsubscribes an existing MailerLite subscriber. - An unsubscribe in MailerLite is recorded back as an opt-out. | Mockup (from the ticket) | Built (`/signup`, cloud) | Built, after "opt out" | |---|---|---| | <img src="https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/f34cdc90b9b7e3e0080943e66b781e0c81ba5058/mockup.png" width="420"> | <img src="https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/f34cdc90b9b7e3e0080943e66b781e0c81ba5058/signup-new.png" width="300"> | <img src="https://raw.githubusercontent.com/Significant-Gravitas/AutoGPT/f34cdc90b9b7e3e0080943e66b781e0c81ba5058/signup-opted-out.png" width="300"> | The built screenshots come from the real page in Chromium against `next dev`. The "Watch the demo" pill was 10BC8 already removed by Significant-Gravitas#15095. **How:** - **Frontend** - `SignupLegalLine` uses the `Text` atom at `body` (14px; `small` is 12px) and the `Link` atom for the policy links, which open in a new tab. - "opt out" / "Undo" is one real `<button type="button">` styled like the links. Focus stays on it after a toggle, and the swapped sentence is in an `aria-live` region. - The choice is a `marketingOptOut` form value. It is locked once a signup is in flight, so the page cannot confirm a refusal that was never sent. - Opting out (not undoing) captures `marketing_opted_out` with `surface: "signup"` through `capturePostHogEvent`, so it is held until the cookie banner is answered rather than dropped. - **Recording consent** - **Email path:** the server action calls the new `POST /auth/user/consent` right after `markAccountCreated("email")`. The page clears the opt-out cookie first, so neither this choice nor one left by an abandoned Google attempt reaches a later Google sign-in. - **With `AUTH_REQUIRE_EMAIL_VERIFICATION` on (Significant-Gravitas#15135):** sign-up returns no session, so nothing is recorded then. The refusal travels in the verification link itself (`/auth/callback?method=email&marketing_opt_out=1`, also on a link resent from the check-your-inbox screen), so it holds whichever browser opens the link and however late. The callback records the terms and the refusal on the account the link provisions. It ignores the parameter for an existing account or a non-email sign-in, so a crafted link can't change anyone's preferences. No cookie is set for email sign-ups. If a mail scanner opens the link first, the account is set up at the owner's next sign-in instead (`login/actions.ts`), which records the terms. A refusal can't be recovered on that path, nor on a fresh link requested later from `/login`; storing it on our side at sign-up is [SECRT-2851](https://linear.app/autogpt/issue/SECRT-2851). - **Google path:** - `handleProviderSignup` sets a 10-minute `agpt_marketing_opt_out=1` cookie before leaving for Google, or clears a stale one. - `/auth/callback` reads and clears that cookie once provisioning succeeds. - It records consent when the account was just created, and also for a returning account when the cookie carries a refusal, since the page has already confirmed it. - A consent-write failure goes to Sentry with the user id and the opt-out choice, so a lost refusal can be re-applied by hand. It never rolls back the signup. - **Backend** - `record_signup_consent` is idempotent. - Terms are stamped again only when the version changes. - The opt-out write is conditional: the first refusal wins, including a concurrent one. - `marketing_opt_out=false` never clears a refusal. - A row deleted mid-write is a 404, not a database error. - `terms_version` must be a version the service recognizes (`YYYY-MM` or `YYYY-MM-DD`, ASCII digits). - **MailerLite** - Every MailerLite write upserts the subscriber, a group removal or a field update included. So `notifications/consent.py` gates every audience change, not just the adds: - the live checkout opener; - the checkout-completed webhook; - the first-subscription tour and the returning-customer changelog; - the subscription field updates; - churn; - the trial group; - the trial conversion. - The audience consumer re-reads the refusal from the database before applying each queued change, through a new `DatabaseManager` RPC, `is_marketing_opted_out`. It drops a change for an account that refused after the change was queued. - A deleted account counts as opted out. - A failed read is retried rather than written through. - A newly recorded refusal queues an `UNSUBSCRIBE` change, exempt from that re-check because it carries the refusal. It marks an existing, still-mailable subscriber unsubscribed with `PUT /subscribers/{id}` and never creates one. - `POST /api/email/mailerlite/webhook` takes MailerLite's `subscriber.unsubscribed` event and sets `marketingOptOutAt` with source `email_unsubscribe`. - Calls are authenticated by the `Signature` header (HMAC-SHA256 of the body under `MAILERLITE_WEBHOOK_SECRET`). - Without a secret, every call is refused with 503. - Billing and trial emails, their dedupe claims and the welcome claim behave exactly as before for opted-out users. - The checkout, group and field backfills drop opted-out accounts in their `plan` step and report how many they skipped. - Skips are logged at debug level by pseudonym, never by address. ### Changes 🏗️ - **Database:** `User.termsAcceptedAt`, `termsVersion`, `marketingOptOutAt`, `marketingOptOutSource` (all nullable), with migration `20261007120000_user_consent_fields`. Adding the columns is a catalog-only change with no table rewrite. The same fields are on the Pydantic `User` and on `BillingEmailRecipient`. - **API:** - `POST /api/auth/user/consent` (`requires_user`; user id from auth only), backed by `record_signup_consent` in `data/user.py`. - `POST /api/email/mailerlite/webhook` (signature-authenticated). - The regenerated `openapi.json`. - **MailerLite:** - `notifications/consent.py` (`marketing_allowed`, `audience_change_allowed`). - Gated in `data/checkout_audience.py`, `notifications/lifecycle.py` and `notifications/trial_audience.py`. `trial_audience` now takes the user; `trial.py` is its only caller. - The consumer re-check in `notifications/notifications.py` and `DatabaseManager.is_marketing_opted_out`. - The `UNSUBSCRIBE` action in `notifications/mailerlite.py`, and the webhook in `api/features/mailerlite/routes.py`. - `checkout_backfill.py`, `mailerlite_backfill.py` (new `skip_opted_out` decision) and `mailerlite_field_backfill.py` skip opted-out accounts, and the `cli/mailerlite_backfill.py` reports show the count. - **Signup page:** `page.tsx`, `useSignupPage.ts`, new `components/SignupLegalLine.tsx`, and `types/auth.ts` (`agreeToTerms` replaced by `marketingOptOut`). - **Signup paths:** `signup/actions.ts`, `auth/callback/route.ts`, `lib/auth/email-verification.ts` and `CheckYourInbox` (the opt-out in the verification link), `login/actions.ts` (terms recorded when a login sets the account up), new `lib/auth/server/recordSignupConsent.ts`, `services/analytics/marketing-opt-out-cookie.ts` and `marketing-opt-out-server.ts`. - **Shared constants:** new `lib/legal.ts` (`TERMS_OF_USE_URL`, `PRIVACY_POLICY_URL`, `TERMS_VERSION = "2026-10"`). - **Analytics:** `marketing_opted_out` in `posthog-events.ts`, its pin test, and `docs/engineering/tracking-plan.md`. - **E2E helper:** `playwright/utils/signup.ts` no longer clicks the removed checkbox. - **Tests:** - Unit, route and real-database tests for consent recording. - An opted-out test for every gated path and backfill. - Tests for the consumer re-check, the unsubscribe action and the webhook. - Page, action, callback and cookie tests. - Details are in the test plan below. **Notes for reviewers / open questions** 1. **Google signups from `/login`.** The Google button on `/login` can also create an account. `/auth/callback` serves both pages, so such an account also records terms acceptance, even though `/login` shows no notice. This follows the spec ("record for every new Google account"; the login page is untouched). If we want a record only when the notice was shown, the options are the same line on `/login`'s Google button, or a signup-intent marker. 2. **"Visible in admin".** The fields are queryable columns on `User`. The in-app admin users page is still an empty placeholder, so there is no in-app view yet. 3. **Rolling deploy.** `get_user_by_id` is a shared Redis cache with a 5-minute TTL, and a `User` pickled by the previous release has no `marketing_opt_out_at` attribute. - The gate treats such a cached user as "not allowed" without raising. For up to 5 minutes after deploy a few MailerLite writes are skipped (the backfills catch them up), and no trial email is held up. - It reads the field defensively, as `get_user_default_chat_route` already does for the same reason. - It does not default to "allowed", because an old pod re-caching a just-opted-out user could then let them through. The consumer re-check reads the database, not the cache. 4. **MailerLite webhook setup.** To turn the webhook on: - Create a `subscriber.unsubscribed` webhook in MailerLite pointing at `/api/email/mailerlite/webhook`. - Set its secret as `MAILERLITE_WEBHOOK_SECRET`. - Until then the endpoint refuses every call. 5. **Spacing.** "Sign up" keeps the login page's `mt-6` after the last input (the `Input` atom reserves room for its error line), so the gap is a little larger than in the mockup. 6. **User-ID checks (`data/*.py`).** - `record_signup_consent` only ever writes the caller's own row, because the route takes the id from `Security(get_user_id)` and never from the body. - The webhook matches the account by the email in the signed MailerLite payload. - `checkout_audience` and `BillingEmailRecipient` keep their existing lookups (authenticated user id / Stripe customer id). ### Agents and large language models used - Claude Code with Claude Opus 5.5 ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] **E2E on a local stack** (`next dev` in cloud mode, REST API, Postgres, Redis, RabbitMQ), with two real email signups: - with "opt out": `termsAcceptedAt` set, `termsVersion = 2026-10`, `marketingOptOutAt` set, `marketingOptOutSource = signup`; - without: the terms are stored and the opt-out stays null. - `/signup?next=/library` keeps `?next=` on the subtitle's Log in link and lands on `/library`. - [x] **Live gate check:** `queue_checkout_opened` against the real database and RabbitMQ. - The opted-out user queued nothing (debug skip by pseudonym). - The other user queued one `checkout_opened` change. - [x] **Browser check:** the legal line is 14px; both policy links open in a new tab; after a click, keyboard focus stays on the toggle (now "Undo") with no lingering ring. - [x] **Migration** applies cleanly on `pgvector/pgvector:pg15` after the full history, and `prisma migrate diff` shows no drift on `User`. `prisma validate` passes. - [x] **Frontend:** `pnpm lint`, `pnpm types`, and the full `vitest` suite in CI's 4 shards all pass. `generate:api` leaves no diff beyond the new endpoints. - [x] **Backend:** `ruff`, `isort`, `black` and full `pyright` pass, and all four CI test shards pass. No new skip or xfail markers. - [x] **New tests fail without the feature:** the page lock, callback cookie ordering, returning-account refusal, Sentry context, stale-cache gate, consent idempotency and race tests each fail when their fix is reverted (checked by temporarily reverting each fix). - [ ] Google signup end to end (needs real Google OAuth credentials): opt out, Continue with Google, check the four fields; and the same without opting out. - [ ] Against a real MailerLite account: - An opted-out account never appears after opening checkout. - A refusal unsubscribes an existing subscriber. - An unsubscribe in MailerLite sets `marketingOptOutAt`. #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) Configuration changes: - `MAILERLITE_WEBHOOK_SECRET` is added, blank in `.env.default`. It is MailerLite's webhook signing secret, and while it is blank the webhook refuses every call. - The additive, nullable-column migration listed under **Changes**. <details> <summary>Examples of configuration changes</summary> - Changing ports - Adding new services that need to communicate with each other - Secrets or environment variable changes - New or infrastructure changes such as databases </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012p8k42PpEJhnAWcnnskWr4 --- _Generated by [Claude Code](https://claude.ai/code/session_012p8k42PpEJhnAWcnnskWr4)_ --- _Generated by [Claude Code](https://claude.ai/code)_ --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Krzysztof Czerwinski <kpczerwinski@gmail.com>
feat(backend): backfill existing subscribers into MailerLite and add … …a trial group (Significant-Gravitas#15026) ### Why / What / How **Why.** The MailerLite code only reacts to new Stripe events, so everyone who subscribed before it shipped is in no group, and there was no way to sync them. Trials were never handled either: `on_checkout_completed` returns early for a trial, and a conversion only sent the notice, so a trial customer never joined any MailerLite group, even after paying. Nick asked for both: sync the existing subscribers, and add a trial group. GTM then asked for server-side status and date fields on every MailerLite subscriber, including plain signups, which have never reached MailerLite from this code and have not arrived there at all since 2026-06-24. **What.** 1. **A one-off backfill:** `poetry run cli mailerlite-backfill`. It is a **dry run by default**, and it writes only with `--apply` plus a confirmation prompt. 2. **A MailerLite trial group** that holds exactly the customers currently on a trial: - they **join** when the trial starts, and again on an un-cancel; - they **leave** on a cancel, a conversion, a failed conversion or the trial ending. - A conversion also joins the paying audience, as a first checkout does: the onboarding tour for a first subscription, otherwise the changelog. 3. **MailerLite status and date fields**, written at signup and on every trial and paid transition, and backfilled for every account. Details, and the field keys for GTM, are below. **How.** - **Backfill.** It reads every Stripe subscription, and matches each customer to their account by email, as the live handlers do. It then reads the current membership of the tour, changelog and trial groups (every subscriber status), and applies only the difference. Rules: - paying → changelog, **unless they are in the tour group**. They are either mid-tour or finished, and MailerLite's own automation owns the tour → changelog handoff; - churned → out of the changelog; - trialing and not set to cancel → trial group; anyone else leaves it; - `incomplete`, `unpaid`, `paused`, and past due on a subscription that began as a trial → left alone. - **How it writes.** It sends the same calls the live handlers make, 50 per `POST /api/batch`. Upsert batches are paced 13s apart for MailerLite's 5-per-minute import limit, and a 429 is retried with backoff by `Requests`. Output is one pseudonymised line per customer (`_pseudonym`, never an email) plus counts. It is idempotent: a second run reads and writes nothing. - **Trial group in the live code.** Every trial transition already produces exactly one trial notice with its own dedupe claim, so the group change rides on it (`trial_audience.py`). The change is queued *before* the notice, inside the same release-on-failure block, so a failed publish lets Stripe's retry make it. Both changes are idempotent. Conversion takes the welcome claim, because the conversion notice is the welcome, so a later resubscription is treated as a returning customer. A MailerLite change never costs the customer their notice. If the audience event refuses an address (for example a reserved domain), the change is skipped. Joining the paying audience after a conversion reports failures instead of raising. New `AudienceAction.ADD_TRIAL` and `REMOVE_TRIAL` follow the existing pattern and routing (`notification.audience.#`). - **Ownership.** The backend owns both edges of the trial group, and this is written into `mailerlite.py`'s one-owner-per-transition docstring. Any MailerLite automation on the trial group may send mail, but must never move people. **Considered and not used:** MailerLite's native Stripe integration only covers products sold through MailerLite's own checkout blocks (Comfort/Power plans). The alternatives are Zapier, Make or MailerBuddy. They are event-driven third parties that would write the same edges without our rules, which is exactly the double-owner conflict the module avoids. #### MailerLite status and date fields, and signup sync GTM asked for server-side MailerLite fields that say where each person stands with us, plus the real signup date. Plain signups never reached MailerLite, and nothing new has arrived there since 2026-06-24. This adds: - **Fields on every audience event.** `AudienceEventModel` now carries `fields`, alongside a group change or on its own (the new `update_fields` action). A group add writes the fields in the same upsert. A removal writes the fields, then removes. Fields are written even while a group ID is blank: a blank trial group means fields only, and a blank tour or changelog group retries as it always has. Missing fields are created in MailerLite before the first write (checked once per process). - **Signup sync.** When `_get_or_create_user` creates the platform `User` row, it queues `signed` plus the signup date (the row's `createdAt`) as a background task. Signup never waits on the broker and never fails because of it. Reserved-domain addresses are skipped. It is queued whatever the API server's own settings are (see below). - **Lifecycle updates.** The trial and paid handlers set the status and the date each transition is named for, from Stripe's own timestamp. A status update never costs a billing email: on the paid path it is reported, not raised. - **Backfill for every account**, not just Stripe customers. It is diffed against MailerLite's current fields, so a rerun resumes an interrupted one. #### Field keys for GTM | Field key | Type | Set when | Value | |---|---|---|---| | `subscription_status` | text | every transition below | one of the six statuses below | | `signup_date` | date | account created | the account's creation date (our DB). Unlike MailerLite's own "Subscribed" date, this is the real signup. Never cleared | | `trial_started_date` | date | trial starts | Stripe `trial_start`. Never cleared | | `subscription_started_date` | date | first checkout, trial converts, resubscribe | Stripe `start_date`; for a converted trial, Stripe `trial_end` (when the first paid period began) | | `subscription_canceled_date` | date | a paid subscription is set to cancel at period end | Stripe `canceled_at`. Cleared on resume or a new subscription | | `subscription_ended_date` | date | a paid subscription ends | Stripe `ended_at`. Cleared on a new subscription | Dates are UTC days (`YYYY-MM-DD`), the only format MailerLite date fields accept. The dates describe the current or most recent subscription. | Event | `subscription_status` | |---|---| | Account created | `signed` | | Trial started, or trial un-cancelled | `in_trial` | | Trial cancelled, trial's first payment failed, trial ended unpaid | `trial_canceled` | | First checkout, trial converted, resubscribed, paid sub un-cancelled | `subscribed` | | Paid sub set to cancel at period end | `subscription_canceled` | | Paid sub ended | `subscription_ended` | A trial that ends without its first payment clearing is `trial_canceled`, not `subscription_ended`. They never paid, so counting them as churned customers would be wrong. It also keeps `in_trial` exactly equal to the trial group's membership. #### Where MailerLite is configured: the notification service only - The API server runs signup and the Stripe webhook. It queues every audience change without reading any `MAILERLITE_*` setting, so those settings are needed only on the notification service, where infra already puts them. The first version checked the token and the trial group on the API side, which prod never sets there, so after deploy nothing would have been queued. Found in review and fixed in 722b6de. - The notification service decides what happens: - **No token:** the change is acknowledged and dropped, with one info log per process. It is not retried or dead-lettered, so self-hosted stacks stay quiet. - **Blank trial group:** only the fields are written. - **Blank tour or changelog group:** the change is retried, then dead-lettered, as before. #### Why nothing reached MailerLite after 2026-06-24 (as far as the code can tell) - No MailerLite code exists anywhere in `master` before Significant-Gravitas#14003 (2026-08-26), and that only moved group membership on Stripe events, never signups. Whatever fed MailerLite until 2026-06-24 lived outside this repo (for example a Supabase or Zapier integration), so it can't be seen from here. The Supabase to Better Auth switch (Significant-Gravitas#13330, on `master` 2026-07-30) is a plausible cause, but it landed after 06-24. - On the code in production today, blank settings do not fail silently or retry forever. The job raises `MailerLiteNotConfigured`, is retried 3 times with backoff, then goes to the dead-letter queue (`failed_notifications_v3`, key `failed.audience`) with a logged error. Nothing replays the dead-letter queue. With this PR, a missing token no longer does this: it is logged once, as above. - Inference: production has run Significant-Gravitas#14003 since 2026-08-26. If `MAILERLITE_*` were set there, every new paying customer since then would have appeared as a new MailerLite subscriber, and GTM sees none after 06-24. **The production MailerLite token and/or group IDs are most likely blank.** Someone with production access should check the env, Sentry for `MAILERLITE_API_TOKEN is not set`, and the dead-letter queue for `failed.audience` messages. Until the token is set, this PR changes nothing in production. ### Changes 🏗️ - **New env var `MAILERLITE_TRIAL_GROUP_ID`** (Config, blank by default, added to `.env.default`). While it is blank, trial changes write only the status and dates, and nobody is moved into or out of a trial group. Conversion still joins the tour or the changelog. - `backend/notifications/mailerlite_backfill.py` + `backend/cli/mailerlite_backfill.py`: the backfill (`cli mailerlite-backfill [--apply] [--yes]`). - `backend/notifications/trial_audience.py`: trial group change per notice kind, and joining the paying audience at conversion. - `trial.py`: `notify_trial` queues the group change with the notice, and joins the paying audience on conversion. - `mailerlite.py`: `add_to_trial` and `remove_from_trial`. Removal is shared with the changelog through `_remove_from_group`. - `data/notifications.py` and `notifications.py`: the new actions and their consumer routing. - **New setting `MAILERLITE_API_URL`** (default `https://connect.mailerlite.com/api`). Only a test stack changes it, to point at a stub, so it is not in `.env.default`. - `data/notifications.py`: `AudienceEventModel.fields`, `AudienceAction.UPDATE_FIELDS`, and the `SubscriberField` / `SubscriptionStatus` enums. - `backend/notifications/subscriber_fields.py`: the status and date for each transition, and the signup event. - `data/user.py`: `_get_or_create_user` queues the signup fields in the background for the row it just created. It only reads that new row's own `id`, `email` and `createdAt`, so no user ID check is needed. - `lifecycle.py` and `trial_audience.py`: each paid and trial transition carries its status and date. - `mailerlite.py`: field writes with every group change or alone, and `ensure_fields` creates missing fields once per process. - `notifications.py` and `mailerlite.configured()`: the consumer acknowledges audience changes when there is no token. `trial_audience.py` and `subscriber_fields.py` read no MailerLite setting (722b6de). - `backend/notifications/mailerlite_field_backfill.py` + `backend/cli/mailerlite_backfill.py`: the field backfill for every account (`cli mailerlite-backfill [--fields-only]`), and `cli mailerlite-fields [--apply]`. **Behaviour to review:** `payment_failed` (the trial is over but the customer has not paid) also removes them from the trial group, because they are no longer on a trial. They join the paying audience if the conversion later succeeds. > [!IMPORTANT] > **New env var: `MAILERLITE_TRIAL_GROUP_ID`.** Create the trial group in MailerLite and set this on the notification service in each environment, next to the other `MAILERLITE_*` values. **While it is blank, nobody joins or leaves a trial group.** Trial changes still write the status and dates, and a conversion still joins the tour or the changelog. **Running the backfill (a human only, never CI or an agent):** 0. First set `MAILERLITE_API_TOKEN` (and the group IDs) on the notification service. Changes queued before that are acknowledged and dropped, and the backfill is what catches everyone up, so it has to run after the token is in place. 1. Dry run first: `poetry run cli mailerlite-backfill`. It needs `STRIPE_API_KEY`, `MAILERLITE_API_TOKEN`, the tour and changelog group IDs, DB access, and optionally `MAILERLITE_TRIAL_GROUP_ID`. It writes nothing, and prints one pseudonymised line per customer plus group counts, then the field counts per status, how many accounts would be written, and how many would be **new MailerLite subscribers**. GTM should approve that last number first: every account becomes an active subscriber, which counts against the MailerLite plan. 2. Review the counts. Then run `poetry run cli mailerlite-backfill --apply`, which asks for confirmation before writing unless `--yes` is given. A rerun is safe: it only writes what is still different, so it also resumes an interrupted run. 3. Adding people to a group through the API fires any MailerLite automation triggered by *joining* that group. Check the changelog and trial groups' automations before `--apply`. For the fields: - `poetry run cli mailerlite-fields` lists the six fields and whether MailerLite has them. `--apply` creates the missing ones. This is optional: the notification service creates them before its first field write. - `--fields-only` writes the status and date fields and leaves group membership alone. It is also the option for when the group IDs are not configured, and then needs only `STRIPE_API_KEY`, `MAILERLITE_API_TOKEN` and database access. - Pacing is 50 upserts per `/batch`, spaced for MailerLite's import limit: about 230 accounts a minute. An interrupted run is resumed by running it again, because only fields that differ from MailerLite are written. **Found while here, not fixed (related to Significant-Gravitas#14992):** `AudienceEventModel.email` is an `EmailStr`, which refuses reserved domains such as `.test` and `.local`. In `lifecycle.on_checkout_completed`, the tour event is built *after* the welcome has been published and claimed. For such a user, the webhook raises, and every Stripe retry takes the returning-customer branch and raises again. This PR guards its own trial paths against that (skip, never fail the notice), but leaves the existing welcome path alone. ### Evidence (tester VM, stub MailerLite + stub Stripe, synthetic `@example.com` customers) BEFORE is `dev` at 2f09a06 and AFTER is ad0a3e7. Each was deployed with `git checkout <sha>` + `deploy-dev --current`, with HEAD recorded. Both services were local HTTP stubs that record every request; no real Stripe or MailerLite account was used. **Backfill, 132 customers:** - BEFORE: `No such command 'mailerlite-backfill'`, and **0** MailerLite calls for 133 existing subscriptions. - AFTER, dry run: 131 decisions checked against the expected ones, **0 mismatches**, 15 reads and no writes. - AFTER, `--apply`: 7 batch calls (123 changelog adds, 1 trial add, 1 changelog removal, 1 trial removal). The first batch was answered with a 429 and succeeded on retry. The mid-tour customer and a newsletter-only subscriber were left alone. - AFTER, second `--apply`: "Nothing to write." **Live trial flow.** This runs the real `lifecycle.py`, `trial.py`, audience consumer and `mailerlite.py`. The DB lookups, Redis claims, email queue and RabbitMQ hop are stubbed at the boundary. | step | before (dev) | after | |---|---|---| | first paid checkout | tour | tour (unchanged) | | trial started | no call | + trial group | | cancelled during trial | no call | − trial group | | un-cancelled | no call | + trial group | | converted | no call | − trial group, + tour | | trial ended unpaid | no call | − trial group | | paid subscription ended | − changelog | − changelog (unchanged) | **Tests.** - New tests on `dev` fail: the modules are missing. - With only the `trial.py` hook reverted: 9 failed, 9 passed. - On ad0a3e7: `backend/notifications/` + `backend/cli/` **316 passed**. pyright, ruff, black and isort are clean on the changed files. - Robustness fix (f36addc): its 3 tests fail on the test-only commit 9fd4037. With the fix, `backend/notifications/` **296 passed**, and ruff, isort and pyright are clean. `backend/cli/` was not re-run for this commit because it needs the live DB, and the fix does not touch the CLI. #### Status and date fields (`db27885e6f`, same stubs) - **Before (dev `2f09a06bad`):** a real Better Auth signup publishes 0 audience messages. The whole trial-to-paid lifecycle publishes one (`remove_changelog`, no fields), and the stub receives 0 MailerLite calls. - **After (`db27885e6f`):** each step gave the expected status and date: | Step | Status | Date set | |---|---|---| | Signup | `signed` | `signup_date` 2026-09-29 | | Trial start | `in_trial` | `trial_started_date` 2026-09-22 | | Trial cancel | `trial_canceled` | | | Trial un-cancel | `in_trial` | | | Convert | `subscribed` | `subscription_started_date` 2026-09-25 | | Paid cancel | `subscription_canceled` | `subscription_canceled_date` 2026-09-27 | | Paid un-cancel | `subscribed` | `subscription_canceled_date` cleared | | End | `subscription_ended` | `subscription_ended_date` 2026-10-25 | | Trial ending unpaid | `trial_canceled` | | - **Fields:** missing before the first write, created by it. - **Backfill:** over 143 accounts, with Stripe subscriptions, pre-existing MailerLite subscribers and invalid emails, every account got the right values. Unsubscribed people stayed unsubscribed. A second run found nothing to write. An `--apply` stopped after its first batch of 50 out of 138 left 88 to write, and the rerun wrote exactly those 88. - **Unit tests:** fail against the base code (21 failed, 7 errors) and pass on the branch (163 passed across `backend/notifications` and the user tests). Not verified against real MailerLite: - that creating a field named `subscription_status` gives it that key (the code fails loudly if it doesn't); - that `null` clears a date field; - the 1000-per-page subscriber listing. #### MailerLite settings only on the notification service (`722b6de996`) This is prod's shape. The API processes (`backend.app` without its NotificationManager) and the webhook handlers ran with `MAILERLITE_*` blank; the process environments were captured. The notification service ran alone, with the settings pointing at the stub. - **Before (`db27885e6f`):** - A real signup published 0 audience messages, and the stub stayed empty. - Trial start, cancel and un-cancel published nothing. - Conversion and the end published only `enroll_tour` / `remove_changelog`, with empty fields. - The subscriber ended up with no status or dates. - **After:** 1 message at signup and 8 across the lifecycle, each with its fields. The stub ended with every status and date in the table above, and the trial group moved in and out. - **No token anywhere (self-hosted):** two signups were queued and acknowledged, the dead-letter queue stayed at its count, one info line was logged, and no MailerLite calls were made. - **Unit tests:** the new ones fail on `db27885e6f` (33 failed, 3 errors) and pass with the fix. `backend/notifications` + `backend/cli`: **383 passed**. ruff is clean. ### Agents and large language models used - Claude Code with Claude Opus 5.5 (1M context) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Unit tests: backfill classification, decisions, batching and pacing, failure accounting, group reads and idempotency; each trial transition's group change, conversion to the tour or changelog, trial group unset, a replayed event, and a failed queue releasing the claim; the consumer routing; the MailerLite trial calls - [x] VM: backfill dry run, `--apply` and a second run against stubs, BEFORE and AFTER - [x] VM: live trial flow start → cancel → un-cancel → convert, and start → end unpaid, BEFORE and AFTER - [x] Unit tests: status and dates per transition, signup sync, field writes with and without a group change, field creation, the field backfill diff and resume - [x] VM: signup, every trial and paid transition, and the field backfill (dry run, interrupted `--apply`, rerun), BEFORE and AFTER - [x] VM: MailerLite settings only on the notification service (prod's shape), BEFORE and AFTER, plus no token anywhere #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/master' into dev # Conflicts: # autogpt_platform/frontend/instrumentation-client.ts
fix(backend/copilot): apply the building-mode guide on restart instea… …d of re-deriving it from history (Significant-Gravitas#14721) ### Why AutoPilot refuses to save an agent it has just designed. `enter_agent_building_mode` must load the agent-building guide before `create_agent` is allowed; on the SDK engine the guide goes into the system prompt, which can only be changed by relaunching the turn. That relaunch applied an **empty** guide and then told the model "Building mode is now active — the complete agent-building guide is in your system prompt", so the gate could never clear, and the user was told the platform is broken. Dev logged it 16 times in six hours across 6 of 11 chat sessions (2026-09-18 20:00Z → 09-19 02:10Z), every one at ERROR: 9 of 9 restarts on the pre-Significant-Gravitas#14714 image (20:09–20:17Z), 7 of 12 after the 00:43Z rollout. Session `c91efb40-559b-45fa-8390-388fa6e516a4` shows it three times inside one turn — 01:59:05.917Z, 01:59:19.811Z and 02:00:27.360Z, each `Building mode requested — interrupting for prompt upgrade` followed ~100 ms later by `Building-mode restart: guide suffix empty — continuing without prompt upgrade`. This predates Significant-Gravitas#14714 (merged 00:38Z 09-19), which touches 16 files and not `builder_context.py`; its rollout took the failure rate from 100% to 58%. ### What `build_builder_system_prompt_suffix` takes `force`, and the restart passes it, so the guide is applied from the fact that the enter tool just ran rather than from a history scan that cannot see it yet. When the suffix is still empty — which now means only that the guide failed to load — the relaunch no longer claims the guide is present. It says the guide could not be loaded, leaves `building_mode_requested` set so the next turn retries, and leaves `guide_in_system_prompt` False so the building-mode gates stay closed, which is correct: the guide really is absent. The ERROR line carries the full session id; the log prefix truncates it to 11 characters. ### How `_apply_building_mode_restart` called `build_builder_system_prompt_suffix(session)`, whose first branch returns `""` unless `session_entered_building_mode(session)` — a predicate derived from persisted message history and documented for "a *prior* turn". The restart calls it microseconds after the enter tool ran, before that tool call is in `session.messages`. `force=True` skips that branch for the one caller that already knows the answer; every other calle 10BC8 r is a turn-start assembly, where the history read is the right question. The failure path leaves `building_mode_requested` set, which would otherwise make `_ready_for_building_mode_restart` fire again at every message boundary for the rest of the turn, so the guard also reads a new turn-scoped `_RetryState.building_mode_restart_failed`. The relaunch itself still happens: the attempt has already been interrupted, so skipping it would end the turn mid-work. ### Open question Why the post-Significant-Gravitas#14714 rate is 58% rather than 0% or 100% is not established. Five restarts on the same image did build the suffix, and `BaseTool.execute` announces every dispatched tool into the in-flight buffer `session_entered_building_mode` reads, so the predicate should have answered True in all twelve. `force` removes the dependency on it either way, but what separates the two groups is unexplained and not guessed at here. ### Verified Executed: `copilot/sdk/building_mode_restart_test.py` and `copilot/builder_context_test.py` (33 passed); `copilot/tools/helpers_test.py`, `copilot/capabilities/dispatch_test.py` and `util/architecture_test.py` (90 passed, 1 deselected — `test_prepare_block_missing_credentials` hangs on clean dev on this machine); `blocks/test/test_block.py`; `ruff check` on the four touched files. Both new tests are mutation-proven. Dropping `force=True` turns `test_guide_applied_although_history_lacks_the_enter_call` red (1 failed / 12 passed); restoring the unconditional confirmation turns `test_empty_suffix_relaunches_without_the_confirmation` red (1 failed / 12 passed). The first runs the real suffix builder rather than a mock on purpose — patching it would have proved the wiring and never that the predicate underneath answers. Reasoned about, not executed: the restart against a live SDK turn on a deployed environment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
fix(frontend/copilot): make completing a Connect card actually connect ( Significant-Gravitas#14316) ### Why / What / How **Why:** This PR makes completing a Connect card in the copilot actually connect the integration and send the follow-up turn. Three separate defects — dropped OAuth scopes, a credentials provider that never learns a new credential exists in a connectors-only chain, and a connectors-only chain card with no completion path — combined so that clicking Connect, signing in, and seeing the "Connected via OAuth" toast left the row still reading "Connect" with no follow-up turn sent. Reported by Reinier in #breakage on 2026-09-03 with a screen recording; Toran noted it was the third bug of this shape in about two weeks. **What:** Fixes three defects on the connect path, and adds one test that drives the whole path end to end so the next drift is caught at the seam rather than in production. **How:** Any one of the three alone produces "nothing happens", which is why this reads as unfixable from the outside: 1. **The required scopes are dropped.** `ConnectorRow` → `ConnectCredentialDialog` → `useConnectCredentialDialog` → `useOAuthConnect` → `getV1InitiateOauthFlow(provider)` — no scopes. The endpoint accepts them (`router.py:135`) and the path this replaced passed them (`useCredentialsInput.ts:259`). The grant comes back without `repo`, `hasRequiredScopes` rejects the credential, and the row can never be satisfied — every retry drops them again. Now threaded from the card's schema down to the initiate call. 2. **Nothing tells the credentials provider a credential exists.** `invalidateConnectionQueries` invalidates a React Query key, and `CredentialsProvider` reloads off that query's cache events — but in a connectors-only chain nothing has that query mounted, so invalidation emits no event and the list is never re-read. `useOAuthConnect` now calls the provider's existing `reload()`, the same way `AyrshareConnectButton` already does. 3. **A connectors-only chain card has no completion path.** `ChainActionCard` renders Proceed only for inputs or questions, and `SetupRequirementsCard`'s auto-send gates on a store whose `markConnected` runs only from `markSent` — after a card has already sent. Such a card now sends once its credential is satisfied, reusing the existing `tryClaimAutoDismiss` claim so parallel cards cannot double-send. Introduced by [**feat(frontend): add copilot tool chain UI** - Significant-Gravitas#13773](Significant-Gravitas#13773) (`a6df3d3381`) behind the `new-tool-ui` flag, which defaulted off; shipped to everyone by [**refactor(frontend): drop the new-tool-ui flag and commit to the tool chain UX** - Significant-Gravitas#14164](Significant-Gravitas#14164) (`948b55ea81`), which dropped the flag. The standalone (non-chain) card still uses the scoped path, which is why this reads as "anymore". ### Changes 🏗️ - `useOAuthConnect`: accepts optional `scopes`, passes them to the initiate call, and reloads the credentials provider after the token exchange. Every other caller is unaffected — the argument is optional and a provider-level connect still asks for defaults. - `ConnectCredentialDialog` / `useConnectCredentialDialog`: forward `schema.credentials_scopes`. This also fixes the builder's "Add credential" flow, which had the same gap. - `SetupRequirementsCard`: a connectors-only card inside a chain auto-sends once its credential is complete. - New `ToolChain/__tests__/connect-card-flow.test.tsx`: three cases driving ToolChain → SetupRequirementsCard → ChainActionCard → ConnectorRow → ConnectCredentialDialog → useOAuthConnect against MSW. ### Verified I executed this locally against the real backend and GitHub client — confirming the authorize URL now carries the correct scope — and through the test suite, where each of the three fixes is pinned by a case that fails on the pre-fix code and passes after, with the surrounding copilot/credentials/onboarding regression suites staying green. The OAuth consent screen itself was reasoned about rather than executed, since completing it needs a human at GitHub's permission page; full command output, the live URL comparison, and the regression sweep are in the [evidence comment below](Significant-Gravitas#14316 (comment)). ### Why this keeps happening All three incidents break the *composition* of the connect chain while every individual link keeps passing its own tests, because every test on this path mocks the layer beneath it: `ToolChain.test.tsx` mocks `SetupRequirementsCard`, `ChainActionCard.test.tsx` mocks `ConnectCredentialDialog`, `ConnectCredentialDialog.test.tsx` stubs `useOAuthConnect`. The scope contract *was* pinned by a test — `CredentialsInput.test.tsx:217` — but on the path [Significant-Gravitas#13773](Significant-Gravitas#13773) replaced; the replacement inherited coverage of its own layer and none of the contract it took over. The 08-24 incident ([**fix(frontend): MCP OAuth popup fails with 'Sign-in window was closed' on COOP providers** - Significant-Gravitas#14141](Significant-Gravitas#14141), `popup.closed` lying under COOP) does **not** share that root cause: it is a browser-platform behaviour no Node test can reproduce, and it needs a Playwright case instead. That distinction, and the remaining gaps this PR does not close, are tracked in [**The copilot's connect chain has no end-to-end test, so each layer's contract drifts independently** - Significant-Gravitas#14315](Significant-Gravitas#14315) (OPEN-3471). ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] The chain's Connect button initiates OAuth carrying the block's required scopes - [x] A connected card flips to "Connected. Continuing…" and sends the follow-up turn - [x] A downgraded grant still reads as not-connected and sends nothing - [x] Every case fails on the pre-fix code for the right reason - [x] Surrounding copilot / credentials / onboarding suites stay green - [x] A finished card re-mounting from chat history sends nothing - [x] An API-key connect continues the chat too - [x] The sign-in offers the existing account for scope upgrade - [x] Live: the authorize URL carries `scope=repo` with the fix and `scope=` without it - [x] Manual run against a real GitHub consent screen — not done, see **Verified** --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Merge remote-tracking branch 'origin/dev'
Merge branch 'master' into dev # Conflicts: # autogpt_platform/backend/backend/copilot/graphiti/ingest.py
Merge branch 'master' into dev # Conflicts: # .github/workflows/platform-single-container-docker.yml
fix(backend/copilot): stop empty untitled dream sessions polluting th… …e chat list (Significant-Gravitas#13332) ## Why The dream pass (Significant-Gravitas#13243) creates a `ChatSession` per pass for its user-facing narrative. Three compounding issues made this user-visible noise: every nightly pass for a user with old facts but no new activity ran all three LLM phases, produced empty operations with an empty narrative, and still minted a session — which then leaked into the normal chat sidebar as an untitled chat containing only "Dream pass completed with no narrative output." Observed locally: one empty untitled chat per enabled user per night. ## What - Dream-kind sessions are hidden from the user chat list, pagination count, and `/search/global` (they remain fetchable by id for the admin Memory Visualizer). Surfacing dreams in the chat UI stays future work (P6). - `apply_operations` no longer creates a session or message when the pass produced zero operations. - Dream sessions that ARE created get a dated title ("Dream summary — YYYY-MM-DD"). - The sanitize prompt's empty-result example no longer teaches the model to return an empty `summary_for_user`; a new hard rule requires a non-empty summary. - Passes now skip before any LLM cost when the user has no episodes newer than their last completed dream (`dream:last_completed:{user_id}` Redis marker — single-key/cluster-safe, fail-open on Redis errors, stamped on the sync success path). ## How - The list filter is raw SQL with `metadata->>'kind' IS DISTINCT FROM 'dream'` — `IS DISTINCT FROM` is load-bearing: legacy sessions have no `kind` key, `->>'kind'` evaluates to SQL `NULL`, and a naive `<>` would hide every normal chat. The Python Prisma client's `JsonFilter` has no path access, hence raw SQL via the existing `query_raw_with_schema` helper; list/count share one WHERE fragment so pagination totals always match. Pinned by mock tests on the WHERE construction plus real-Postgres integration tests for the NULL semantics. - The empty-pass guard returns stats without a `session_id` key (both consumers read via `.get`, and the batch callback already narrows non-str to `None`), so no signature changes ripple into `batch_callbacks`. - The no-new-activity skip compares episode `valid_at` (with `created_at` fallback) against the marker; unparseable timestamps count as new (fail-open). The batch path doesn't stamp the marker yet — batch users simply never skip (fail-open, noted in code). ## Checklist - [x] Changes covered by tests (69 passing: mock + real-Postgres integration) - [x] `poetry run format` / lint / pyright clean - [x] Out-of-scope changes < 20% - [x] `data/*.py` untouched; `copilot/db.py` queries are user-id scoped 🤖 Generated with [Claude Code](https://claude.com/claude-code) Fixes SECRT-2480 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Krzysztof Czerwinski <kpczerwinski@gmail.com>
PreviousNext