Outbox: make the generic outbox usable for multiple consumers by thomasdarimont · Pull Request #54139 · keycloak/keycloak

thomasdarimont · 2026-10-19T20:21:11Z

Makes the generic transactional outbox (org.keycloak.events.outbox, OUTBOX_ENTRY) shareable, so that a second consumer such as outbound SCIM can use it next to SSF push delivery. It follows the feedback in #53786; a SCIM port to this branch runs on H2, PostgreSQL and MySQL with two nodes.

Background and usage: Outbox explainer and user guide.

What changes

One cluster lock per consumer. Each drainer locks under outbox-drainer:<entryKind>, so drainers of different kinds no longer exclude each other. Ticks run on their own executor instead of the shared timer thread.
Deliveries run outside transactions. A tick claims the due rows with a lease (CLAIM_TOKEN, CLAIMED_UNTIL). Per row, the handler's prepare runs in a short transaction, the delivery runs with no transaction and no bound session, and a record transaction applies the result only if the row still holds the lease. No connection is held during a destination call, and the rows of a crashed node become due again when the lease expires.
One tick per owner at a time. The claim skips owners that have a row in flight, and the lock is held for the longer of the interval and the claim lease, so per-owner order holds even when ticks overlap across nodes.
Admin operations win. Hold, dead-letter, requeue, kind migration and expiry clear the claim, so an in-flight delivery cannot overwrite them.
DEFER outcome. A handler can reschedule a row without counting an attempt; pendingMaxAge still bounds a defer loop.
Fairness, parallelism, tick budget (all opt-in). perOwnerBatchSize caps how many rows one owner takes per round. deliveryConcurrency delivers different owners in parallel and keeps one owner's rows in order. tickBudget hands unreached rows back to the next tick; the first row is always delivered.
Consumer API. Handlers and listeners see a read-only OutboxEntry (in server-spi-private). OutboxDrainerListener observes ticks and transitions, and composite(...) stacks several. OutboxDeliveryResult.withEffect(...) carries state that must commit with the transition. OutboxStore gains requeue, make-due, delayed enqueue, per-owner and per-subject listing, updateMetadata and markSuperseded.
Metrics. OutboxMetricsListener publishes keycloak.outbox.* meters per kind. SSF extends it instead of keeping its own series; the SSF binder's never-wired tick and depth members are removed.
Logging. Handler reasons and exception messages, which may contain a receiver's response text, are logged at DEBUG only.

Schema (`jpa-changelog-27.0.0.xml`)

New nullable columns CLAIM_TOKEN and CLAIMED_UNTIL.
CREATED_AT, NEXT_ATTEMPT_AT and DELIVERED_AT change from TIMESTAMP to BIGINT epoch milliseconds, like every other Keycloak table. TIMESTAMP has whole-second precision on MySQL and MariaDB, which left the order within a second undefined.
The back-fill task reads the old values as UTC and makes every queued row due immediately, so a time-zone offset cannot schedule rows into the future. It is skipped when the table does not exist (manual migration on a pre-26.7 database).

Covered in the 27.0.0 upgrading guide and release notes.

SSF configuration

Property	Default
`outbox-drainer-claim-lease`	5m
`outbox-drainer-tick-budget`	0 (unbounded); must be shorter than the lease, checked at startup
`outbox-drainer-per-receiver-batch-size`	0 (off)
`outbox-drainer-delivery-concurrency`	1

The defaults keep today's behaviour.

Decisions and limits

The handler contract changes without a deprecation path, because the SSF transmitter including the outbox is experimental.
Listener callbacks run after the transition has committed, each in its own transaction, so a failing listener can never undo a transition. State that must commit with the transition belongs in withEffect. A failing effect rolls the record back and the row is retried after its lease, so effects must be idempotent, like deliveries.
The owner fence does not serialize two claim transactions that run at the same instant. That requires a claim transaction to outlive the cluster lock, which is held at least for the claim lease. This is documented in the claim javadoc rather than solved with a second per-owner lock.
A few store queries the SCIM port still writes itself (distinct counts, newest row per destination, delete by ids) are left for a follow-up once the port has settled.

Testing

New OutboxDrainerTaskTests covers transitions, failing handlers, effects and listeners, budget, lease takeover, owner fairness and parallel delivery. OutboxStoreTests is extended for leases, requeue, listing and the subject operations. OutboxMetricsListenerTest and SsfOutboxMetricsListenerTest are new.
The existing SSF push, poll, stream management and event emitter suites pass.
Migration: the data conversion was verified on a Postgres dev database and, by @xgp, on MySQL 8.4 with 80 real rows from 26.7.5 and both the database and the JVM off UTC (no time-zone shift). MariaDB, Oracle and MSSQL are covered by the CI migration job only, which runs the DDL on an empty table.

Closes #54137

Co-Authored-By: Claude Fable 5.1

Copilot

🟡 Changes recommended

Unresolved transaction isolation, lifecycle fencing, delivery control, replay, metrics, and migration issues prevent approval.

7 open issues

What changed in this PR

Generalizes Keycloak’s transactional outbox so SSF and additional consumers can share delivery infrastructure.

Changes:

Adds per-kind scheduling, leased delivery, deferral, fairness, and parallel processing.
Introduces consumer interfaces, replay operations, and shared metrics.
Migrates timestamps to epoch milliseconds and expands test coverage.

File	Description
tests/base/src/test/java/org/keycloak/tests/events/outbox/OutboxStoreTests.java	Tests leases, replay, deferral, and queries.
tests/base/src/test/java/org/keycloak/tests/events/outbox/OutboxDrainerTaskTests.java	Tests delivery transitions and scheduling behavior.
ssf/transmitter/src/test/java/org/keycloak/ssf/transmitter/outbox/SsfOutboxMetricsListenerTest.java	Tests SSF outbox metrics.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/outbox/SsfPushDeliveryHandler.java	Splits preparation from delivery.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/outbox/SsfOutboxMetricsListener.java	Adds SSF-specific metrics integration.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/metrics/SsfMetricsBinder.java	Delegates generic outbox metrics.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/delivery/SecurityEventTokenDispatcher.java	Updates status Javadoc link.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/delivery/poll/PollRequest.java	Updates status Javadoc link.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/delivery/poll/PollDeliveryService.java	Imports relocated status enum.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/DefaultSsfTransmitterProviderFactory.java	Configures scheduling, delivery, and metrics.
ssf/transmitter/src/main/java/org/keycloak/ssf/transmitter/admin/SsfEventStatsRepresentation.java	Updates status Javadoc link.
ssf/tests/base/src/test/java/org/keycloak/tests/ssf/transmitter/SsfTransmitterStreamManagementTests.java	Imports relocated status enum.
ssf/services/src/main/java/org/keycloak/ssf/services/admin/SsfAdminResource.java	Imports relocated status enum.
server-spi-private/src/main/java/org/keycloak/events/outbox/OutboxEntryStatus.java	Relocates statuses and defines terminal states.
server-spi-private/src/main/java/org/keycloak/events/outbox/OutboxEntry.java	Defines read-only consumer view.
model/storage-private/src/main/java/org/keycloak/services/scheduled/ClusterAwareScheduledTaskRunner.java	Supports explicit cluster lock keys.
model/jpa/src/test/java/org/keycloak/events/outbox/OutboxMetricsListenerTest.java	Tests generic outbox metrics.
model/jpa/src/main/resources/META-INF/jpa-changelog-master.xml	Registers the migration.
model/jpa/src/main/resources/META-INF/jpa-changelog-27.0.0.xml	Adds leases and converts timestamps.
model/jpa/src/main/java/org/keycloak/models/jpa/entities/OutboxEntryEntity.java	Adds lease fields and consumer queries.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxStore.java	Implements claiming, deferral, replay, and listings.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxMetricsListener.java	Publishes shared outbox metrics.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerTickSummary.java	Exposes per-tick counters.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerTask.java	Implements phased, leased delivery.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerScheduler.java	Dispatches ticks to an executor.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerListener.java	Defines observation callbacks.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDeliveryResult.java	Adds deferred-delivery results.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDeliveryOutcome.java	Adds DEFER outcome.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDeliveryHandler.java	Defines preparation contract.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDelivery.java	Defines executable delivery phase.
model/jpa/src/main/java/org/keycloak/events/outbox/OutboxConfig.java	Adds lease, budget, and parallelism settings.
model/jpa/src/main/java/org/keycloak/connections/jpa/updater/liquibase/custom/JpaUpdate27_0_0_OutboxTimestampsToEpochMillis.java	Backfills epoch-millisecond timestamps.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Cross-node ordering, listener transaction isolation, and minimum tick progress remain unresolved.

3 open issues

7 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Owner fencing, interrupted-worker completion, and expiry-callback transaction isolation remain unresolved.

4 open issues

3 resolved since last review

_{Review effort: Balanced}

BD0

xgp · 2026-10-20T08:22:19Z

Thanks @thomasdarimont. I rebased the SCIM provisioning port onto this branch without conflicts: thomasdarimont#1254. Below is feedback from building a second consumer on it.

1. Cluster lock TTL vs. tick length. OutboxDrainerScheduler.schedule(session, task, intervalMillis) and SSF's createDrainerScheduledTaskRunner both use the drain interval as the ClusterAwareScheduledTaskRunner lock timeout (30s for SSF). A tick can run much longer than that: the default tickBudget is unbounded and the claim lease is 5m. Once the lock expires, a second node can start a tick for the same kind. Leases stop the two nodes from claiming the same rows. But findDueForDrain / findDueForOwner do not exclude an owner that already has a leased row, so the second node can claim and deliver that owner's newer rows before the first node finishes the older ones. Per-owner ordering then breaks across nodes. Suggestions:

tie the lock TTL to the claim lease (or to tickBudget plus a margin) instead of the interval, and/or
skip owners that have a live lease when claiming.

(A sub-second interval also truncates to a 0s lock, since intervalMillis / 1000 is used.)

2. Effects that must commit with the transition. A consumer often has state to write together with the outcome. SCIM writes the link (remote id and hash), the destination health, and its per-entry metadata. Today the only hook in the record transaction is OutboxDrainerListener. notify() swallows its failures, and a handler gets that one listener slot. So SCIM keeps a Map<rowId, Effect> from prepare/execute to the callbacks, and it can't also install OutboxMetricsListener. Two small changes would fix both:

let OutboxDeliveryResult (or OutboxDelivery) carry an optional Consumer<KeycloakSession> that the drainer runs in the record transaction, where a failure rolls the record back, and
add an OutboxDrainerListener.composite(...) helper, so a consumer can stack its own listener with the generic metrics one.

3. Store operations a consumer still has to write HQL for. The SCIM provider holds about ten statements on OutboxEntryEntity, because OutboxStore has no equivalent for these:

a subject's rows (ownerId + entryType), newest or pending first
updating one row's METADATA
settling a pending row as delivered without a call (it was superseded by a newer row for the same subject)
deleting a subject's other settled rows
COUNT(DISTINCT entryType) per owner and status, for status pages
the distinct (realmId, ownerId) pairs of a kind

Not all of these need to land in this PR. The first three in particular look generic, though: SSF could also coalesce per subject. Putting them on the store would keep consumers off the entity.

4. Upgrade docs. The issue lists a release-notes and upgrading-guide entry for the OUTBOX_ENTRY conversion. changes-27_0_0.adoc doesn't have it yet. It's worth noting there that the 27.0.0 migration makes every queued row due immediately.

5. Migration coverage. The description says the DDL ran on H2 and the data conversion on Postgres. MySQL/MariaDB (TIMESTAMP with a session time zone), MSSQL and Oracle are the riskier databases for the back-fill. A migration test that starts from 26.7 data on those would help.

6. Logging in prepareSafely / executeSafely. These log the exception, message and stack trace at WARN. The message of a handler exception can carry a receiver's response text, which the PR otherwise keeps at DEBUG. Consider logging only the exception class at WARN and the message at DEBUG.

With the SCIM port on top of this branch, the following pass: SCIM provisioning on dev-mem, PostgreSQL and MySQL, including two nodes; inbound SCIM; SSF tests/base; and the Outbox*Tests.

thomasdarimont · 2026-10-20T08:29:55Z

Thanks @xgp for your feedback! I think "1. Cluster lock TTL vs. tick length." was already adressed inbetween - you might want to update your branch. I'll take a look at 2. and 3. (for the most common operations). I'll also update the docs for 4.
Ad 5. I'll manually test the migration from a Keycloak 26.7 to for MariaDB / Oracle.
Ad 6. I'll change the log to use debug.

thomasdarimont · 2026-10-20T08:42:27Z

@xgp I adressed 2., 3., 4. and 6. now. You might want to rebase your SCIM branch again.

Here are the details:

Lock TTL: fixed since the second review round. OutboxDrainerRunner (which replaced the
ClusterAwareScheduledTaskRunner wrapper) holds the cluster lock for the longer of the interval and the
claim lease, at least one second, and findDueForDrain / findOwnersWithDueRows / findDueForOwner
skip every owner that has a row under a live lease (except rows the claiming tick holds itself, so the
owner-fair rounds still work). An owner is delivered by one tick at a time even when a tick outlives its
lock.
Effects: OutboxDeliveryResult.withEffect(Consumer<KeycloakSession>). The drainer runs it inside the
record transaction after the transition; if it throws, the record rolls back and the row is retried after
its lease, so effects must be idempotent like the delivery. OutboxDrainerListener.composite(...) stacks
listeners with per-delegate failure isolation. Listener callbacks themselves now run after the commit in
a transaction of their own, so they can no longer undo a transition (that was a Copilot finding).
Store: added listByOwnerAndType(kind, owner, type, status, newestFirst, first, max),
updateMetadata(kind, id, metadata) and markSuperseded(row, reason) (DELIVERED without an attempt,
clears a claim, rejects terminal rows). The other three (delete a subject's other settled rows, distinct
type counts, distinct realm/owner pairs) look SCIM-shaped to me; happy to add them in a follow-up once
the port shows their final form, or take a PR once this settled.
Docs: upgrading guide (27.0.0 notable changes) and release notes entries added, including that the
migration makes every queued row due immediately.
Migration: the CI migration job covers the DDL on postgres, mysql, mariadb, oracle and mssql from
24.0.4 (empty table). The back-fill I am verifying by hand on a 26.7 Postgres database and on seeded
MySQL and MariaDB containers before the PR leaves draft; a seeded 26.7 migration test would need the
Arquillian migration harness, which I would rather not extend in this PR. If you can run the migration
on one of your MySQL/MariaDB databases with real rows, the converted count in the log and a spot check
of created_at would be a welcome data point.
Logging: prepareSafely / executeSafely (and the record-failure path) now log the exception class
at WARN and the message with stack trace at DEBUG; the message still lands in last_error.

xgp · 2026-10-20T10:51:26Z

@thomasdarimont Thanks, I rebased the SCIM branch onto these changes and moved it to the new APIs (withEffect, the composite listener, OutboxDrainerRunner, and the new store operations). I also ran the migration on MySQL with real 26.7.5 rows, with both MySQL and the JVM off UTC. All 80 rows converted correctly with no time-zone shift. Details: thomasdarimont#1254 (comment)

xgp · 2026-11-08T13:18:50Z

@thomasdarimont Any update on where this sits in terms of getting it merged for a future version?

Porting a second consumer (outbound SCIM) to the generic outbox showed that it could not share the table with SSF push delivery: every OutboxDrainerTask locked on the same class-name key, a handler could not reschedule a row without burning an attempt, handlers ran inside the drainer transaction with a connection pinned during HTTP calls, delivery was serial per kind, one owner could fill every batch, ticks ran on the shared timer thread, and the TIMESTAMP columns had whole-second precision on MySQL and MariaDB. Key each drainer's cluster lock by entry kind and run ticks on an executor. Add a DEFER outcome that reschedules without counting an attempt. Rework a tick as claim, prepare, deliver, record: a claim transaction leases the due rows with a per-tick token and a lease end (CLAIM_TOKEN, CLAIMED_UNTIL), the handler's prepare phase returns an OutboxDelivery that executes outside any transaction, and the record is conditional on the token. A tick budget hands unprocessed rows back, an optional per-owner batch cap claims owner by owner, and optional delivery workers deliver different owners in parallel with one owner's rows kept in order. Expose a read-only OutboxEntry view in server-spi-private to handlers and listeners, add requeue, makeDueForOwner, listByOwner and delayed enqueue to OutboxStore, add OutboxDrainerListener with tick and per-transition callbacks, and a generic Micrometer OutboxMetricsListener that SSF extends instead of keeping its own series. Store OUTBOX_ENTRY timestamps as BIGINT epoch milliseconds like every other table, converted by a 27.0.0 changeset with a back-fill task. Log handler reasons at DEBUG only. Closes keycloak#54137 Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

Run ticks through a dedicated OutboxDrainerRunner that takes the cluster lock on a session without a transaction, run the tick callbacks in short transactions and bind no session during delivery. Require token, PENDING status and matching kind before preparing and recording, and clear the claim in every lifecycle bulk update so an in-flight delivery cannot overwrite an admin transition. Start the tick budget after the claim, count the tick thread as one of the delivery workers, reset created_at on requeue, expire stale queued rows one by one with a PENDING_MAX_AGE dead-letter cause, and skip the timestamp backfill when OUTBOX_ENTRY does not exist. Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

… commit Hold the cluster lock for the longer of the interval and the claim lease, and skip owners with a row in flight when claiming (except rows the claiming tick holds itself), so an owner is delivered by one tick at a time even when ticks overlap across nodes. Let the first row of a tick through the budget check so a tick always makes progress. Run row listener callbacks after the record transaction committed, in a transaction of their own, so a failing listener cannot undo a transition. Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

…le sentinel Wait for every delivery worker even when the tick thread is interrupted, so the tick never releases its lock while deliveries are still running. Run each expiry callback in its own transaction, as the listener contract promises. Use a non-empty no-token sentinel for the owner fence, since Oracle stores an empty string as NULL. Document what the owner fence does and does not serialize. Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

CodeQL flagged an implicit long-to-int narrowing in the interrupted worker test Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

Let a delivery result carry an effect the drainer runs inside the record transaction, so consumer state commits with the transition and a failing effect rolls the record back. Add a composite listener so a consumer can stack its own listener with the metrics one. Add the subject-level store operations a second consumer needed: list a subject's rows, update a row's metadata, settle a superseded row as delivered. Log handler exceptions by class at WARN and in full at DEBUG. Document the OUTBOX_ENTRY conversion in the upgrading guide and the release notes. Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

thomasdarimont · 2026-11-12T07:17:47Z

I rebased concurrent main and updated the PR description. I'll ask for another review pass in todays maintainers call.

The 1 failing test is unrelated to these changes and is due to general flakiness in the RefreshTokenTest.

keycloak-github-bot Bot added the team/core-iam label Oct 19, 2026

thomasdarimont requested a balanced review from Copilot October 19, 2026 20:28

Copilot started reviewing on behalf of thomasdarimont October 19, 2026 20:29

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

thomasdarimont requested a balanced review from Copilot October 19, 2026 21:15

Copilot started reviewing on behalf of thomasdarimont October 19, 2026 21:16

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Comment thread model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerRunner.java Resolved

Comment thread model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerTask.java Resolved

Comment thread model/jpa/src/main/java/org/keycloak/events/outbox/OutboxDrainerTask.java Resolved

xgp mentioned this pull request Oct 19, 2026

Add SCIM provisioning to external applications (experimental) thomasdarimont/keycloak#1254

Open

thomasdarimont requested a balanced review from Copilot October 19, 2026 21:56

Copilot started reviewing on behalf of thomasdarimont October 19, 2026 21:56

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

github-advanced-security Bot found potential problems Oct 20, 2026

View reviewed changes

Comment thread tests/base/src/test/java/org/keycloak/tests/events/outbox/OutboxDrainerTaskTests.java Fixed

xgp mentioned this pull request Oct 20, 2026

Outbox: make the generic outbox usable for multiple consumers #54137

Open

This was referenced Oct 25, 2026

SSF push outbox stalls when a receiver's error response cannot be stored in LAST_ERROR #54319

Closed

Store outbox errors that contain null characters or exceed the column in bytes #54320

Merged

thomasdarimont added 6 commits November 12, 2026 00:55

Outbox: widen a test counter to long

e3033a8

CodeQL flagged an implicit long-to-int narrowing in the interrupted worker test Signed-off-by: Thomas Darimont <thomas.darimont@googlemail.com>

thomasdarimont force-pushed the issue/gh-54137-outbox-enhancements branch from ea40366 to f298605 Compare November 12, 2026 01:25

Navigation Menu

Repository navigation

Outbox: make the generic outbox usable for multiple consumers - #54139

Outbox: make the generic outbox usable for multiple consumers#54139
thomasdarimont wants to merge 6 commits into
keycloak:mainfrom
thomasdarimont:issue/gh-54137-outbox-enhancements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Uh oh!

What changes

Schema (jpa-changelog-27.0.0.xml)

SSF configuration

Decisions and limits

Testing

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Schema (`jpa-changelog-27.0.0.xml`)