Generic outbox (`org.keycloak.events.outbox`): gaps for non-SSF consumers such as outbound SCIM #53786

xgp · 2026-10-09T09:10:41Z

xgp
Oct 9, 2026

@thomasdarimont, following up on our conversation at KeyConf26 yesterday about using the outbox you built for SSF for other kinds of guaranteed delivery.

I maintain an outbound SCIM provisioning extension (pushing users and groups from Keycloak to SaaS apps) that I'd like to contribute upstream. It had its own outbox; I've moved its storage onto OutboxStore and OUTBOX_ENTRY (one row per change, its own entry kind), and it now runs alongside SSF's drainer with neither touching the other's rows. To use OutboxDrainerTask itself as well, a second consumer needs the following. I'm happy to send PRs for any of them.

Cluster lock key collision (bug). ClusterAwareScheduledTaskRunner keys its lock by task.getClass().getSimpleName(), so every OutboxDrainerTask in a cluster, SSF's included, shares the key "OutboxDrainerTask". A second consumer scheduled the same way would make one of the two skip its ticks. Proposal: derive the key from the entryKind, for example outbox:<entryKind>.
Retry without spending an attempt. RETRY always increments ATTEMPTS; SsfPushDeliveryHandler has a comment asking for exactly this. Proposal: OutboxDeliveryResult.defer(Instant notBefore, String reason), which reschedules without counting an attempt. That covers a receiver that's temporarily down, rate limiting, and waiting on a dependency.
Owner-scoped draining. One drainer per kind delivers rows in sequence, so one slow receiver delays every other receiver of that kind. Proposal: lockDueForDrain(kind, ownerId, limit), which respects NEXT_ATTEMPT_AT (IDX_OUTBOX_OWNER already covers it), and an option for a drainer to work per owner.
Delivery inside one transaction. A batch is locked and every handler call runs in a single transaction with no time limit. Slow receivers can exceed the transaction timeout and roll back a whole batch, and HTTP calls hold row locks. Proposal: claim the rows, deliver outside the transaction, then record the transition per row, plus a per-tick time budget.
Smaller API additions:
- re-arm an existing row (DEAD_LETTER or DELIVERED back to PENDING), which doubles as a replay API;
- list rows by owner and status, paged;
- enqueue with a not-before time;
- a callback when the drainer dead-letters a row, so handlers can record metrics.

Items 1 and 2 seem small and would help SSF as well. Would you take PRs for those first? I'd also like your view on 3 and 4, and on whether the outbox should become a documented SPI for other consumers.

cc @ssilvert @pedroigor

thomasdarimont · 2026-10-09T13:51:19Z

thomasdarimont
Oct 9, 2026
Collaborator

Thanks @xgp, all five points are addressed on my branch and will go up as a small PR series.

Please give this branch a try with your SCIM implementation: https://github.com/thomasdarimont/keycloak/tree/poc/outbox-improvements

Here is what I changed:

Introduced lock key per entry kind. ClusterAwareScheduledTaskRunner takes an explicit lock key, and OutboxDrainerTask.getTaskName() returns outbox-drainer:, used for both the cluster lock and the timer name.
Allow to mark an entry as deferred without spending an attempt. OutboxDeliveryResult.defer(Instant|Duration, reason) reschedules the row without counting an attempt. Past instants are clamped to now, and pendingMaxAge still bounds an endless defer loop.
"Owner-fair" draining. OutboxConfig.perOwnerBatchSize switches the claim to owner-fair mode: owners with due rows are visited longest-waiting first, capped per owner per round, going round again until the batch is full. There is no separate drainer per owner; together with the tick budget from point 4 it should cover the slow-receiver case without extra timers or lock keys.
Delivery outside the batch transaction. A claim transaction leases the due rows with a per-tick token and pushes next_attempt_at out by the lease. Each row is then delivered and recorded in transactions of its own, with the record conditional on the token, so a tick whose lease expired cannot overwrite a newer attempt. A tick budget hands unprocessed claimed rows back. Crashed nodes need no reclaim sweep since the lease expiry makes their rows due again.
Smaller additions. requeue(kind, ids) and requeueDeadLetterForOwner re-arm DEAD_LETTER and DELIVERED rows in place, which doubles as replay. listByOwner pages by owner and optional status. enqueuePending takes a not-beforev instant. OutboxDrainerListener has no-op defaults for tick start and end and for every transition, including dead letter with its cause. On top of it, OutboxMetricsListener publishes keycloak.outbox.* meters labeled by kind (tick rate and duration, last tick, transitions, dead letters, depth), so a consumer gets operational metrics without writing its own.

On your SPI question: handlers and listeners now receive a read-only OutboxEntry from server-spi-private instead of the JPA entity, and OutboxEntryStatus moved next to it.
An "out-of-tree" consumer therefore no longer needs model/jpa on its compile classpath for the delivery side. The store itself still lives in model/jpa and is what in-tree consumers use to enqueue; whether that becomes a documented SPI as well is a separate decision for the team, and I would keep it out of this series.

For your SCIM provider the relevant knobs are IMHO the per-owner batch cap, the tick budget, and defer for rate-limited targets.

Please give it a try.
Happy to take a look at your handler once the PRs are in.

8000

4 replies

thomasdarimont Oct 10, 2026
Collaborator

I created an explainer and a user guide that includes the OutboxStore with the suggested improvements from the branch
https://gist.github.com/thomasdarimont/9748cf198eaac9c038939a7ed1d73a4c

xgp Oct 10, 2026
Author

Thanks @thomasdarimont, this works. I ported scim-outbox to your branch and removed our own drainer entirely. It now runs one OutboxDrainerTask for kind scim-outbox with an OutboxDeliveryHandler, using the owner-fair claim, the tick budget, and defer for both destination backoff and groups waiting for their members. requeue replaced our replay, and the listener callbacks let us commit our own row metadata atomically with your transitions. Our full integration suite (208 tests) passes on an image built from your branch on H2, PostgreSQL and MySQL; on PostgreSQL and MySQL that includes two nodes sharing one cluster lock, restarts, and SSF running alongside.

A few things came up while porting, roughly in order of importance:

OUTBOX_ENTRY timestamps have whole-second precision on MySQL/MariaDB. The 26.7.0 changeset declares CREATED_AT, NEXT_ATTEMPT_AT and DELIVERED_AT as plain TIMESTAMP, so rows written in the same second tie, and ORDER BY createdAt / nextAttemptAt falls back to the random id. That bit us twice, on MySQL only: deciding "is there a newer row?" from createdAt dropped changes, and reading a subject's latest outcome as its newest row returned a stale delivered row instead of a dead letter written in the same second. We now compare only against pending rows, and keep at most one settled row per subject by pruning the others in your record transaction, so nothing depends on creation-time order. DATETIME(6) / TIMESTAMP(6) there, or a sequence column, would make arrival order reliable for every consumer.
The handler runs inside a transaction. deliverInOwnTransaction reads the row and calls the handler in the same transaction. A pooled connection therefore stays checked out, and a JTA transaction stays open, for the whole HTTP call. No lock is held, but with a slow receiver and a large batch it pins connections and runs into the transaction timeout. Could the row be read in its own short transaction, and the handler called with a session but no open transaction?
One delivery thread per kind, cluster-wide. Owner fairness fixes who gets the batch, not throughput: every destination is delivered serially, and in our two-node test the node holding outbox-drainer:scim-outbox kept it for the whole backlog. We can live with that, since a backed-off destination is deferred without a call. Optional parallelism across owners within a tick would bring back what we had with a lock per destination.
SSF's drainer runs on Keycloak's timer thread. BasicTimerProvider is a single java.util.Timer, so a slow SSF receiver delays every other scheduled task on the node. We run the task on our own executor through ClusterAwareScheduledTaskRunner. It may be worth SSF doing the same, or the drainer offering it.
The dead-letter reason is logged at WARN. applyResult logs result.errorMessage() for DEAD_LETTER. Our rule is that a receiver's response text never goes above DEBUG. So we return only the error class and status as the reason, and write the full text to the row from onDeadLetter. Logging the reason at DEBUG, or a separate log text, would remove that workaround.
Making an owner's deferred rows due again. When an admin fixes a destination (rotated token, re-enabled), we pull its deferred rows forward with our own HQL on claimToken IS NULL. Something like OutboxStore.makeDue(kind, ownerId) would keep consumers off the entity.

Smaller points: "claim lost before delivery" is logged at INFO. nextAttemptAt now doubles as the lease, so admin views show the lease end for rows in flight.

As far as using this when contributing the outbound SCIM feature to core, do you think it makes sense to do the whole thing as one PR, or to wait until your outbox changes are committed, and then PR against that?

thomasdarimont Oct 10, 2026
Collaborator

Thanks for your report / suggestions, and for porting the full suite in your scim implementation.

I just updated the branch again. All six points were valid; they are addressed on the branch now:

On Timestamps. The three outbox columns turned out to be the only TIMESTAMP columns in Keycloak, everything else is BIGINT epoch millis. They are now BIGINT as well, converted by a migration in the 27.0.0 changelog, so arrival order is reliable on every database. The ORDER BY of the due queries also got created_at, id as tie-breakers.
Regarding the handler inside a transaction. The handler contract is two-phase now: prepare(session, row) runs in a short transaction and returns an OutboxDelivery; the drainer runs that outside any transaction, with no session bound to the thread. Settled results (OutboxDelivery.settled(...)) cover the orphaned and deferred cases you decide in prepare. No connection is held during a call any more.
One delivery thread per kind. OutboxConfig.deliveryConcurrency allows to spread the claimed rows over workers of the outbox-delivery- executor, grouped by owner: one owner's rows stay in order on one worker, different owners go in parallel. Sizeable per kind with spi-executors-default-outbox-delivery--max. The cluster-wide lock per kind stays, so this parallelises within a tick, not across nodes.
Timer thread. OutboxDrainerScheduler.schedule(session, task, interval) lets the timer only trigger and runs the tick on the outbox-drainer executor, with the
cluster-aware runner keyed by the task name and a same-node overlap guard. SSF uses it now; With this you can drop your own executor wiring.
Log levels. WARN carries id, kind, correlation id and cause only; the handler reason goes to DEBUG and to last_error. Claim-lost moved to DEBUG as well.
OutboxStore.makeDueForOwner(kind, ownerId) pulls an owner's deferred PENDING rows forward, leaving rows in flight to their tick.

On the lease: you were right that overloading nextAttemptAt muddles admin views. The lease end now lives in its own CLAIMED_UNTIL column; nextAttemptAt only ever means "when the row became due", and in-flight rows are visible as token plus lease in the future. The claim changeset also moved out of the released 26.8.0 changelog into a new 27.0.0 one, which it should have been in from the start.

Porting note for your handler: deliver(session, row) becomes prepare(session, row) returning the delivery closure; resolve your target and credentials in prepare, capture plain values, and make the call in the closure. The listener callbacks you use for your own row metadata still run inside the record transaction, on a worker thread when concurrency is above one.

For the SCIM contribution I would suggest a draft PR against my top branch once this settles, so the reviewers of the outbox series see the second consumer, and a rebase onto main once the outbox PRs are in. Please do not carry a copy of the outbox changes in the SCIM PR.

I updated the explainer and user guide according to the latest changes here: https://gist.github.com/thomasdarimont/9748cf198eaac9c038939a7ed1d73a4c

xgp Oct 10, 2026
Author

Thanks @thomasdarimont, that was quick, and it all landed well. I ported scim-outbox to the updated branch (11e75b9):

Two-phase handler. prepare reads the destination, its health and the subject, resolves the token and computes the plan. The delivery does only HTTP on the values it captured, with no session. Every write an outcome causes (link, health backoff, error class, deferral count, the forced flag a superseded row hands on) now runs in your record transaction from the listener callbacks, so it commits together with the row's transition. Before, those writes were a separate transaction after the call. So the change actually made our side simpler and more correct, not just cheaper on connections.
deliveryConcurrency. On, default 4. Keeping one owner's rows in order on one worker is exactly what we need, because our destination backoff relies on it.
Epoch-millis timestamps removed our MySQL tie problems. We kept pruning settled rows per subject anyway, since our kind has no retention.
Dead-letter reason now at DEBUG. The deadLetterText workaround is gone. We return the stored text (≤ 300 chars) as the reason.
makeDueForOwner replaced our own HQL.

Our full suite (209 tests, now including one where a slow receiver must not hold back another in the same tick) passes on an image built from the branch, on H2, PostgreSQL and MySQL, with two nodes, restarts and SSF alongside.

Three things from this round, none of them blocking:

Writes that must commit with the transition depend on listener callbacks, and those swallow exceptions. notify() logs and ignores a callback failure, which is right for observers. For us, though, the callbacks are where the outcome's writes happen. If one fails after a database error, the record transaction is usually rollback-only and the row comes back after the lease, which is fine. If it fails before touching the database, the transition commits without our writes. We also have to carry each outcome from execute to the callback in a map keyed by row id. A way for the result to carry its own record step would fix both: for example, OutboxDeliveryResult with a Consumer<KeycloakSession> that the drainer runs in the record transaction before the transition, where a throw rolls the record back. Then the listener could stay a pure observer.
We kept our own trigger instead of OutboxDrainerScheduler. We start a tick right after a commit that enqueued rows, and start another at once when a tick filled its batch or ran out of budget, so latency is one delivery rather than one interval. schedule only covers the periodic trigger, and its same-node guard is private, so a wake from outside could still overlap with it. The cluster lock would catch that, but the wake would then be skipped. If schedule returned a handle with a wake() that shares the guard (and re-runs once if a trigger came in during a tick), we could drop our executor wiring as you suggested.
prepare can read a cached model from before the commit, which may be worth a line in the guide. The user cache enlists its invalidations with enlistAfterCompletion, so they run after the database commit. A drainer woken by that commit, or already mid-tick, can claim the new row and read the user through session.users() before the cache has been invalidated. On MySQL we saw a department change planned from the old state, judged "unchanged" and settled without being sent. We now read the subject through StoreManagers.userStorageManager(), the same delegate the cache loads from. SSF doesn't hit this because its payload is captured at enqueue. Any kind that reads current state in prepare does.

On the contribution: understood. Once the outbox series settles, I'll open a draft PR against your top branch with only the SCIM provider, no copy of the outbox changes, and rebase onto main once your PRs are in.

xgp · 2026-10-11T06:49:12Z

xgp
Oct 11, 2026
Author

@thomasdarimont SCIM outbox PR against your branch thomasdarimont#1253

0 replies

Navigation Menu

Repository navigation

Generic outbox (`org.keycloak.events.outbox`): gaps for non-SSF consumers such as outbound SCIM #53786

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Generic outbox (org.keycloak.events.outbox): gaps for non-SSF consumers such as outbound SCIM #53786

Uh oh!

xgp Oct 9, 2026

Replies: 2 comments · 4 replies

Uh oh!

thomasdarimont Oct 9, 2026 Collaborator

Uh oh!

thomasdarimont Oct 10, 2026 Collaborator

Uh oh!

xgp Oct 10, 2026 Author

Uh oh!

thomasdarimont Oct 10, 2026 Collaborator

Uh oh!

xgp Oct 10, 2026 Author

Uh oh!

xgp Oct 11, 2026 Author

Generic outbox (`org.keycloak.events.outbox`): gaps for non-SSF consumers such as outbound SCIM #53786

xgp
Oct 9, 2026

Replies: 2 comments 4 replies

thomasdarimont
Oct 9, 2026
Collaborator

thomasdarimont Oct 10, 2026
Collaborator

xgp Oct 10, 2026
Author

thomasdarimont Oct 10, 2026
Collaborator

xgp Oct 10, 2026
Author

xgp
Oct 11, 2026
Author