8000 Restore potentially lost entries of revoked items into the cache by ahus1 · Pull Request #54353 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

Restore potentially lost entries of revoked items into the cache - #54353

Open
ahus1 wants to merge 7 commits into
keycloak:mainfrom
ahus1:is-54347-reload-revoked-tokens
Open

ahus1 wants to merge 7 commits into
keycloak:mainfrom
ahus1:is-54347-reload-revoked-tokens

Conversation

@ahus1
@ahus1 ahus1 commented Oct 26, 2026 •
Copy link
Copy Markdown
Member

Closes #54347

This is a best-effort defense-in-depth measure, not a guarantee.

Rationale

Instead of adding a database lookup to every cache miss on the token-validation path, the cache is reconciled with the database when the cluster membership changes.
Token validation stays cache-only and fast, and the cost is paid only in the rare event of a node leaving or a partition merging.

Decisions

  • Only the coordinator scans the database, as writes to the distributed cache restore the entries for the whole cluster.
  • Nodes joining the cluster don't trigger a reload, so scaling up doesn't scan the database.
  • Existing entries are never overwritten, so the lifespan of entries written by other nodes isn't restarted.
  • No retry: if the reload fails, a warning is logged and the LOADED marker is removed, so the next node that starts up loads the persisted revocations again. Nodes that are already running keep their state until then.

Assumptions

  • Entries are lost when the owning node leaves, not while the cache rebalances, so the reload doesn't need to wait for the rebalancing to finish.
  • The database stays available during the reload. If it doesn't, the best-effort fallback above applies.

Closes keycloak#54347

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Closes keycloak#54347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery can stop before rebalancing completes, and the regression test has unresolved synchronization and timeout issues.

3 open issues
What changed in this PR

This PR addresses #54347 by restoring persisted, unexpired token revocations that may be lost from the Infinispan cache during cluster membership changes.

Changes:

  • Adds asynchronous, coordinator-only recovery after member departures and cluster merges.
  • Updates JPA revocation queries to project results directly and use long-valued timestamps.
  • Adds a split-brain regression test for restoring lost revocations after a merge.
File Description
testsuite/​model/​src/​test/​java/​org/​keycloak/​testsuite/​model/​infinispan/​EmbeddedInfinispanSplitBrainTest.java Tests restoration of persisted revocations after a cluster merge.
model/​jpa/​src/​main/​java/​org/​keycloak/​models/​jpa/​session/​JpaRevokedTokensPersisterProvider.java Refines revocation queries and expiration timestamp handling.
model/​infinispan/​src/​main/​java/​org/​keycloak/​models/​sessions/​infinispan/​InfinispanSingleUseObjectProviderFactory.java Adds membership listeners and background recovery with retries.

Review effort: Balanced

ahus1 and others added 2 commits October 27, 2026 10:20
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@gmx.net>
Closes keycloak#54347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
@ahus1
ahus1 force-pushed the is-54347-reload-revoked-tokens branch from b25b4fe to 11ffa3f Compare October 27, 2026 09:49
@ahus1
ahus1 requested a balanced review from Copilot October 27, 2026 09:49
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive asynchronous recovery needs human validation, and correctness fixes remain unresolved.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Guard null cache during failed factory initialization cleanup

model/​infinispan/​src/​main/​java/​org/​keycloak/​models/​sessions/​infinispan/​InfinispanSingleUseObjectProviderFactory.java:149

Factory cleanup also runs after initialization fails, when persistRevokedTokens can be true but postInit() has not assigned singleUseObjectCache (see KeycloakModelTest.createKeycloakSessionFactory()). This dereference then throws a NullPointerException, masking the startup failure and aborting remaining provider cleanup; guard against a null cache before removing the listener.

Medium severity Use current time per record during token recovery

model/​infinispan/​src/​main/​java/​org/​keycloak/​models/​sessions/​infinispan/​InfinispanSingleUseObjectProviderFactory.java:199

On a long reload, this fixed timestamp makes later entries live past their persisted expiry and can reinsert tokens that expired while the scan was running. Compute the current time per record, as before, so recovery skips expired rows and does not extend cache retention by the scan duration.

3 resolved since last review

Review effort: Balanced

…voked token record

Closes keycloak#54347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

Review effort: Lite

…lling

Closes keycloak#54347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive revocation recovery relies on interactions between Infinispan membership notifications and cache state transfer, requiring clustered runtime validation of merge recovery, owner departure, and reload-failure fallback.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

@ahus1
ahus1 commented Oct 28, 2026 •
Copy link
Copy Markdown
Member Author

Security review follow-up using Codex:

I reviewed the residual behavior of this best-effort mitigation against the redesign tracked in #54373.

There is no need to fail all token introspection requests closed while a reload is running. With the action token cache configured with two owners, an ordinary rolling update removes one owner at a time and the surviving owner retains the revocation entry. The residual replay window requires concurrent loss of both owners, or equivalent exceptional cache loss, and lasts until the asynchronous reload restores the entry. A reload failure can extend that window until another node starts.

Those residual architectural limitations are covered by #54373:

  • each node builds its node-local revocation state from the database before it is marked ready and accepts dependent requests, with targeted database confirmation as the alternative during startup;
  • later rebuilds keep the existing local state active, incorporate concurrent revocation events, and replace or augment it without globally blocking introspection;
  • node loss no longer loses revocations through distributed-cache ownership;
  • one-time action tokens and SAML assertions use a database-arbitrated consume-once operation;
  • failed or missed rebuilds are recovered by subsequent triggers and hourly 10000 reconciliation; and
  • concurrent rebuild triggers are coalesced so topology churn does not queue repeated full scans.

Conclusion: I do not see an additional merge-blocking security finding in this PR within its explicitly documented best-effort scope. The remaining replay exposure and recovery limitations require the redesign already tracked by #54373.

@ahus1
ahus1 marked this pull request as ready for review October 29, 2026 09:23
@ahus1
ahus1 requested review from a team as code owners October 29, 2026 09:23
@ahus1
ahus1 requested a review from a team as a code owner October 29, 2026 09:23
@ahus1
ahus1 enabled auto-merge (squash) October 29, 2026 09:23
@ahus1
ahus1 requested a review from ruchikajha95 October 29, 2026 09:23
@ahus1 ahus1 changed the title Restore potentially lost expired entries into the cache Restore potentially lost entries of revoked or used items into the cache Nov 3, 2026
@ahus1 ahus1 changed the title Restore potentially lost entries of revoked or used items into the cache Restore potentially lost entries of revoked items into the cache Nov 3, 2026
Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 16:23
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cluster recovery across failures and membership changes warrants human assessment beyond the published review.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache-Only Revoked-Token Validation might fail if cache entries were lost

2 participants

0