Repository navigation
New Infinispan node joining the cluster becomes a command target/coordinator before its cache components are ready - ISPN000136 failures during Keycloak rolling restarts #52088
Description
Before reporting an issue
- I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
infinispan
Describe the bug
Running Keycloak 26.6.4 (Infinispan 16.0.8), embedded caching, cache-stack=jdbc-ping, on Kubernetes with a canary rolling-update strategy (4 replicas). During every rolling deploy, new pods join the JGroups/Infinispan cluster at JVM startup - independent of and before Kubernetes readiness. Once joined, Infinispan begins reassigning ownership of clustered cache segments (actionTokens, authenticationSessions) to the new members while state transfer is still in progress. During this window we see real ISPN000136 command failures (PutKeyValueCommand/GetKeyValueCommand) logged by the new pods themselves, e.g.:
ISPN000136: Error executing command PutKeyValueCommand on Cache 'actionTokens', writing keys [refreshLock::]
Some instances carry org.infinispan.commons.TimeoutException at CommandAckCollector$BaseCollector.call(), suggesting the new pod may itself be coordinating the write and timing out waiting for an ack from another (possibly also still-syncing) owner - though we're not fully certain of the exact RPC direction, which is part of why we're asking here.
We confirmed this is not a Kubernetes-readiness problem. We tried delaying the readiness probe (initialDelaySeconds) so new pods wouldn't receive external HTTP traffic until well after our observed rebalance window, and the failures reproduced identically. Cluster membership and cache-topology reassignment happen at the Infinispan/JGroups level regardless of the Kubernetes probe state, so gating HTTP traffic to the new pod doesn't touch the actual mechanism.
We'd appreciate help understanding: (a) whether a newly-joined node is expected to become a command target/coordinator for these caches before its own cache components have completed startup, and (b) whether there's a supported way to delay a node's inclusion in the consistent hash until it's actually able to serve as an owner - analogous to how 26.7.0 now blocks shutdown until topology is stable (#44620), but for the join side.
Version
26.6.4
Regression
- The issue is a regression
Expected behavior
A node joining the cluster should not become a target for cache commands on segments it hasn't yet received, and/or in-flight commands touching a segment mid-transfer should not fail/timeout - regardless of Kubernetes readiness state.
Actual behavior
New pods produce real ISPN000136 write/read failures in the 1-3 minutes after joining, visible to end users as failed token refresh / broken login.
How to Reproduce?
- Run Keycloak with KC_CACHE=ispn, KC_CACHE_STACK=jdbc-ping, 4+ replicas, embedded caching, behind a Kubernetes rolling/canary deploy.
- Trigger a rolling update replacing all replicas.
- Observe logs during rollout: ISPN000136 errors on actionTokens/authenticationSessions correlated with pods that just joined.
- We ruled out readiness-probe timing as a factor - delaying initialDelaySeconds well past the observed rebalance window did not prevent the failures.
- In our environment, a single rollout produced 500+ such failures over roughly 60-100 seconds, reproduced across three separate rollouts over several days.
Anything else?
Related/prior art:
- Possible data loss when scaling Keycloak StatefulSet down (on Kubernetes) #44620 - the shutdown-side equivalent, fixed in 26.7.0.
- Infinispan rebalancing during initial bootstrap of Keycloak gives false sense of readiness codecentric/helm-charts#363 - an independent report of what looks like the same startup-side race, closed stale with no fix.
Activity
krzysztofpozoga commentedon Sep 2, 2026
I want to follow up on this issue:
Every rolling deployment (and separately any Kubernetes-initiated node replacement that recycles a pod outside of a deploy) has a window where surviving/new pods return real, client-facing 500s on
POST /token. We've traced this back across multiple occurrences over several months, most recently ~425 real errors (84% on/token) in a single deployment. We are using rolling deployment strategy with 4 replicas,numOwnersat default (floored to 2 for the affected caches) withpersistent-user-sessionsfeature enabled but perCacheConfigurator, this does not coveractionTokens/authenticationSessions/loginFailures.Observed log signatures, always together:
ISPN000427: Timeout after 15 seconds waiting for acks — the stockremote-timeoutdefaultISPN000136: cache write/read failure onactionTokens/authenticationSessionsISPN000208: No live owners found for segments ... of cacheauthenticationSessions— during the departing-pod windowISPN000299: Unable to acquire lock after 10 seconds for keyrefreshLock:...— downstream of the aboveTraced the exact code path for the
ISPN000427case:RefreshTokenGrantType.process→TokenManager.refreshAccessToken→createTemporaryExclusiveLockForTokenRefreshOperation→InfinispanSingleUseObjectProvider.putIfAbsentonactionTokens.What we've already tried in our own deployment (all still in place, none fully close the gap alone)
over JGroups — a control plane entirely separate from the Kubernetes readiness gate.
So none of the above touch the actual ack-wait: the stock
remote-timeout="15000"on these three caches is tight enough that, during active rebalancing right around pod departure, real writes still exceed it.The two paths we're considering and we would like your input before doing anything:
1. Tune
remote-timeout(and add a state-transfer timeout) foractionTokens/authenticationSessions/loginFailuresvia a customcache-ispn.xmlWe can replicate what
CacheConfigurator.getCacheConfiguration()already sets programmatically (DIST_SYNCmode,APPLICATION_OBJECT_TYPEencoding) so we're not silently changing anything else — but we're hesitant to ship a hand-maintained cache topology definition that has to keep tracking Keycloak's internal defaults release over release, and we can't validate it under real production-scale load in a lower environment.remote-timeoutfor these three specific caches something that is considered safe and intended for this use case (rolling Kubernetes deployments), or does it risk assumptions made elsewhere in the codebase?2. Externalize these caches to a remote/external Infinispan cluster instead of embedded, accepting the operational cost, so cache membership is decoupled from Keycloak pod lifecycle entirely
/loginFailures(all in-memory-only, never DB-backed even withpersistent-user-sessions`)?ahus1 commentedon Sep 3, 2026
Sorry to hear this is causing you problems, and thanks for reaching out.
The recommendation is to use a stateful set for deploying Keycloak, which will enforce the one-pod-rollout-at-a-time. This also ensures that no data is lost, as the caches for authentication sessions have two owners.
Q: Can you please check the startup log of your nodes - does it state that virtual threads are enabled? Please share the startup-log of your node here.
Q: As you're mentioning the
loginFailurescache - how many entries does your cache have, and how much time does the rebalancing take? Do you have persistent user sessions enabled? Do you use offline sessions?In general: With 26.6, once a node joins, it is part of Infinispan read/write requests. While the rebalancing is in progress, a node will target both the old node and the new node for fetching data, so there should be no error shown to the user.
Q: You reported errors in the log. To double check: Did you see also user facing errors (like 500 errors in the HTTP server log, or prolonged response times)?
The graceful shutdown in any Keycloak release depends on Keycloak receiving the signal. If you build your custom containers, with possibly your own entrypoint, see https://www.keycloak.org/server/containers#custom-entrypoint-shell-scripts for how to write them in a way that Keycloak receives the signals.
Q: To verify the clean shutdown, please share the shutdown log here.
Now to your other questions how we think about the future here:
statelesspreview feature. In a nutshell, it stores the information from distributed caches in the database, and with that, any login call will only touch the database, and never any neighbor node. I don't yet recommend it for production use (although it should work OK with PostgreSQL). We plan to make it supported in 26.8.0 (this is ongoing). See https://www.keycloak.org/2026/07/multi-cluster-v2-and-stateless-mode for the blog post, and follow Stateless mode: make it a supported feature #50642 for the road to making this supported.pruivo commentedon Sep 3, 2026
@corujito, what does this mean? Are you using the Keycloak Operator or other 3rd party tool? If it is deployed as a
Deploymentinstead of aStatefulSet, it explains the errors.Please share your configuration and the logs from all Pods during the event. Thank you.
corujito commentedon Sep 4, 2026
@ahus1 @pruivo
Thanks for helping on this.
I shared shutdown/startup logs via email that probably answers most of the questions.
Virtual threads support: disabled
persistent user sessions enabled
rebalance happens in milliseconds
Not a StatefulSet - we use an Argo Rollouts canary with dynamicStableScale: true, which scales the old ReplicaSet down as the new one scales up. With 4 replicas, in practice this increments one pod at a time, but it's not strictly serialized like a StatefulSet's OrderedReady.
ahus1 commentedon Sep 6, 2026
Thank you for the log files. I did some drilling with the help of AI and this is what I think is happening - let's confirm with @pruivo next week:
For (1), I'll submit a fix that we'll backport for 26.7, but there are no community releases for 26.6 any more. You'll benefit from it once a patch release is available and you upgrade. While this is not available, the next paragraph should provide you a workaround.
For (2), I'll submit a fix with a default of 3 seconds remote timeout. As you suggested above, you can override this setting via the XML file. Once you upgrade, this override should no longer be necessary.
For (3), I assume you're might be using TLS Passthrough? Or your loadbalancer takes a bit longer than 1 second to take the Keycloak Pod out of rotation? I suggest to enlarge
shutdown-delayto 10 or 30 seconds (see https://www.keycloak.org/server/reverseproxy.html#_when_to_adjust_shutdown_timeouts) to allow for a cleaner shutdown.The message ISPN000312 seems to be noise in the case of Keycloak's distributed caches with two owners, I'll see that we'll suppress it.
1 remaining item
More graceful leaving of distributed caches
ahus1 commentedon Sep 8, 2026
@corujito - the change is now in main. I don't think the change will make it into the next 26.7 patch release, but the one after.
Note that after feedback from @pruivo I updated also the lock timeout.
Sorry for this bug, and thanks for reporting it with a clear description together with anonymized log files that allowed us to hunt this down and make Keycloak better for everyone.
More graceful leaving of distributed caches
More graceful leaving of distributed caches