Does the offline client session DB fallback (PR #17490) also cover entries lost via ISPN000313 abrupt leavers? #54146

danielSoluix · 2026-10-20T04:00:21Z

danielSoluix
Oct 20, 2026

We're on Keycloak 19.0.3, which we know is unsupported. We're planning to upgrade and would like
to confirm that the upgrade covers our case before relying on it.

Background

We're seeing invalid_grant: Session doesn't have required client on offline token refresh.
We believe this is the case described in #21402 (same version, distributed caches, during pod
restarts) and fixed by #9959 / PR #17490 in 21.1.0, where the offline user session is found
but its client session is missing from the offlineClientSessions cache, and 19.x doesn't load
it from the database.

The difference in our case is how the client session goes missing. We don't limit cache size
(offlineSessions / offlineClientSessions are distributed-cache owners="2", with no
max-count and no persistence store; full config below). Instead, two pods left the cluster
abruptly at about the same time, and Infinispan logged:

FATAL [org.infinispan.CLUSTER] [Context=offlineClientSessions] ISPN000313: Lost data because
      of abrupt leavers [keycloak-2-xxxxx, keycloak-3-xxxxx]

(The same message was logged for offlineSessions and the other session caches.) An offline
session that existed before this event later failed with the error above and never recovered.
In an earlier occurrence, we checked the database while the error was happening and the
OFFLINE_USER_SESSION and OFFLINE_CLIENT_SESSION rows were both present and correctly linked.

cache-ispn.xml (6 pods, embedded Infinispan)

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

    <cache-container name="keycloak">
        <transport lock-timeout="60000"/>
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>

Questions

As far as we can tell from the PR Unexpected invalid_grant error on offline session refresh when client… #17490 diff, InfinispanUserSessionProvider.getClientSession
falls back to UserSessionPersisterProvider.loadClientSession whenever an offline client
session is missing from the cache, regardless of why. Is that right, i.e. would a client
session lost through ISPN000313 be reloaded from the database the same way as one evicted
by a cache limit?
The fallback appears to need the client session ID from the user session entity
(if (clientSessionId == null) return null). Can the user session entity in
offlineSessions lose its reference to the client session in an abrupt-leaver scenario,
so the fallback never runs? Or is the reference always intact when the user session itself
is still found?
On current versions with persistent-user-session enabled (the default since 26), is this
scenario handled differently?

Thanks. We can share more sanitized logs if useful.

Answered by JuanTorchia

Oct 21, 2026

I went through the code at 19.0.3, 21.1.0 and 26.8.1. Your case is covered from 21.1.0 on, and what you see on 19.0.3 matches the cause you describe.

Yes. In 21.1.0 getClientSession doesn't care why the entry is missing. If it's not in offlineClientSessions and the session is offline, it goes to persister.loadClientSession(...) (L388-L406). The DB lookup is by user session id + client id + offline flag (JpaUserSessionPersisterProvider L386-L408), not by the cached UUID. That UUID is only a cache key, OFFLINE_CLIENT_SESSION doesn't store it, and after the reload the new one is written back into the user session (L1044-L1061). So lost by ISPN000313 or evicted by a limit is the same path.
…

View full answer

JuanTorchia · 2026-10-21T23:43:03Z

JuanTorchia
Oct 21, 2026

I went through the code at 19.0.3, 21.1.0 and 26.8.1. Your case is covered from 21.1.0 on, and what you see on 19.0.3 matches the cause you describe.

Yes. In 21.1.0 getClientSession doesn't care why the entry is missing. If it's not in offlineClientSessions and the session is offline, it goes to persister.loadClientSession(...) (L388-L406). The DB lookup is by user session id + client id + offline flag (JpaUserSessionPersisterProvider L386-L408), not by the cached UUID. That UUID is only a cache key, OFFLINE_CLIENT_SESSION doesn't store it, and after the reload the new one is written back into the user session (L1044-L1061). So lost by ISPN000313 or evicted by a limit is the same path.
The client session reference lives inside the UserSessionEntity value, and the client session is a different key in a different cache, so after the abrupt leave you can lose one without the other:

user session lost: it's reloaded from DB with its client sessions and the map is rebuilt. This already works on 19.0.3 (getUserSession, loadUserSession).
user session survived, client session lost: the map still has the old UUID. Nothing removes a mapping because the client session is gone, only when the client itself is deleted (UserSessionAdapter L110-L126). So clientSessionId is not null and the fallback runs. On 19.0.3 there is no DB lookup at that point (L397-L405) and the user session stays in cache, which would explain why it never recovers.

What I can't rule out from the Keycloak side is a write that was in flight exactly when the two pods left.

Yes, different. In 26.8.1 the map is gone (the old field is reserved in UserSessionEntity) and client sessions are keyed by user session id + client id, so question 2 doesn't apply anymore. With persistent sessions a cache miss on either one loads from DB (client, user). Also note that with your config (no max-count) Keycloak will set these caches to 10000 entries and owners=1 on startup (CacheConfigurator L249-L271), the DB is the source of truth there.

If you want to check it before relying on it: create a bunch of offline sessions (1000 or so), kill -9 two pods at the same time, wait for ISPN000313 and refresh all the tokens. With DEBUG on org.keycloak.models.sessions.infinispan you should see Offline client session is not found in cache, try to load from db on 21.x+. On 26 with persistent sessions the message is TRACE: Client-session not found in cache, loading from persister.

On 19.0.3 I'd expect a full stop and start of all nodes to clear the stale entries, a rolling restart probably not since state transfer keeps them. I haven't tested this one, it's only from reading the code.

1 reply

danielSoluix Oct 28, 2026
Author

Thanks a lot for the detailed analysis, it was very helpful.

We tested your suggestion: we created 1,000 offline sessions, abruptly killed two
non-coordinator pods at the same time, and confirmed that Infinispan logged
ISPN000313: Lost data because of abrupt leavers for offlineClientSessions.
After the pods recovered, we refreshed all 1,000 tokens on Keycloak 26.0.7:
all 1000 refreshes succeeded and none failed with Session doesn't have required client.

This matches your explanation, so we're planning to upgrade.

Navigation Menu

Repository navigation

Does the offline client session DB fallback (PR #17490) also cover entries lost via ISPN000313 abrupt leavers? #54146

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Does the offline client session DB fallback (PR #17490) also cover entries lost via ISPN000313 abrupt leavers? #54146

Uh oh!

danielSoluix Oct 20, 2026

Background

Questions

Replies: 1 comment · 1 reply

Uh oh!

JuanTorchia Oct 21, 2026

Uh oh!

Uh oh!

danielSoluix Oct 28, 2026 Author

danielSoluix
Oct 20, 2026

Replies: 1 comment 1 reply

JuanTorchia
Oct 21, 2026

danielSoluix Oct 28, 2026
Author