Use shorter lifetimes for offline session cache entries in memory #24274

thomasdarimont · 2023-10-25T08:18:33Z

thomasdarimont
Oct 25, 2023
Collaborator

Ideas for Optimizing Offline Session handling

Memory consumption

Offline Sessions are held in memory for too long

Problem

Description of the problem
When using many offline sessions, a customer of mine had enormous problems with high memory requirements (sometimes several 10GB). The machines regularly ran out of memory, and GC efforts consumed a lot of CPU resources.
A few resourceful software developers had an IMHO simple and clever idea: keep offline session cache entries shorter in memory than in the database.

Scenario

Example scenario in which the problem arises

Keycloak cluster has about 25 million users with about 10 million offline sessions for a mobile app.

Offline session loading is configured as "lazy" -> preloadOfflineSessionsFromDatabase=false
Offline Session Idle: 30 days
Offline Session Max: 180 days

Idea/Solution

Description of the proposed solution / idea

Since Keycloak can lazy load offline sessions from the database, Keycloak does not have to keep all loaded offline sessions permanently in memory but can limit the TTL of the cached entries in memory to a value of ~12h - 24 h.
The reasoning is that power users who use the app regularly perform a token refresh (with the offline refresh token) often enough so that the offline session remains in memory. Offline sessions of users who use the app only occasionally do not have to be permanently in memory and can be automatically removed from memory after 12h-24h. If such an occasional user returns, his offline session is reloaded "lazily" from the database at the cost of increased latency for database access.

In the database, the offline session entries are still stored according to their "Offline Session IDLE" or "Offline Session Max" for longer.

Approach for implementation

How could the proposed solution be implemented?

When OfflineSession / OfflineClientSessions entries are stored in Infinispan, the TTL is currently set to -1 with a max idle value. Instead, one could introduce a new server configuration property like "offlineSessionCacheLifespanOverride" (int in millis) and thus (if specified) determine the (shorter) TTL time, which could be configured on the InfinispanUserSessionProviderFactory.

The configuration property could then be used to override the lifespan of the cache entries.

This will then keep the offline session Infinispan cache entry in memory for a (much) shorter time and remove it automatically after the TTL has expired. Of course, the offline session cache entry remains stored longer in the database.

Effort / Complexity

_How much effort must be put into this to make it work? _

Effort: Low
Complexity: Low

Migration

How to migrate existing offline sessions, if necessary

Existing offline session cache entries that still have the long TTLs, could be adjusted on next access and configured with the shorter TTL such that the system "heals" itself and releases more memory over-time.

Expected Results

Expected Results from this optimization, potentially based on PoCs and load-tests)

In the project, a customer tried this and experienced significantly reduced memory consumption (during a longer-running load test) and improved system stability.

thomasdarimont · 2023-10-25T10:46:40Z

thomasdarimont
Oct 25, 2023
Collaborator Author

A PoC implementation of this can be found here: #24302

The current implementation uses server wie configuration, but this could also be supported on realm level, with the following precedence:

Realm
Global Override
Default

The offline session cache entry lifetimes can be configured via the following properties:

# 300000 = 5 mins for testing
# 60000 = 1 min for testing
spi-user-sessions-infinispan-offline-session-cache-entry-lifespan-override=60000
spi-user-sessions-infinispan-offline-client-session-cache-entry-lifespan-override=60000


# Debug Logging enabled to see the effect:
log-level=info,org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider:debug

Test Script:

KC_ISSUER_URL=http://localhost:8080/realms/test
KC_CLIENT_ID=demo-client
KC_CLIENT_SECRET=xxx
KC_USER_USERNAME=tester
KC_USER_PASSWORD=test

curl -v -k \
    -X POST \
    -d client_id=$KC_CLIENT_ID \
    -d client_secret=$KC_CLIENT_SECRET \
    -d username=$KC_USER_USERNAME \
    -d password=$KC_USER_PASSWORD \
    -d grant_type=password \
    -d scope="openid profile offline_access" \
     $KC_ISSUER_URL/protocol/openid-connect/token 

KC_REFRESH_TOKEN=xxx

curl -k \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "client_id=$KC_CLIENT_ID&client_secret=$KC_CLIENT_SECRET&grant_type=refresh_token&refresh_token=$KC_REFRESH_TOKEN" \
   $KC_ISSUER_URL/protocol/openid-connect/token \
     | jq -C .

Example session:

...
2023-10-25 13:02:51,116 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.

auth request with scope offline_access

2023-10-25 13:03:11,112 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) Offline user-session not found in infinispan, attempting UserSessionPersisterProvider lookup for sessionId=44f3a8cd-6105-496a-bf9a-71b4f69328df
2023-10-25 13:03:11,120 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) Offline user-session not found in UserSessionPersisterProvider for sessionId=44f3a8cd-6105-496a-bf9a-71b4f69328df

Token refresh

Token refresh

Offline refresh token cache entry lifespan timeout ...

2023-10-25 13:06:03,810 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) Offline user-session not found in infinispan, attempting UserSessionPersisterProvider lookup for sessionId=cc6b0cb4-c9aa-4afb-9c37-136e0515c7e4
2023-10-25 13:06:03,814 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) Attempting to import user-session for sessionId=cc6b0cb4-c9aa-4afb-9c37-136e0515c7e4 offline=true
2023-10-25 13:06:03,820 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) user-session imported, trying another lookup for sessionId=cc6b0cb4-c9aa-4afb-9c37-136e0515c7e4 offline=true
2023-10-25 13:06:03,820 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProvider] (executor-thread-1) user-session found after import for sessionId=cc6b0cb4-c9aa-4afb-9c37-136e0515c7e4 offline=true

Token refresh

3 replies

thomasdarimont Oct 25, 2023
Collaborator Author

@mposolda @hmlnarik The PoC looks promising perhaps this could be considered as an optimization option for the upcoming Keycloak releases.
It is IMHO a small change with potentially large memory savings at the cost of some additional latency.

hmlnarik Oct 25, 2023

Thank you @thomasdarimont. I've not checked the code yet, but having the option to passivating offline sessions faster while having the option to reload them anytime looks sane and desirable. Adding a provider option to that also looks like a good way to go.
Cc: @martin-kanis

martin-kanis Oct 27, 2023
Collaborator

@thomasdarimont Thank you for bringing this interesting idea. I'll bring this on @keycloak/store board for a discussion. Just a note, we have some improvements around sessions durability and footprint on our radar.

thomasdarimont · 2024-02-21T16:09:43Z

thomasdarimont
Feb 21, 2024
Collaborator Author

Note that some of this recommendations have been implemented in the meantime via #26998

0 replies

Navigation Menu

Repository navigation

Use shorter lifetimes for offline session cache entries in memory #24274

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Use shorter lifetimes for offline session cache entries in memory #24274

Uh oh!

Uh oh!

thomasdarimont Oct 25, 2023 Collaborator

Memory consumption

Offline Sessions are held in memory for too long

Problem

Scenario

Idea/Solution

Approach for implementation

Effort / Complexity

Migration

Expected Results

Replies: 2 comments · 3 replies

Uh oh!

Uh oh!

thomasdarimont Oct 25, 2023 Collaborator Author

Uh oh!

Uh oh!

thomasdarimont Oct 25, 2023 Collaborator Author

Uh oh!

hmlnarik Oct 25, 2023

Uh oh!

martin-kanis Oct 27, 2023 Collaborator

Uh oh!

thomasdarimont Feb 21, 2024 Collaborator Author

thomasdarimont
Oct 25, 2023
Collaborator

Replies: 2 comments 3 replies

thomasdarimont
Oct 25, 2023
Collaborator Author

thomasdarimont Oct 25, 2023
Collaborator Author

martin-kanis Oct 27, 2023
Collaborator

thomasdarimont
Feb 21, 2024
Collaborator Author