Repository navigation
Replies: 9 comments 17 replies
|
For the |
|
Another alternative is to remove the caching layer: #46496 I strongly believe a stateless application is the way to go. No clustering, no extra exposed ports, no latency to worry about, and support single cluster, multi-az, multi-region, and multi-planet. The only cache that has some use is the Note that Hibernate 2LC is available for user to inject their preferred caching solution. |
|
Perspective as a beginner: I spent days already just wrestling with weird situations where, for example, changes to token mappers and such would see no effect until I change some completely random value (in my case, I set and unset Direct access grants). This is really painful. I can only suspect that caching is the reason for this, and I've seen other slightly-related questions on SO and such where people do, and then undo, random things to get things working. I agree that @pruivo that stateless first is the way to go. |
@reinux - please open a bug that include the Keycloak version and a reproducer. We'll investigate. |
|
Apologies in advance if I misunderstand anything, I’m still relatively new to this area and am learning as I go, so I hope my comments are taken in the constructive spirit intended. I completely agree with @pruivo ’s proposal to remove the cache layer entirely. After watching the recording ( https://youtu.be/rHeOQejcCQI?t=231 ) and reviewing the recent community survey results ( https://www.keycloak.org/2026/03/distribution-surver-results ) , it seems clear that the Keycloak cache layer or the caching approach, has become a source of complexity, operational challenges, and division within the community. Given the technical benefits @pruivo mentioned, along with the opportunity to simplify the architecture and create greater alignment across the community, the proposal feels very compelling to me. As a CNCF project, shouldn’t Keycloak also strive to align as closely as possible with true cloud-native principles? |
@ahus1 Do you have a document or issue somewhere that articulates your definition of "fast enough"? We (Phase Two) have a bunch of different implementation of "run[ning] Keycloak without caching" that we've done over the years, and they met our (or customer) requirements. However, knowing what you (and the Maintainers) mean by "fast enough", and why Keycloak is using caching in the first place (what problem were you trying to solve?) would be really helpful to enable us to make specific feedback. |
|
This is not so much about what maintainers want, but more about what users are used to and might need in their environments. For some, every millisecond counts. You can already disable some caches today. I'll post some hypotheses below later today. to get feedback from the community about response times. |
I'm glad to see your sense of humor is still intact.
Appreciate those. We'll review and make comments. Similar to @sschu's comment, we have several customers in the 1-5M sessions range (both normal and offline), with 10k default Users in the local cache, so we're basically hitting the database every time. Performance is less than ~8-10ms average difference for a cache hit versus miss, which is acceptable for these use cases. We haven't yet profiled optimizations on the database side for this, but suspect there is more performance to be gained there. We haven't yet turned off caching of persistent sessions in production environments, but our tests indicate a similar impact as the Users cache example above. |
|
Thank you for your comment. When it comes to sessions, it is not so much about the total sessions stored in the database, but the number of active users within the lifetime of an access token, which would be approx. the number of token refreshes in the period of an access token lifetime. It would be good to have that number to put it in relation to the caches (which are for the sessions the size of the cache in each node multiplied by the number of nodes). The caches also provide also a cache hit ratio, but that's no metric describing a user facing behavior, so I appreciate your mentioning of cache hit vs miss latency. cc: @sschu |
|
The ratio of hits to misses in our scenario for offline sessions is 20,46:1. For users cache its still an impressive 0,17:1. |
Hypothesis 1: Disable caching of persistent sessions is OKRationale: We want to see if we can simplify a Keycloak setup with fewer or no caches. This input will provide us data for the session caches. We’ll run different experiments for different caches to track the effects on response times and CPU usage. While we could come up with synthetic test cases, we would like to see numbers from real production environments and possible challenges. TL;DR: Disable the session cache and post your results in this thread In a multi-cluster setup of Keycloak, this is already disabled by default, so we already have some numbers on this one. Feel free to share your results in private by sending it to the email address listed in the profile of @ahus1. Expected outcome
Prerequisites
Cache size default10k entries * number of pods Extra expected Database usage when cache is disabled
SELECTs are usually faster than any INSERT/UPDATE, so the additional load should only add some percentages to the CPU load of the database. If running this, make sure your database CPU usage is not over 60% already on average usage. Test first in a test environment. AssumptionIf the number of active sessions in the access token lifetime is larger than the cache sizes, your session cache is probably not very effective currently anyway. How to disable the cachePersistent sessions enabled (default) + QuestionsFor your production or test environment, read the values before/after with comparable load:
Link to guides |
|
Hello @erenkan / @muhammedogz / @BenLad17 / @bastianccm - it would be great if you could help out here and provide performance data for us. If you don't want to share data in public or have questions, you can share it via the known private channels, or via the email address in my profile. Thanks! |
|
Hi all, Following up on the question about disabling Keycloak's embedded caches (session and user). We ran an isolated benchmark to test this properly,and our conclusion is to keep both enabled: full cache ablation regressed every workload we tried, and disabling the user cache alone isn't something we'd recommend for lookup-heavy traffic. Disabling just the session cache was mostly neutral, with one clustered scenario showing a tail-latency risk we're still treating as unconfirmed rather than settled. Separately, while digging into why a cache miss is expensive, we found and fixed a composite-role N+1 pattern in Keycloak itself. That's already merged upstream as #50370 (26.7.0). We've sent the full evidence and methodology directly to @ahus1 by email, since it's not something we're hosting publicly. Thanks. |
|
Hello @erenkan, thank you for all the tests. I've reviewed the package. My summary on the results: It is great to see so many extra data points that you've added to this. While disabling caches has trade-offs, I still think they are good for scenarios like the "stateless" setup. Maybe you want to share with the community the different response times you measured with for the user-cache-disabled scenario so readers can put it into perspective. Reading the package, I understand that "full cache ablation" means disabling also the realm cache. I agree, this is very expensive, and shouldn't be used. This was "only" about (1) not caching users and (2) not caching user sessions. In your setup, I see you added a large number of users, and then increased the ISPN cache to cover 10% of those users. In the cache-enabled setup, this lead to a high cache hit ratio (I won't cite the numbers here, as you didn't share them publicly, but maybe you want to elaborate on that part so the community know about it). In our team we were also wondering how effective the cache is, given that each node has its own cache, and caching a user on one node doesn't have a positive effect if the next request for that user is handled by another node. Can you share production metrics? As the team developing the code, we wonder how people (like you) run Keycloak in production (do you increase the user cache size) and do you see a good cache hit ratio? It would be good for me to understand what "lookup-heavy traffic" means, in and what scenarios you see it in production, and when you don't see it. When comparing the absolute numbers for (user) lookup requests, they increased significantly in absolute numbers, I agree. And if you don't have any other traffic ongoing on the database, the increase in DB load is also significant. I wonder that for a database that is handling user logins and token refreshes that lead to a significant base load (assuming persistent sessions are enabled), this would make a smaller difference. A reasonably sized page cache in the database would also eliminate the physical reads. I'm very grateful for the #50370 contribution, and I'm happy that it is now available in 27.0. While we don't plan to remove the realm cache, it will help a lot with cold caches and cache misses. The upcoming release contains one more commit e14e8b5 which wasn't there when you ran your tests. This should improve the user-cache-disabled scenario. Feel free to reply in the private email chat with data you don't want to share publicly. |
Hypothesis 2: Disable caching of users is OKRationale: We want to see if we can simplify a Keycloak setup with fewer or no caches to eventually become stateless. This input will provide us data for the session caches. We’ll run different experiments for different caches to track the effects on response times and CPU usage. While we can come up with synthetic test cases, we would like to see numbers from real production environments and possible challenges. TL;DR: Disable the user cache and post your results in this thread Feel free to share your results in private by sending it to the email address listed in the profile of @ahus1. Expected outcome
Prerequisites
Cache size default10k entries on each Keycloak node Extra Database usage when cache is disabled
SELECTs are usually faster than any INSERT/UPDATE, so the additional load should only add some percentages to the CPU load of the database. If running this, make sure your database CPU usage is not over 60% already on average usage. Test first in a test environment. AssumptionUsers are cached only per node. If a second request touches a different node then it doesn’t benefit from the cache entries from the first request, so your caches might not be very effective currently anyway. How to disable the user cache
QuestionsFor your production or test environment, read the values before/after with comparable load:
Link to guides |
|
I cannot provide hard data but in our installation, we have 2M offline sessions in the caches (InfinispanUserProvider) but only the default of 10K in the Users cache. This means the cache is not really used and every token refresh trigger a user load (select on USER_ENTITY, USER_ROLE_MAPPING, USER_GROUP_MEMBERSHIP, USER_ATTRIBUTE). And this works fine. |
|
Thank you @ahus1 for opening this discussion and inviting community feedback. We ran Keycloak v22 with a self-hosted external Infinispan cluster on AWS serverless infrastructure (Fargate) and eventually had to move away from it. The core issue: JGroups requires stable node identities and direct inter-container communication , neither of which ephemeral/serverless containers reliably provide. Node failures during routine maintenance caused Infinispan members to drop and rejoin unpredictably, leading to split-brain scenarios and disrupted session state. Beyond serverless, this affects any team with modern deployment practices like rolling deployments, OS patches, CVE cycles all cycle containers, and every restart is a membership change that triggers rebalancing and risks a partition window. The operational burden of self-hosting Infinispan, combined with the 'run the engine' overhead ultimately led us to move away and implement a Managed Redis-based solution as alternative for external infinispan. We fully support Hypothesis (Keycloak setup with fewer or no caches and using Persistent Store). For teams already running HA PostgreSQL (RDS/Aurora), the database is the natural authority for durable state. Removing a second distributed cluster that has to stay consistent with it simplifies the failure model — a stateless Keycloak backed by PostgreSQL seems the right direction. What about the other cache types?Sessions are a great first step, but a truly stateless Keycloak means addressing everything currently in Infinispan — authenticationSessions, loginFailures, actionTokens, work, and the local caches (realms, users, authorization, keys, crl). Has Keycloak core team done any feasibility study for migrating all of these to the persistent store? On latency and L1/L2 cachingWe understand that pushing everything to PostgreSQL could introduce latency under high concur 10000 rency. Has the team done any performance testing here? We're also curious whether an L1/L2 model has been previously considered:
For teams who want no managed cache dependency, even a simple L1 Cache + PostgreSQL two-tier model would be a significant improvement over the current setup. Three independent Redis/Valkey implementations — worth paying attention toThree separate teams, without coordination, built Redis/Valkey cache providers for Keycloak:
Three teams, different orgs, same root cause, same solution. That's a strong signal. A first-class path to persistent store — optionally with L1/L2 — would consolidate this rather than leaving scattered implementation across separate repos. Open Questions
These baselines would also help teams as they plan and prepare for performance testing to determine the destination state for Keycloak caching. We're happy to contribute performance testing for the persistent store solution with L1/L2 — and please do let us know if Keycloak or Red Hat has already executed benchmark testing with this configuration so we can use those as a baseline. Thanks for driving this forward @ahus1 it's the architectural direction that would benefit lot of enterprise teams and the community !! |
|
@Anoopgopi about authenticationSessions, loginFailures, actionTokens caches, see my answer below. The work cache will stick around for a while because it has to handle the invalidation message, and the local caches will stay. They can be disabled but they have an huge performance impact, more tham 10x slower. |
|
@Anoopgopi - thank you for the extended answer about when the Infinispan based solution wasn't a good fit for you. You have also been around for quite a while as you mention Keycloak 22, thank you for being a user of Keycloak! First, some things that changed since then, and below some tasks were I would like to get some help:
Pedro and I are working in the same team, and we're currently exploring ideas to have fewer caches. I assume you're using your Redis implementation in production? Are you using persistent user sessions, that is: All user and client sessions are stored in the database? If yes, then I'd like to get feedback for the following questions:
Thanks! |
|
Thank you @ahus1 for taking the time to review and respond in details ! Apologies for the delay in getting back to this thread. I wanted to do some re-testing with Keycloak v26.6.3 using the persistent store (with user sessions enabled) before confirming back here. My colleague @rsquared226 will be sharing the results of the testing and some of the observation we had during our testing shortly On your questions related to our current setup: Our Redis implementation is in production. We store all user sessions,authenticationSessions, loginFailures, actionTokens, and single-use objects (work queue entries) in Redis. This design eliminates the need for any synchronisation with the embedded Infinispan cluster, which is not a recommended or viable pattern for our current serverless architecture(due to below reasons).
That context is also why the current v26.6.3 persistent store, while a meaningful improvement, does not fully meet our architecture in its current state, it still leaves several entry types in the embedded Infinispan layer that require clustering, which reintroduces the same operational challenges we moved away from Our plan is to continue with Redis for now, but the work @pruivo #49469 is doing on a truly stateless/cacheless Keycloak looks very promising , a genuine stateless Keycloak would be a significant win for teams running on modern serverless and cloud-native infrastructure. Thank you @pruivo @ahus1 for confirming on the questions, and a big thank you for the SPIKE work on #49469 I strongly believe investing in this direction would benefit the broader Keycloak community, and we are very much looking forward to seeing it progress. We are also willing to try out the solution(+ contribute) and will pass along any feedback from our side. |
|
Hey all! To give additional context on our deployment strategies and our tests: Deployment patternExpanding on the deployment pattern for a serverless architecture, there may not always be an option for doing rolling restarts (starting/terminating nodes one-by-one). For example (for any change rollout or patch updates):
This deployment strategy applies to Keycloak nodes (embedded Infinispan) and external Infinispan nodes. Since all old nodes would be terminated at once rather than a rolling restart, we run into a split-brain state with a lot of rebalancing even with Keycloak 26.6.3, causing high response times and 5XXs during that period of time. At times, we've needed to step in and manually resolve these issues. Below we describe some of these issues that we've observed with the exact test setup. What we did test (Keycloak 26.6.3, embedded -- not external -- Infinispan, Test scenarios and results
The deployment failure, in short: at the end of the traffic-shift (from old nodes to new nodes), one new node would stop responding to cache-write acks while still answering JGroups heartbeats — never marked dead, never evicted. 98% of cache-write timeouts in the logs were One thing to mention is that we noticed that the The potential of cacheless modeWhat excites us is that cacheless mode (#49469) may support our deployment pattern — most of what hurt us are some in-memory caches that Next steps on our side: we will do additional testing on the nightly / 26.7.0 cacheless build to see whether it resolves the deployment-toggle issue. We can share results back here or on #49469, then look for places we can contribute from there. Thank you both for driving this! cc: @Anoopgopi |
|
Thank you @rsquared226 for sharing this. TL:DR: This reply turned to out to be a longer reply than expected. The first step would be to check if nodes are shutting down gracefully. See the end of this email for more details. You describe a setup running in multiple regions, which separate Infinispan clusters in each region. Please note that this will lead to stale local caches when one region updates the realm data, and the other one will never receive an invalidation. Once #49469 is in place, this will should no longer be a problem. About a concurrent shutdown of nodes: This will partially improve with 26.7.0, as @pruivo made the embedded Infinispan shutdown more graceful, see #48341. This change is available in the nightly builds. It should cover most cases until we upgrade to Infinispan 16.2+ which has more advanced functionality, see https://infinispan.org/blog/2026/06/03/infinispan-16-2. We are deferring the upgrade of Infinispan until the next Infinispan LTS release is out. To analyze the behavior you saw for 26.6.3 on node switchover/node leave: Please share (either here, or in an email to the email address stated in my profile) the log file of the new node (ideally the full log including the startup log) and the one that was showing the errors (the part where nodes where started or left including the errors above).
While I could see situations where some requests for a short while, I stepping in manually should not be necessary. This points towards a bug that we should fix, or a configuration that needs improvement. If would be great if you could describe the observed behavior in those situations, and how long you waited for this to be resolved. There are some mechanism to detect split-brains and how to recover from split-brains, but they depend on having readiness probes enabled, FDSOCK port connectivity, and giving it some time (up to a minute). Split-brain detection is improved in 26.7, see #47970. The first thing to check is if the nodes perform a graceful shutdown; this can be determined in the log of the node. You describe that the entry is removed from the JGROUPS_PING table, but as this is managed by the leader, removing the entry can be triggered by both a graceful or a sudden exit of a node. For that, please share a snippet of the shutdown log. If it ends with "(Shutdown thread) Keycloak stopped in X.XXXs", it has stopped gracefully. A graceful shutdown depends on Keycloak receiving the correct signal from the operating system. If you added a custom entrypoint for your container, see https://www.keycloak.org/server/containers#_custom_entrypoint_shell_scripts on how leverage "exec" to ensure that Keycloak receives those signals. |
|
FYI, I've created a SPIKE this morning: #49469 |
|
The spike has progressed, and it is now on a good way to be an experimental feature in 26.7. It can be used in single-cluster and multi-cluster setups, Please try it out and provide us with feedback: #49469 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
In Keycloak, “caches” are used for different things:
This is quite a mix of functionality. While it was sometimes difficult to set things up with older versions, the most recent versions should be a lot more simple to set up and upgrade.
This post summarizes what we have done in the past, what we think the current state is, and where we want to head in the future. Please comment on this discussion to provide ideas and also context in which setups this works well or where it needs improvements.
How we made them simpler to use
We evolved the area of caching in the past based on user feedback:
Working for the vast majority of users?
To gather some information from the Keycloak repository: We have not seen many open issue around Infinispan lately
We plan to continue to improve it, but it would require some specific information about a user’s setups, performance analysis, etc. We would appreciate it if people would engage in GitHub issues to provide context and data about the problems they are experiencing. We’ll be happy to dive into details with you.
How caching should evolve
What currently is the field of “caching” in Keycloak, is actually a mix of different functionalities. It can be split-up and changed depending on different architecture decisions on the overall Keycloak project.
Here’s some of the ongoing work and alternatives that already exist:
There is also the experimental feature “clusterless” which allows storing the user sessions only in the external Infinispan and not in the database.
Looking further into the future:
Changing things one step at a time
So where should we start to change things? Post your comment below to engage in this conversation!
All reactions