Improved scalability over number of realms #11074

davoustp · 2022-04-01T15:40:11Z

davoustp
Apr 1, 2022

Update 2025-10-05: With Keycloak 26.4, it should be fine to run Keycloak with 1k+ realms as long as you keep increasing the realm cache. See #11074 (reply in thread) for details.

Hi,

As described in KEYCLOAK-4593, Keycloak struggles to scale beyond 100-200 realms. This proves to be a road-block to embrace Keycloak as the main component of a large scale multi-tenanted solution.

Regardless of the efforts on the under-development Map-based storage subsystem (which should fix some of the underlying issues at least), I performed a quite intensive investigation session around this scalability problem, which is described here with findings and proposals.

Reproducing the issues

Running Keycloak

Keycloak is run using Docker and a Postgres engine:

docker run -d --name postgres-keycloak \
  --network keycloak \
  -e POSTGRES_USER="postgres" \
  -e POSTGRES_PASSWORD="postgres" \
  -e POSTGRES_DB="keycloak" \
  postgres:14.2-bullseye

docker run -it --rm --name keycloak \
  --network keycloak \
  --cap-add SYS_ADMIN \
  -p 8080:8080 \
  -p 8787:8787 \
  -p 8999:8999 \
  -e KEYCLOAK_ADMIN="keycloak" \
  -e KEYCLOAK_ADMIN_PASSWORD="keycloak" \
  -e KC_DB="postgres" \
  -e KC_DB_URL="jdbc:postgresql://postgres-keycloak.keycloak:5432/keycloak" \
  -e KC_DB_USERNAME="postgres" \
  -e KC_DB_PASSWORD="postgres" \
  -e DEBUG="true" \
  -e DEBUG_PORT="*:8787" \
  -e JAVA_OPTS_APPEND="-Xmx1g \
    -Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.rmi.port=8999 \
    -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.local. -Djava.rmi.server.hostname="$(hostname)" \
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/keycloak.hprof" \
  quay.io/keycloak/keycloak:17.0.0 start-dev \
    --log-level=INFO

Heapdump on OOM is enabled as well as JMX to monitor the instance.

Increasing access token lifespan

The same access token is used across the entire session. As a result, you'll need to extend the lifespan of access tokens (including implicit flow) in master realm to a value larger than default (Settings / Token), for example 1 day.
You also need to set the same lifespan onto the admin-cli client (Client / admin-cli / Settings / Advanced Settings).

Measuring tenant creation duration

Creating the realms is performed using shell scripting (useful to be run against any existing Keycloak, even though this could be done using a Java-based project as the ones existing in the codebase):
keycloak-create-realms.zip

This script just needs curl, jq(from https://stedolan.github.io/jq/ ) and GNU gdate.

Running it is straighforward:

./keycloak-create-realms.sh

It will authenticate as the root admin (change credentials if different from above KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD values), and attempts to create 1000 realms.
It outputs the time spent for each operation for each realm ordinal.

Measuring Admin console load time

With Keycloak being cold started, the amount of time required to login as root admin to manage realms is measured manually for various number of tenants in Keycloak, using the browser developer console (Finish duration measure using Chrome), from the moment the login form is submitted to the moment the console page is fully loaded.

Measures with version 17.0.0

All measures below were performed with stock version 17.0.0.

Note that the same measures have also been performed against the latest codebase (commit fa87d462108899d38f22242a8abdb36e08cd1af0), with the same results.

Realm creation time with version 17.0.0

Grabbing the output from the script above allows to graph the time spent to create a new realm against the total number of created realms:

The tenant creation duration diverges exponentially when the number of tenants grows.
You can also note that measure was stopped at ~620 tenants, because it was taking ages at that point.

Admin console load time with version 17.0.0

Here is the outcome for stock Keycloak 17.0.0:

Here again, the load time is increasing exponentially with the number of realms.

Findings and proposals

These are the main findings, I may have skipped some minor ones, but they all fall along these same lines.

Adding/removing children roles to/from parent role triggers full JPA collection load

A very well known issue with JPA/Hibernate: adding or removing a role to the list of children roles actually triggers loading the OneToMany relationship, before inserting the proper record into the COMPOSITE_ROLE join table.

Proposed optimizations:

Emerge the JPA entity corresponding to the join table
Upon addition/removal of a child role:
- if the children collection of this role is already loaded, keep using the current collection addition/removal operation
- if the children collection is not loaded, then perform the addition/removal operation using the join table entity directly

Role composition suffers from N+1 load issue

Because of the composite nature of roles, the set of roles is actually a directed graph, implemented within the database schema using the COMPOSITE_ROLE join table .

The children roles are loaded from their parent using standard JPA/Hibernate OneToMany mapping, which goes through the join table.

As a result, when attempting to expand all roles with their direct or indirect children roles (implemented within RoleUtils as a recursive traversal of roles, in current codebase), this generates a huge amount of queries to the system, especially visible onto the ${role_admin} role (the root role of many other roles).

Computing this set of roles (which is the transitive closure of the initial role set, in graph/set theory wording) requires a different approach for scaling.

Proposed optimizations:

Using a transitive closure table would probably be the ideal choice, but this would require quite significant changes and book-keeping, so I ruled it out.

Instead, the proposal is to introduce bulk loading for both roles and role composition.
Performing the role expansion algorithm can be depicted as:

find all role ids by bulk-fetching the role composition records, with one bulk lookup per graph edge hop (this is where role expansion actually occurs)
bulk load all roles by their ids

As a result, for a graph of n roles composed in a graph of depth d, the expansion now requires only d (simple) lookups instead of n (complex joined) lookups.

As a bonus, the role composition structures collected during the first phase allows to avoid another N+1 issue in the caching layer (which eagerly caches role composites).

There are quite a few things around this one:

implies that the model SPI is augmented with the appropriate signatures for bulk retrieval
bulk loading in JPA can be performed as select ... from ... where id in :ids JPA query (as an alternative, the Hibernate-specific loadByMultipleIds can be used instead) - requires to enable Hibernate's in_clause_padding to avoid cluttering statement caches
the JPA implementation must also take care to limit the number of ids to bulk load using a select ... from ... where id in :ids JPA query (as Oracle and other engines do have limit - 1000 for Oracle)
the implementation of these signatures in the Infinispan caching layer can be optimally done for both bulk lookups, keeping the underlying storage layer hits minimal

Bulk-loading large collections by ids to optimize caching layer hit ratio

This is especially true for retrieving the list of realms: the current implementation cannot do anything but hit the storage layer to retrieving all the realm (and cache the resulting realms in the process).

Proposed optimizations:

Perform this as a two-step lookup: first collect all entity ids, then bulk load them (which gives the opportunity to leverage the caching layer).

Accessing entities just for grabbing their ids

This happens quite frequently: role mappings, clients, default role, client scopes...
This means loading a lot of entities while only their ids are required.

Proposed optimizations:

Extend the model to get access to these readily available ids when appropriate.

Loading large JPA dataset clutters the persistence context and degrades flush-time efficiency

When loading a large set of entities from the JPA layer, these will be kept into the persistence context (first-level cache in Hibernate wording): most of the time, these entities are converted to application-level model instances without retaining the underlying JPA entity, which nevertheless stays in the persistence context.

As a result, any time a flush is required (explicit, JPQL, HQL, native query), then the Hibernate layer needs to scan the persistence context and perform a dirty check on each appropriate object to actually execute the proper persistence operation(s) if need be. The thing is that this dirty checking mechanism is expensive, and its cost grows quadratically with the number of items to scan.
See https://stackoverflow.com/a/18948517 for a pretty decent explanation.

Proposed optimizations:

This one is a little bit tricky, because it involves detaching JPA-loaded entities: detaching an entity which is otherwise referred to by another part of the code leads to failing insert or detached exceptions.

To do this is safe-enough manner, it first requires to analyze the object model and cascade the DETACH operation to relationships down to the graph. In other words, careful code analysis is required.

Second, detaching an entity during an operation that loads it implies that it was not already loaded previously (or the graph of objects may have been modified prior to the operation and a detach will actually loose all modifications).
This can be achieved by looking into the Hibernate persistence context using the Hibernate SPI to check if the identifier of an object about to be loaded exists in here.

Investigation outcome

Performing these optimisations as a measure-optimize-repeat process led to the results shown in this section.

Note that the same vertical scales are used to allow to properly compare to the initial performance baseline.

Realm creation time with optimisations (forked from main 18.x branch)

The realm creation time with the optimisation-enabled branch is depicted below:

This now exhibits a linear, almost constant creation time, within the sub-second range.

Going up to 3k created realms:

Low-slope linear profile is confirmed when number of realms grows.

Admin console load time with optimisations (forked from main 18.x branch)

The admin console load time using the optimisation-enabled branch is shown below:

Again, this now shows a linear admin console load time (20 seconds to load with 1000 realms).

Going up to 3k created realms:

Linearity still looks ok, with a load time of 50 seconds for 3000 realms.

Code changes

All the code changes has been done in fork https://github.com/davoustp/keycloak/tree/KEYCLOAK-4593-investigations

Please note that this is an investigation branch, so I did NOT collapse the various commits, but I find it valuable so that the community can have a look into it.

I'd say that I hate such large code changes, which I usually break apart in some smaller, self-sufficient units, but doing so when performing these optimisation loops (and discovering the codebase internals as well) was too high a challenge.

Nevertheless, all test base integration tests pass, so it looks I did not break too many things along. ;-)
I did not attempt to go too deep into the map storage layer, since this is a hot topic right now (not sure how to run the test suite against a map storage provider, btw), so a deeper look is probably needed.

Please let me know if this is of interest to you guys, and let's discuss it to see if and how it can be contributed to Keycloak. :-)

thomasdarimont · 2022-04-02T22:45:06Z

thomasdarimont
Apr 2, 2022
Collaborator

That's a lot to ingest but this are indeed impressive results, thanks for sharing!

Did your changes also improve the performance for managing large numbers of clients (> 10000)?

4 replies

davoustp Apr 3, 2022
Author

Did your changes also improve the performance for managing large numbers of clients (> 10000)?

It would need some measures here, probably. Are you referring to an existing issue? Is it just being able to create 10k+ clients in a single realm, or spread across multiple realms ? Admin console loading? Something else?
I'm happy to have a go and check if there is an impact of the changes to this scenario, just let me know.

thomasdarimont Apr 3, 2022
Collaborator

Not in particular, but there are (were) several issues with realms that needs to manage a large number of clients, e.g.:
KEYCLOAK-8275, https://keycloak.discourse.group/t/maximum-number-of-clients-applications-in-a-realm/461/15 and many other google groups discussions that mention this.

There are multiple scenarios:

having a large number of realms with a small set of clients each (this should work already, with this fix) (SaaS Product Scenario)
having a small number of realms with a large set of clients each (IDaaS for B2B Partners / IoT scenario)
having a large number of realms with potentially a large set of clients each (General purpose IDaaS / B2C IoT scenario)

The largest number of clients in a realm I've seen so far were ~40k and with this the system was quite slow (even with some cache size tuning). Although it is possible to support an arbitrary number of clients via the client-storage and a custom org.keycloak.storage.client.ClientStorageProvider it would be great if Keycloak could handle large amounts of clients (100k+-1m+) well out of the box.

I'm also curious to learn what the folks of the Keycloak team think about your optimizations in the context of the new upcoming storage architecture.

davoustp Apr 3, 2022
Author

Regarding the impact on capability to handled 100k/1m clients:

Just had a go with the code provided with KEYCLOAK-8275 against stock Keycloak 17.0, and after adjusting the code to latest codebase (a missing commit to finalize realm creation, deployment model change with Quarkus so now manually copied to the container and restart), it looks to me that it performs already pretty well (unless the testing methodology does not reflect the problem at hand - can't tell myself here).

It took 09:22 (562 seconds) to create 1 million clients into the testRealm.
Snapshot of the metrics it produces:

-- Timers ----------------------------------------------------------------------
client-registration
             count = 928284
         mean rate = 1719.01 calls/second
     1-minute rate = 1748.08 calls/second

The database indeed shows these clients:

keycloak=# select count(*) from client where realm_id = 'testRealm';
  count  
---------
 1000006
(1 row)

And finally, connecting to the admin console, selecting the testRealm then finally going to Clients is instantaneous, including next / previous page, as well as searching:

Looks like it was adressed prior to the optimisations I propose here ;-)

stianst Aug 25, 2022
Maintainer

Can't remember when the improvements around large number of clients where done, but we did a lot of improvements around here and KC should scale easily enough to 100K+ clients now.

laskasn · 2022-04-04T08:22:44Z

laskasn
Apr 4, 2022

This is a really interesting optimisation!
@davoustp Have you also tried this in the "standalone high availability mode"? Where there are a couple of nodes invalidating their realms and loading in parallel data from the DB? It would be interesting to see how much it improves in that mode too.

We have also pinpointed quite a few time ago another issue with keycloak, when it has a high number of Identity Providers within a realm.
In our production environments, we have federations of Identity providers with 3-4 thousand idps, and the keycloak works terribly with that large number of IdPs, mostly because the IdPs are a subelement (tightly coupled) of the realm.
We had done a preliminary optimisation for this issue, allowing the admin console to work smoothly even with that large number of IdPs in the realm, which fixes (mostly) the problem, but the real solution to achieve full scalability, would be to decouple the Identity Providers from the realm model (just as it's done with the Clients and the Users).

2 replies

davoustp Apr 4, 2022
Author

HI @laskasn
I did run the HA integration test, but the test scope looks pretty narrow.
And I'm be really more comfortable if I can check that I properly handled cache entries invalidation checks in the provided changes - and this cannot be tested with a single node.
Currently, I'm a bit at loss to understand which Quarkus-based distro configuration knobs I need to turn to have multiple containers (would be running behind a reverse proxy, no pb here) and run the script against this cluster.
If you can point me to some directions I could use, I'll run the same against a fully load balanced clustered Keycloak (I'd go for 3 nodes as there are situations you can't reproduce with 2 nodes).
Any idea?

davoustp Apr 4, 2022
Author

Got it, here it goes.

# Increase the host Kernel parameters to allow proper JGroups performance
docker run -it --rm --privileged --pid=host justincormack/nsenter1 /bin/sh -c '\
  echo 26214400 > /proc/sys/net/core/rmem_max; \
  echo 26214400 > /proc/sys/net/core/wmem_max'

Then create the first node:

node=1
docker run -it --rm --name keycloak${node} \
  --network keycloak \
  --cap-add SYS_ADMIN \
  -p 808${node}:8080 \
  -p 878${node}:8787 \
  -p 899${node}:899${node} \
  -e KEYCLOAK_ADMIN="keycloak" \
  -e KEYCLOAK_ADMIN_PASSWORD="keycloak" \
  -e KC_DB="postgres" \
  -e KC_DB_URL="jdbc:postgresql://postgres-keycloak.keycloak:5432/keycloak" \
  -e KC_DB_USERNAME="postgres" \
  -e KC_DB_PASSWORD="postgres" \
  -e DEBUG="true" \
  -e DEBUG_PORT="*:8787" \
  -e JAVA_OPTS_APPEND="-Xmx1g \
    -Dcom.sun.management.jmxremote.port=899${node} -Dcom.sun.management.jmxremote.rmi.port=899${node} \
    -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.local. -Djava.rmi.server.hostname="$(hostname)" \
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/keycloak.hprof" \
  quay.io/keycloak/keycloak:18.0.0-SNAPSHOT start-dev \
  --log-level=INFO \
  --proxy=edge \
  --cache=ispn --cache-stack=udp

Wait for the schema to be created, then run the two others nodes, by running the same command in separate shells with node=2 and node=3.

Finally, run a nginx load balancer:

cat > ./nginx-keycloak.conf <<EOF
upstream keycloak {
  server keycloak1.keycloak:8080;
  server keycloak2.keycloak:8080;
  server keycloak3.keycloak:8080;
}

server {
    listen       8080;
    listen  [::]:8080;
    server_name  localhost;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port  $server_port;

    location / {
        proxy_pass http://keycloak;
    }
}
EOF

# Create the container without running it
docker create --name nginx-keycloak \
  --network keycloak \
  -p 8080:8080 \
  nginx:1.21.6-alpine
# Copy the conf file
docker cp ./nginx-keycloak.conf nginx-keycloak:/etc/nginx/conf.d/default.conf
# Finally start it
docker start nginx-keycloak

You should be able to hit http://localhost:8080, logging properly and using the admin console as usual.

Note that the nginx here is setup here to use round-robin, so each request will hit the next backend - this probably defeats the cookie-based approach (unless session are clustered and replicated across nodes?), but I wanted to be in the worst-case scenario to uncover any potential issue.

I first manually checked that any change in realms and roles on one node (each node exposes its own port, so you can by-pass the load balancer here), which properly invalidates previously cached data in the other nodes (refresh shows the updated data on the other nodes).

I ran the realm creation scripts up to 3k realms, no error occurred, and the creation time graph is:

Same linear pattern, with an additional cost around 100 ms - not sure where it comes from. Maybe synchronous Infinispan invalidation across nodes (did not have a look at the caching layer definition here, so this is a wild guess)?

Login into the Admin console with 3k realms using the load balancer is obviously more costly: it takes around 110 seconds.
This is expected because the API requests are round-robin balanced across the 3 code-started nodes, so it basically populates the caches from the database 3 times in this case.
By-passing the load balancer and hitting a single node (similar to what would occur if using sticky session at the LB level), the load time is back to 50 seconds.

pedroigor · 2022-04-07T11:56:57Z

pedroigor
Apr 7, 2022
Collaborator

@davoustp You have been busy :) Indeed, it is very detailed work and we should definitely benefit from it.

FYI, some time ago we did a similar analysis. At that time, the goal was to achieve 5k realms and evaluate the effort needed to achieve that.

In a nutshell, we ended up with a few PRs merged but the main one wasn't. Please, see #7712 and check the description for a bit more details about the changes.

I think the PR above contains some of the suggestions/changes you are doing and that is great.

I would love to resume this discussion and related changes because at that time we gave up on the "multi-realm" story to focus on other things. The main argument was the new store and how it is going to improve this area. Including changes to how we should be doing SaaS.

If we can continue this effort in parallel with our current priorities, that would be awesome.

8 replies

pedroigor Apr 7, 2022
Collaborator

If you mean how you can use the tool, the steps in the readme file should help. If not, we can improve.

And I forgot about something very important. Before moving with this we should probably have @stianst, @mposolda, and @hmlnarik approvals too ...

davoustp Apr 7, 2022
Author

If you mean how you can use the tool, the steps in the readme file should help. If not, we can improve.

I was more thinking about how to use the tool in this specific context (I already scanned the readme). :-)

And I forgot about something very important. Before moving with this we should probably have @stianst, @mposolda, and @hmlnarik approvals too ...

Sure, any step required from me to get their attention on this?

pedroigor Apr 7, 2022
Collaborator

Regarding the tool, you can use the "dataset" provider. This provider allows you to call specific endpoints to automatically provision realms and their data. The endpoints accept some parameters to set the number of realms, clients, users, etc. We can always add more if needed.

W.r.t. to their attention, let's wait for their reply here. The data you provided is much more detailed than what I did in the past so I hope we have more input to justify the investiment.

ahus1 Apr 8, 2022
Collaborator

@davoustp - I will soon be working o 8BD0 n the Red Hat engineering side to validate the performance of the new store. For that we plan to use the keycloak-benchmark project Pedro Igor mentioned above, so contributions to that project or tests that build on top of it are very welcome to ensure the new store will work nicely.

Without getting ahead of the maintainers to reply, big changes in the existing JPA would require extensive reviews and testing to get them approved. With the new store having priority, it will be difficult to find the time.

On the other hand, for the changes that are server-spi level, there is a (short) window once Keycloak 18 has been released to adapt existing interfaces. I wonder if your changes break an existing API as you seem to only add methods.

Therefore, as @pedroigor suggested, I'd rather keep the changes small and one at a time.

davoustp Apr 8, 2022
Author

I will soon be working on the Red Hat engineering side to validate the performance of the new store. For that we plan to use the keycloak-benchmark project Pedro Igor mentioned above, so contributions to that project or tests that build on top of it are very welcome to ensure the new store will work nicely.

Nice! For sure, will do.

Without getting ahead of the maintainers to reply, big changes in the existing JPA would require extensive reviews and testing to get them approved. With the new store having priority, it will be difficult to find the time.

Ok, I get that. The changes are not that big, really, but yes, I do get the concern.

On the other hand, for the changes that are server-spi level, there is a (short) window once Keycloak 18 has been released to adapt existing interfaces. I wonder if your changes break an existing API as you seem to only add methods.

That's correct. I added a few methods, without changing existing ones, and provided default implementations whenever possible to keep things simple for existing code.

Therefore, as @pedroigor suggested, I'd rather keep the changes small and one at a time.
Yep, I got that. Thx!

davoustp · 2022-04-08T12:24:26Z

davoustp
Apr 8, 2022
Author

@pedroigor @ahus1
keycloak-benchmark is a breeze to use, guys. Kudos!

What I did:

cloned and built https://github.com/keycloak/keycloak-benchmark
started keycloak
copied over the provider using:
docker cp dataset/target/keycloak-benchmark-dataset-*.jar keycloak:/opt/keycloak/providers/
restarted keycloak to deploy the added provider
hit the realm provisioning URI:
http://localhost:8080/realms/master/dataset/create-realms?count=3000&groups-per-realm=0&users-per-realm=0&clients-per-realm=0&realm-roles-per-realm=0
(I wanted no add'l role, users or clients to be able to compare with my previous results)

Output from the task with the branch containing the proposed improvements:

2022-04-08 12:11:20,921 INFO  [org.keycloak.benchmark.dataset.TaskManager] (Thread-5)
  FINISHED TASK: Creation of 3000 realms from realm-0 to realm-2999, started: Fri Apr 08 12:04:14 GMT 2022

The creation of 3,000 realms took 426 seconds in my environment.

I suppose that this provider is a more a provisioning facility than a benchmarking-focused endpoint.
There is obviously no metric to measure per-realm creation time (so not able to graph it like I did previously).

Would you suggest some more moves using keycloak-benchmark?

3 replies

pedroigor Apr 8, 2022
Collaborator

Yeah, the dataset is basically about provisioning. There are no metrics or reports from it.

However, the benchmark tool is based on Gatling and gives you more detailed information when checking runtime performance. Not sure if this tool applies to your case though because there we have test scenarios for authentication and a few Admin APIs.

davoustp Apr 8, 2022
Author

Would it make sense to have some light-weight metrics to it (compared to doing a full-blown Gatling scenario)?

pedroigor Apr 8, 2022
Collaborator

That would be awesome and should allow us to evaluate provisioning without introducing the HTTP layer with a focus on the server internals.

saguntumkar · 2022-04-19T07:38:26Z

saguntumkar
Apr 19, 2022

@davoustp I couldn't find your PR related to these changes. What's the final decision? Is your PR merged or we are not merging and waiting for the new store implementation in keycloak 18 to fix the multiple realms issue? I am evaluating the multiple realms vs single realms + multiple groups approach as we would need to create around 20k realms. We are currently in the process of migrating to keycloak 17. So any input from @pedroigor and you would help us making informed decision.

2 replies

davoustp Apr 19, 2022
Author

Hi @saguntumkar,
All the changes are in fork https://github.com/davoustp/keycloak/tree/KEYCLOAK-4593-investigations .
This allows to have a look and check what changed, but will require to be split in various PRs to make to the main branch, as @pedroigor suggested.
We're awaiting from main Keycloak folks to have a look and provide some directions.

sreekesh93 Apr 21, 2022

Hi @davoustp ,
Interested to see your findings, I have a query, after enabling JMX by setting params in JAVA_OPTS_APPEND were you able to connect jmx port? I am facing issue with connection. If it got worked really appreciate your support, a separate discussion is started for this in thread #11503

stianst · 2022-04-22T13:39:09Z

stianst
Apr 22, 2022
Maintainer

@mposolda @pedroigor @hmlnarik please review this and provide some input on how we can move this forward

0 replies

mposolda · 2022-04-27T10:17:09Z

mposolda
Apr 27, 2022
Maintainer

This is awesome effort and great investigation! Your effort for this is appreciated. I have few points below (some probably already mentioned and discussed before):

It will be good to "decompose" the work into multiple PRs. The ideal is as small and isolated PRs as possible, so we can discuss individual points in more details.
There are lots of new additions in the server-spi module in the model SPI. AFAIK these changes need to be discussed and agreed with @hmlnarik and the team working on the new store. We can add them if they makes sense in the context of the new store and are not rather additional complication/variable for that work. The new store is the current priority for the Keycloak team.
You mentioned that all the tests are working for you. Which DB did you tested with? One point here is, that all DB changes need to be tested also with all the databases, which Keycloak currently supports (MySQL, MariaDB, PostgreSQL, Oracle, MSSQL, PostgresPlus). Here in RH, we have internal pipeline and we are able to run the functional tests with all the databases.
I wonder if the optimizations you propose improve performance for all the databases or if there is some chance that some DBs may not be that optimal? I believe that this is not the case and performance benefit is for all the DBs, but not 100% sure about it. For example I have some concerns about the queries like select compositerole from CompositeRoleEntity compositerole where compositerole.compositeId in :roleIds in the case when there is very big amount of records in the in clause (few thousands or so). Actually I am not even 100% sure if this work fine from the functionality perspective. Which raises the more general question if we should have also some "functional" testing with the big amount of records to make sure that refactoring doesn't break any DB vendor in the environments with big amount of records in the DB?
Backwards incompatible changes in the server-spi are always a bit problematic. But fortunately, it seems that your work doesn't have lots of them - I noticed just something in the RoleUtils class. In some cases, we can justify backwards incompatible changes (if really needed) and write about them to migration guide, so I hope that this will be fine. Just FYI.
For the admin console, my vote is towards the refactoring of the "logic" itself rather than trying to optimize everything just at the model layer. AFAIR the problematic part is the WhoAmI when administrator from the master realm is authenticated as this needs to fill the list of all roles of all realms. Instead of this, we can focus just on the realm, which is selected by the current URL (EG. realm foo in case fo the URL like http://localhost:8081/auth/admin/master/console/#/realms/foo/roles is accessed) and fill the roles just for this realm. This should be sufficient to know what roles related to particular realm, the administrator of the master realm has access to. For the "generic" endpoints (EG. server-info endpoint), we can have dedicated role, which administrator from master realm needs to have to be able to access this endpoint (just for the master realm. When administrator from the other realm is authenticated, we can probably keep same behaviour as now). Also we can improve the "realm selector" on the left side of the admin console to be paginated and support searching when admin fills initial letters of the realm name (EG. similar to "User Selector" when creating new User Policy in "Authorization" tab of some client). This will help that it is not needed to "eagerly" resolve all the realms, which the particular administrator has access to, but just a small subset of them.

Regarding this point, I am not saying that we should not optimize at the store layer as well (as optimizing store performance is also beneficial). Just thinking that if we optimize "logic" layer, we will automatically optimize Keycloak for both the old and new store.

Conclusion: I am personally not sure how to move forward. Hopefully we can have some feedback soon from the new store team if the changes in the model SPI (EG. especially new methods for retrieving list of IDs for particular objects and retrieve the list of objects by list of IDs etc) makes sense for the new store. In the meantime, you can try to send some initial PR to the keycloak main with some smaller/isolated subset of your optimizations and we can possibly discuss here being more focused just on the particular/smaller change. In parallel, we can rethink how to optimize the WhoAmI (my point 6) - I wonder it will probably makes sense to postpone once new admin console is the default as it will require some changes in the UI and will be nice to avoid doing them for both new and old admin console.

3 replies

10BC8

davoustp Apr 28, 2022
Author

Hi @mposolda
Thanks for the extensive feedback! 👍
Sorry for the response delay, I was busy elsewhere.

It will be good to "decompose" the work into multiple PRs. The ideal is as small and isolated PRs as possible, so we can discuss individual points in more details.

Indeed, that's the intent. As you mention in your conclusion, I will start splitting the code changes in separated, smaller PRs.
I'll need to do it gradually, since some changes do rely on other changes, so some PRs will be depending on other PRs.
I'll try to make it as simple as possible.

There are lots of new additions in the server-spi module in the model SPI. ...

Ok, got that.

You mentioned that all the tests are working for you. Which DB did you tested with? One point here is, that all DB changes need to be tested also with all the databases, which Keycloak currently supports (MySQL, MariaDB, PostgreSQL, Oracle, MSSQL, PostgresPlus). Here in RH, we have internal pipeline and we are able to run the functional tests with all the databases.

I ran all the integration tests against Postgres. I can probably run them against Oracle and SQL Svr (as we already do that in our own products and CI pipelines), but it probably make more sense to leverage your existing pipeline at RH. Obviously, I don't have access to this pipeline, so what is the best approach here? Just going with the PRs and awaiting for feedback when the PR has been tested in the pipeline?

I wonder if the optimizations you propose improve performance for all the databases or if there is some chance that some DBs may not be that optimal? I believe that this is not the case and performance benefit is for all the DBs, but not 100% sure about it.

I'm pretty sure that all databases will benefit from it, because the change is about how the tables are queried - switching from a 1+N querying pattern to a log(N) + k querying pattern (queries being much more simple and fast in addition to this - these are basically bulk lookups by PK, instead of complex joins).

For example I have some concerns about the queries like select compositerole from CompositeRoleEntity compositerole where compositerole.compositeId in :roleIds in the case when there is very big amount of records in the in clause (few thousands or so). Actually I am not even 100% sure if this work fine from the functionality perspective.

The SELECT ... WHERE x IN (list) is standard SQL and is supported by JPA and Hibernate so I'd be very surprised that we run into a problem here.
But you're totally correct in expecting issues with the number of items into the list of ids provided to the query: each DB engine has its own limitation in this respect. For example, Oracle is 1000 max, Postgres is signed smallint.MAX...
That's what I mention into the initial post:

the JPA implementation must also take care to limit the number of ids to bulk load using a select ... from ... where id in :ids JPA query (as Oracle and other engines do have limit - 1000 for Oracle)

This has been implemented into the code changes, for example at https://github.com/davoustp/keycloak/blob/23e06582ff3b81b29d0e0cdb84eae0f9bcdec5d6/model/jpa/src/main/java/org/keycloak/models/jpa/JpaRealmProvider.java#L532
The nb of items is here capped at 512 items (being a power of 2 to optimize against Hibernate in_clause_padding feature which minimizes the nb of cached statements for these cases).
Now that I have the list of supported DB engines, I can have a quick look for each to see what is their own limits re: the IN clause.

Which raises the more general question if we should have also some "functional" testing with the big amount of records to make sure that refactoring doesn't break any DB vendor in the environments with big amount of records in the DB?

Yes, that would be great, but I would be at loss to propose something here. I did use the keycloak-benchmark provisioning facility to check the overall performance improvement, but this is a far cry from a dedicated, purpose-built integration test...

Backwards incompatible changes in the server-spi are always a bit problematic. But fortunately, it seems that your work doesn't have lots of them - I noticed just something in the RoleUtils class. In some cases, we can justify backwards incompatible changes (if really needed) and write about them to migration guide, so I hope that this will be fine. Just FYI.

Yes, I tried to keep these to the very bare minimal - maybe it can be stripped down even more (will keep that in mind when splitting into individual PRs).

For the admin console, my vote is towards the refactoring of the "logic" itself rather than trying to optimize everything just at the model layer. ...

I fully agree. I didn't want to enter a full change against the UI in addition to the model/storage system, so that's why I went the optimisation route, to ensure that most showstoppers were gone from a scalability perspective.
I think loading a single realm (and related roles) instead of the full realm and role list would be a much better approach - even more with searching and paging.

Conclusion: I am personally not sure how to move forward. Hopefully we can have some feedback soon from the new store team if the changes in the model SPI (EG. especially new methods for retrieving list of IDs for particular objects and retrieve the list of objects by list of IDs etc) makes sense for the new store.

That's probably one the very first PRs I'll extract, so we can discuss it. For sure it would benefit the map-jpa implementation, and probably not so much for the key-value based map stores (in this case, a naive implementation performing the lookups would be enough from a performance perspective). But let's wait for the storage team to provide their feedback.

In parallel, we can rethink how to optimize the WhoAmI (my point 6) - I wonder it will probably makes sense to postpone once new admin console is the default as it will require some changes in the UI and will be nice to avoid doing them for both new and old admin console.

Yep, I think that makes sense to review these changes after the other model / storage PRs have been dealt with, especially if only the new UI is concerned and go for the single-realm loading approach.

Again, thanks for your feedback!

ahus1 Apr 28, 2022
Collaborator

Hi, I'm joining into this discussion for the storage team, and I'm happy to review the PRs from the storage team side of things.

It's good to hear that you'll be testing PostgreSQL, and also look into restrictions that OracleDB would have. Once there is a PR and we see that it works for the GitHub actions, I can run it also against the internal pipeline for the other databases, please ping me for that.

Regarding the SPIs: there is a small window for Keycloak 19 and possibly 20 were a change could be done a little bit easier with regards to backward compatibility, still the earlier the better.

That being said, please ping me (@ahus1) on any PR you create to this. I'd also like to see a PR for the https://github.com/keycloak/keycloak-benchmark/tree/main/benchmark/src/main/scala/keycloak/scenario/admin with a scenario that could be similar to "CreateClients".

davoustp Apr 28, 2022
Author

Hi @ahus1
Thanks for joining ;-)

It's good to hear that you'll be testing PostgreSQL, and also look into restrictions that OracleDB would have. Once there is a PR and we see that it works for the GitHub actions, I can run it also against the internal pipeline for the other databases, please ping me for that.

I will, for sure.

Regarding the SPIs: there is a small window for Keycloak 19 and possibly 20 were a change could be done a little bit easier with regards to backward compatibility, still the earlier the better.

Ok, will do this asap.

That being said, please ping me (@ahus1) on any PR you create to this. I'd also like to see a PR for the https://github.com/keycloak/keycloak-benchmark/tree/main/benchmark/src/main/scala/keycloak/scenario/admin with a scenario that could be similar to "CreateClients".

Will have a look at creating the Gatling scenario.

Thx!

davoustp · 2022-04-29T08:42:17Z

davoustp
Apr 29, 2022
Author

@ahus1 @pedroigor
Just submitted PR to fix issue keycloak/keycloak-benchmark#46 : it prevents current Gatling scenarii to run against Quarkus-based Keycloak.

0 replies

ahus1 · 2022-04-29T09:19:03Z

ahus1
Apr 29, 2022
Collaborator

@davoustp - thank you 🎉 for the PR. I'll have a look; please feel free to mention me directly on PRs.

2 replies

sreekesh93 May 9, 2022

Hi @davoustp ,Any updates on the realm creation PR and will it be part of next keycloak release ?.

davoustp May 9, 2022
Author

Hi @sreekesh93,
I started to split all the changes into individual PRs, and some more to add benchmark scenario to https://github.com/keycloak/keycloak-benchmark/. The first PR showed up with a problem which I'm currently investigating.

ahus1 · 2022-06-03T13:00:19Z

ahus1
Jun 3, 2022
Collaborator

The performance analysis of the PRs showed the slowdown stems from the one-client-per-realm that is added to the master realm and that slows down the evaluation of roles and composite roles.

I'd like to analyze if those roles could be removed, or only kept temporarily to setup an admin-per-realm. Please join the discussion here #12332 to dive deeper into that topic. Thanks!

0 replies

chpshinmarq · 2022-06-17T04:30:11Z

chpshinmarq
Jun 17, 2022

@davoustp Hi is there any guide that we can follow to get your forked branch up and working to local or maybe push it through docker build?

7 replies

< 10BC8 form data-replace-remote-form-target="child-comments-discussioncomment-2968695" data-replace-remote-form="true" data-turbo="false" action="/keycloak/keycloak/discussions/11074/comments/2968695/threads" accept-charset="UTF-8" method="get">

vasyan1992 Dec 16, 2022

Hi @davoustp
I built keycloak from you fork and deployed it with postgres
In my case perfomance for keycloak realm creation and admin console login doesn't look better
Maybe there are some point how to tune it?
I kept 3 pods of keycloak and uploaded up to 400 Realms and after I wasn't able to login admin console

vasyan1992 Dec 16, 2022

This is Dockerfile I used
https://github.com/vasyan1992/Keycloak-from-source/blob/main/Dockerfile

davoustp Dec 19, 2022
Author

Hi @vasyan1992 , could you give a try with the built-in Docker image from the forked repo instead of your own, by following the official build instructions?

This boils down to running the command below from the quarkus directory:

mvn -f ../pom.xml clean install -DskipTestsuite -DskipExamples -DskipTests \
&& cd container/ \
&& cp ../dist/target/keycloak-18.0.0-SNAPSHOT.tar.gz . \
&& docker build --build-arg KEYCLOAK_DIST=keycloak-18.0.0-SNAPSHOT.tar.gz . -t quay.io/keycloak/keycloak:18.0.0-SNAPSHOT \
&& cd ..

then execute the Docker command described at top of this document...
Let me know how this goes.

davoustp Dec 27, 2022
Author

@vasyan1992
How is this going?
I received another notification from another user who had the same issue, so I'm wondering if there is something wrong into the instructions.
So I ran through the instructions again, and ran the script that creates 1k realms: I still get the expected performance gain.

The only thing that I could think of is that you're running the wrong image: the fork builds the quay.io/keycloak/keycloak:18.0.0-SNAPSHOT image locally, so that's the one to use of course, not the quay.io/keycloak/keycloak:17.0.0 image as described above (that was to perform the baseline measurement against the standard Keycloak distrib).

skhro87 Jan 31, 2023

@davoustp that notifcation was maybe from me, I first tried to build and didn't see the performance improvements.
I realized I actually wasn't on the right branch KEYCLOAK-4593-investigations, that's why I didn't see the improvements.
I realized and deleted my comment here. On the right branch, it works!
Thanks for your work on this! <3

nmaster · 2022-07-26T07:55:05Z

nmaster
Jul 26, 2022

Any chance this improvement becomes production-ready soon?

1 reply

chpshinmarq Jul 26, 2022

Up! I think this is still an issue

stianst · 2022-08-25T10:19:40Z

stianst
Aug 25, 2022
Maintainer

@davoustp thanks for the detailed investigation and all the work, as @mposolda pointed out here the next steps is really to start breaking this into smaller issues and corresponding PRs. That will help further discussions and reviews, and we can start getting some of the improvements merged bit by bit.

It may be an idea to create an epic in GH issues that links to individual issues so we can track all the issues that needs to be solved for a full solution to this problem.

Seeing as it's been a while since there's been any updates from you I wanted to ask if you're still looking into this? If you are blocked waiting for us I'll do what I can here to unblock you moving ahead with this.

7 replies

ahus1 Aug 30, 2022
Collaborator

Hi @stianst - I made an attempt to apply some of the ideas to the map store in #13146. After a discussion with @hmlnarik and @vramik we agree that this is possible, but put this on hold until the tree store is available and the team capacity allows for it.

mposolda Aug 31, 2022
Maintainer

@stianst +1 to review if we can improve admin console (WhoAmI etc) and ideally fetch just the roles for the current realm. I suggested something similar above (point 6) some time ago in the comment #11074 (comment)

davoustp Aug 31, 2022
Author

Hi there,
Even if optimizing the UI and related API endpoints to restrict role lookup to a single realm is desirable (in any case) and would help the UI-end of the stick, you'd still have the other side of it to cope with: the same role composite loading performance problem happens when creating realms using UI or API (and importing realms as well, btw, since this is the same underlying code).
So the scalability issue would still remain...
Cheers

mposolda Sep 1, 2022
Maintainer

@davoustp When creating realms, you currently need to have create-realms role of the master realm. So if we simplify in a manner that for example:

Realm master have only single role admin . Guy in that role has permission to manage anything in any realm and also permission to create/delete realms. So like "admin of the world" . There won't be any other roles in the master realm and not thousands of composite roles like now
When someone want fine-grained permission to administer individual realms, you need to have account in the realm itself.

In other words, no possibility for fine-grain permissions from the administrators of master realm. AFAIK that would work as the master realm is the only realm, which is causing troubles. The other individual realms have just a few roles to manage just the realm itself (roles of the realm-admin client of individual realms).

sschu Sep 2, 2022
Collaborator

@mposolda That's not compatible with the scenario we are using Keycloak in. We provide Keycloak instances to our internal customers. We give our customers rights to manage their own realms from the master realm. At the same time, they cannot change anything in the master realm. This way, they cannot shoot themselves in the foot by making configuration changes in their realms - they will always have their guaranteed realm admin users we give them. At the same time, when troubleshooting issues, we use read-only users in the master realm with access to the customer realms, so we don't accidentally modify data. So the ability to have per-realm manageability from the master realm is really important for us.

matt-rauch · 2023-02-21T09:16:34Z

matt-rauch
Feb 21, 2023

@davoustp @stianst
Are there any plans to bring the improvements to production soon? We would need the scalibility improvements soon, since our customer-base is starting to grow steadily in the last months.

2 replies

matt-rauch May 10, 2023

@davoustp @stianst @mposolda @pedroigor @hmlnarik
Sorry for the multi-mention, but I'm desperately waiting for an answer on my original question:

Are there any plans to bring the improvements to production soon? We would need the scalibility improvements soon, since our customer-base is starting to grow steadily in the last months.

davoustp May 16, 2023
Author

Hi @matt-rauch ,
As to your question, I'm still monitoring this scalability issue, but no progress so far that I'm aware of.
As far as I understood, some of the optimization ideas from this initiative may be brought to the new store which is also bringing additional goodies (such as zero-downtime upgrades), but I've got no visibility onto this - am kind of blind to what's happening here ;-)
I suspect running the benchmark scenario introduced with keycloak/keycloak-benchmark#60 against a KC with the new store would help check how far it gets...

kkcmadhu · 2023-05-16T12:03:29Z

kkcmadhu
May 16, 2023

has any one tried the new store, to see how much of this is address in the new store?
last time i tried the new store, there were still some performance issues around it e.g #15174

Will give it a try again when i find time.. but i am too eagerly looking for scaling keycloak natively and for zero down time upgrades.

Presently, i scale by having multiple keycloak clusters, and putting a front proxy (envoy proxy) before all my keycloak clusters to that to external world it all looks like one single keycloak instance :)

8 replies

matt-rauch Aug 4, 2023

@kkcmadhu Your workaround with the front load balancer sounds interesting and we consider to implement it ourself.

Could you elaborate how you configured the routing at the front load balancer to make multiple keycloak clusters look like one?
Are there any other restrictions to that approach?
What is the maximum number of keycloak realms you host per cluster?

kkcmadhu Aug 4, 2023

I support about 8k realms, i pack about 300 to 500 realms per keycloak and run them behind envoy proxy.
every keycloak cluster ( we call it a farm) has 2 nodes, and becomes a envoy cluster endpoint..
we use envoy java control plane to dynamically add routes etc.. Been in prod for about 3 years and servers us well..

Some major challenge are

there are more than one master realm now, and to create a new realm you wll have to dispatch the request to specific keycloak (known upfront), or get to specific keyclok and create realm there.
once new realm is created one needs to configure your envoy and add route for the newly provisioned realm /like wise remove it after deprovision.
the static identifier for each keycloak defers( after keycloak 17 i guess).. so you may need spl routing rules for static resources.
You are now managing multiple keycloak, version upgrades could be trick... we choose to upgrade all our keycloak farms all at once. ( else you might end up maning different versions of keycloak , all under single url at once, and api compatibility issues could crop up).

Certainly cumbersome to begin with , but once you automate things its easy sail..
On latency, i did not see any big issues because of the extra envoy hop..

JohnPolansky Aug 4, 2023

@kkcmadhu

Thanks for sharing the details of your setup we've actually done something similar in our design with multiple keycloaks routing the requests to the correct keycloak. However we've not been able to achieve 300-500 realms per keycloak we are seeing "slowness" with loading keycloak UI and authentication at around 150 realms. I'm starting to think that other factors weigh on this like the number of users/clients/roles per realm may weigh in on the slowness. I was curious if you don't mind sharing:

What CPU / Mem per node are you using? We are currently using 2 cpu x 4 GB of ram per node. we tried increase this but didn't notice any reasonable improvement in performance when facing the slowness
When you login to the admin console and the browser is loading the initial page with all the realms how long does this take on your larger keycloaks? We are seeing 5-10 second load times which is manageable but again we have a fraction of the realms you do. If we go with more realms that time gets slow and even the standard authentication times get longer.
How do you know when you've reached the limit of realms per cluster? Do you have a calculation or do you just judge it on slowness?

We are in the process of upgrading to Keycloak 21 and are hoping there might be some improvement we are currently on Keycloak 16.

Thanks for your time,
John

kkcmadhu Aug 7, 2023

I have variety of cpu/mem based on customer segments, we use k8s, and our nodes are large as we have many other services running along with keycloak, typically we allocate a max of 2 core and 8 gig for large deployments and 1 core , 4 gig for others. At times we have upto 3 pods in a kecloak cluster (generally 2 for HA). we disable stickiness.
For static resources, i pool all of them into single envoy cluster and let all the static resource request go to any of the keycloak deployments ( its not much helpful but comes handy at times when load is high).
If you can leverage cloudflare/ cloud front etc those to help in static page access a bit(but again not very useful) in my opinion.

Our typical realm may not have more than 500 users, 1 or two identiy provider, and not more than 15 clients .. so never ran into the issues which you mentioned.

We restricts/ avoid using they keycloak console as much as possible and try to do stuff using api (in our case) we disable keycloak console access to customers too only our internal ops team uses keycloak admin console (that too very restricted). typical things like chaning url of client etc we do it using apis.

My observation is after keycloak 17+ there is significant degradation in realm creation time, dead locks (when realm creation and deletion ) is in progress at the same time.
In fact we saw the time to get token also increase, but thats not very high..

Typically the realm creation time doubles for every 100 realms in my opinion.. and there are about 50 ms extra for each each token requests , when you add about 50 realms or so..

In my opinion, if you avoid using keycloak admin console, and have some kind of retry mechanisms for your realm creation deletion etc. you will be able to handle a part of problem. But its certainly significant effort "outside" keycloak and takes a little time to build and mature.. and needs constant maintenance as and when keycloak are upgraded.

Answering your specific questions :

What CPU / Mem per node are you using? We are currently using 2 cpu x 4 GB of ram per node. we tried increase this but didn't notice any reasonable improvement in performance when facing the slowness

we use much bigger box 4 cpu, 8 or 16 gig k8s node, but its not dedicated to keycloak alone, we run variety of serivce there.

When you login to the admin console and the browser is loading the initial page with all the realms how long does this take on your larger keycloaks? We are seeing 5-10 second load times which is manageable but again we have a fraction of the realms you do. If we go with more realms that time gets slow and even the standard authentication times get longer.

Admin console, we avoid as much as possible and rely on the apis, We do not use cli much as you will have to log on the nodes, but in my opinion using the api are much better trade off (cli needs pod access, webconsole has known issues).

How do you know when you've reached the limit of realms per cluster? Do you have a calculation or do you just judge it on slowness?
From our past perf test, we know realm creation time doubles every 100 realms, We want realm creation to be in less than 5 mins, Realm deletion/is a batch process for us, our SLA is keycloak uptime, and decent response time for token/login calls, we found that the magic number is any where around 300 to 500 realms per keycloak for decent performance of login, since we have envoy in front, all our routes are persisted to a db, whenever a new route is added , we start throwing alerts when keycloak is 80% full ( say 400 realms), and we provision a new one, or aggressively cleanup "

I wouldn't say its super effective, but keeps us going, without burning midnight oils.

JohnPolansky Aug 7, 2023

Thank you for the detailed comments. We do use the API for automation like creating deleting realms, clients, users.. The Admin Console is mostly for troubleshooting and manual effort. Thank you for the details on the keycloak 17+ upgrade slowness we will keep an eye out.

amdev74 · 2023-06-21T13:35:49Z

amdev74
Jun 21, 2023

I'm using this keycloak discussion about scalability issues to point out that there is also issues on generating user's claim in /userinfo when there is thousands of roles in the realm and especially composite 10BC8 roles. Even with just one realm.

More info in this ticket

#12885

0 replies

stianst · 2023-06-29T14:29:12Z

stianst
Jun 29, 2023
Maintainer

Thinking out load around many realms scalability in admin console. There seems to be two primary issues:

Listing realms due to realm drop-down - solution here I think is obvious and add pagination for the realms + some search/sever-side filtering
WhoAmI endpoint - honestly don't know why it's adding roles for all realms, as the admin console is only displaying one realm at a time. So perhaps this can be relatively easily resolved by the whoAmI endpoint returning only roles the admin has for the currently selected realm

There's a few other things that may be issues with scaling large amounts of realms, but fixing those two issues should be a good start.

5 replies

twwd Jun 29, 2023

For the second point, there already exists the PR #8851

twwd Jun 29, 2023

It is probable not compatible to the new admin console 🤔

stianst Jun 29, 2023
Maintainer

Cool, as a first little thing then let's try to get that reviewed/merged :)

stianst Jun 30, 2023
Maintainer

If anyone here wants to take a look at updating #8851 that'd be more than welcome :)

kkcmadhu Nov 14, 2023

how is it different from whats already being offered by phaseII as keycloak extension?
https://github.com/p2-inc/keycloak-orgs

kkcmadhu · 2023-07-04T05:38:58Z

kkcmadhu
Jul 4, 2023

Also in the context of large number of realms, its just not how many number of realms we are able to support. but the concurrency also is a big concern, i am not able to create more than 2 or 3 relams at a time when i have more than 100 realms,

Also , We get into super ugly dead lock problems if create and delete realms are in progress at the same time.
Look here

4 replies

sschu Jul 4, 2023
Collaborator

I would assume realm creation and deletion to happen quite rarely. I am curious what is your use case where this is happening often?

danielFesenmeyer Jul 4, 2023

Don't want to answer for @kkcmadhu. But like to mention what I have observed: That realm creation/deletion issue is a problem when running automated tests in parallel (against the same keycloak instance). The main problem is deleting test realms when other - even unrelated tests on other realms - are still running.

kkcmadhu Jul 4, 2023

Yes ,we too see it in automated test cases.
Besides, in I run a SaaS application and for isolation i allocate 1 realm per customers, when their trial periods expire we cleanup realms,
this is at time done in batch too. and the same time there are customers who are trying to signup (and hence create realms).

Agreed its not as frequent as login/token calls , but not improbable and in reality we have seen it quite a lot.

sschu Jul 4, 2023
Collaborator

Thanks, that makes sense.

stianst · 2023-07-26T10:27:58Z

stianst
Jul 26, 2023
Maintainer

We are considering adding an organization concept within realms. An organization will be something you can associate users and clients to, probably have a organization level admin, ability to link IdPs to, etc.

I wondering if that would solve most of the needs for having larger amounts of realms and that we may want to focus on such an org concept more than supporting 1000s of realms?

5 replies

kkcmadhu Jul 26, 2023

While org is a nice concept and helps in few use cases, it will not help SaaS applications which are multi tenant in nature.
To provide maximum isolation , We actually onboard each of our customer as a separate realm and let them manage their users, use-role association, Identity providers alone
All our SaaS offerings are actually clients in customers realm..
Say if we have 10 offering those become 10 clients. and a customer can do SSO across our product lines..

So org would not benefit use case where tenant isolations are priority like ours.

matt-rauch Jul 26, 2023

I fully agree to @kkcmadhu 's statement. It would not help us with our B2B SaaS service aswell, because our clients need the realm level isolation. In markets like Germany it is actually a hard requirment by the BSI (Bundesamt für Sicherheit in der Informationstechnik to provide this kind of isolation, when you service governmental clients.

MgLionel Jul 27, 2023

For us, both solutions (organization/multiple realms) would be applicable.

kkcmadhu Nov 14, 2023

org support exists in keycloak extension https://github.com/p2-inc/keycloak-orgs

SilentGert Nov 29, 2023

To chime in on this discussion. We also use Keycloak in a B2B SaaS setup and create a separate realm per customer. The application runs on a different sub domain for each customer. So that would make our client setup interesting, because they would have to support either a long list of redirect URIs (not sure if I want a wild card here) or we would have to come up with a different naming pattern to ensure clients can live next to each other. At the moment they use the same alias in different realms.

Furthermore, we have differences in authentication flows and the login screen depending on the customer needs. Authentication flows could potentially be reused for different pattern, but would separate login screens work?

To me it sounds like either the organization concept is not sufficient for some of our use cases or it becomes another realm separation just with a different name. Then I wonder if the time is not better invested in improving the existing realm separation.

thomasdarimont · 2023-07-26T11:17:28Z

thomasdarimont
Jul 26, 2023
Collaborator

It makes sense to have support for both.

SaaS Provider Use-case: Usually, when one company provides a B2B (or B2B2C) SaaS offering, then the individual "business partner tenants" can be represented as an organization.
Keycloak Hosting Use-case: However, if you need to provide Keycloak hosting for different companies, which are not related to one another, it might be necessary to use dedicated realms. Providing a dedicated Keycloak system per company is possible but IMHO, in many scenarios a considerable waste of resources.

From my experience, use-case 1) is more common than 2) and should be prioritized if possible.

2 replies

kkcmadhu Jul 27, 2023

in my opinion, org is fairly stright forward to acheive (even today) with keycloak connect (identity broker) and few broker/identity mapper and few claims mapper while printing jwt token.. which is what we do to realize org structure presently.

I kind of get @stianst point of view, i think he is trying to emphasis master realm is already some kind of org structure where one can go ahead and create realms from master realm and master relam has clients representing each of the actual realm (xxx-realm-management) .. (and possibly most performance problems originates there)

But this master realm/ super admin relam is spl in nature, i dont think bring a full fledged "org" capability is a replacement of all realms and resource with in realms being managed from master realm.

Also.. With org, one need to consider about the token size.. even today, the master realm token tends to get really huge if you have lots of realm in your key cloak deployment.. The same could actually happen with org too..

I agree with other that "org" is totally different from complete realm isolation (and many of us will still need/heavily depdend on realm level isolations which keycloak presently offers)...

I get the point that "org" could make things easy from an operations point of view, however I am not really able to understand how "org" could actually solve the inherent performance issue with respect to large number of realms, and increase in token size and the dead lock issues and high latency in realm creation/deletion when a keycloak instance/cluster already has more than 50 or 100 realms in it.

pedroigor Nov 29, 2023
Collaborator

From my PoV, we need and can improve multi-realm support to at least something better than today. Several attempts were made in the past but we never moved forward. There are some low-hanging fruits we can get from these past initiatives as they attack well-known issues when you have a multi-realm deployment.

IMO, without physical isolation the scalability of deployments will always be a problem, and depending on their realms workload you will end up with too many gears assigned to deployment so that CPU, RAM, Network, Database, and so forth are going to be pushed to their limits. For instance, in an HA deployment with an Active/Passive(Active) that could be even worse to maintain due to the amount of data going between the different sites, etc.

My point is, that scalability should be relative and constrained regardless. However, we definitely have improvements to today's runtime when doing multi-realm. These can perhaps be enough to solve most of the use cases discussed.

ChBerrich · 2023-11-08T09:14:16Z

ChBerrich
Nov 8, 2023

We face also this issue on our platform, do you have any plan to solve/improve it soon ?

4 replies

pedroigor Nov 29, 2023
Collaborator

Yes, one of the key goals for 2024 is multi-tenancy. If we are going to focus on the org concept or support multiple realms, it is still unclear. More likely we will start with org due to some hard requirements we have in our backlog.

ghfalcon7 Dec 21, 2023

any ETAs we can look forward to? we are getting to the point of evaluating alternatives to keycloak because of this but the migration effort would very high and we would rather avoid it if this issue will be solved soon.

UXabre Jan 29, 2024

@pedroigor and @stianst , both of you seem to mention an "org concept" but to me it's kinda vague what this concept looks like? Is there a page or something that describes what it is compared to multi-realm? Perhaps we can also assist with some PRs to help push this forward if we'd only know what it is and where this org concept is tracked (if anybody is already working on it at all)

thomasdarimont Jan 29, 2024
Collaborator

There are some discussions ongoing around supporting dedicated org structures within a realm, see:
#19055
#23948
p2-inc/keycloak-orgs#45
#19054

lordvlad · 2023-11-29T13:10:23Z

lordvlad
Nov 29, 2023

For what it's worth, I've got a PoC for a 'sharded' keycloak which provides a way to scale keycloak horizontally beyond 200 realms. The PoC has a single KC & DB for every shard, but you can easily use a kc cluster and replicated db for every shard. https://github.com/lordvlad/keycloak-host-and-realm-chooser

Not a full solution but it might give some people an idea to build upon. I'm not very proud for messing with the redirect_uri, maybe theres a better option.

1 reply

indera Aug 21, 2024

The fact that we have to shard is so sad :(

jarski · 2023-12-08T07:35:42Z

jarski
Dec 8, 2023

Does the performance improvements here get priority now as the map store is discontinued? I think the low scalability of realms is the toughest challenge with Keycloak at the moment.

3 replies

indera Aug 21, 2024

Admin console UI loading issue is supposed to be fixed with pagination changes here (fresh from the press)

#32218

And one more improvement was done
#21072

jarski Oct 15, 2024

That's nice. Any chance that the peformance of tenant creation would be improved too?

rspu Oct 22, 2024

We would also be very interested in better scalability with many realms. Although the new organization feature is certainly a good addition to Keycloak it covers IMO a different use case. If you operate Keycloak as part of a SaaS solution you most probably want complete separation of your customers, e.g. separate user pools.

Without having a detailed look into all of the amazing work the author of this thread has already done it seems to me he found out the problems and a way to fix them. So maybe there are some low hanging fruits and less risky changes that would at least improve the situation. Otherwise I also don't see a different solution than sharding the realms to multiple Keycloak instances like @lordvlad suggested.

dmpv-cyber · 2025-04-10T13:39:20Z

dmpv-cyber
Apr 10, 2025

Hi,

While investigating a solution for the keycloak scalability issue we found out about this issue, and the great work that @davoustp have done on this investigation, sadly it was already on an older version of keycloak and the fix is not easily applied to the latest keycloak version.

While looking into the solution that @davoustp purposes, we found out that if we increase the cache settings, like it was did on the original implementation, to be higher than the ones that are default namely this environment variables:

KC_CACHE_EMBEDDED_REALMS_MAX_COUNT to 200000
KC_CACHE_EMBEDDED_USERS_MAX_COUNT to 20000
KC_CACHE_EMBEDDED_AUTHORIZATION_MAX_COUNT to 20000

Keycloak becomes usable again, we can navigate through the admin console without huge delays even with more than 1k realms init.

What we would like to know is that if we can except issues to happen because of having that huge increase on the cache, other than the memory increase by keycloak.

Thank you in advance.

10 replies

ahus1 Apr 11, 2025
Collaborator

Thanks. To add to the cache sizes: The realm cache holds realm related data. This includes also clients, client scopes, groups, roles etc. As it caches those values by ID and name, one entity might take up multiple cache entries, and it might depend on your usage of Keycloak on how you access the entities.

So you should size it that the actively used entities with in a time frame are cached. So if your list of actively used clients, increases, you should increase your cache size. It quite much depends on the instance's usage patterns and datasets, so general advice is difficult

The general rule might be: Look at the eviction cache metric and the cache entries size: If the eviction is high and the cache entries level is also high, increase the cache size. If the cache entries level is low, consider and the eviction is low, reduce the cache size. There might be other patterns where entries are remove a lot of the time which I've seen when people update their realm every second or so from a customer extension that would then lead to clearing it from the cache and causing a lot of slow requests.

ahus1 Oct 5, 2025
Collaborator

I did one test today to scale Keycloak to 1000 realms with 26.4, and didn't see much of a degradation.

As long as you keep the realm cache in-step with about 50 entries per realm, the performance seems be good. A quick test in the Admin UI didn't show any REST endpoints that stalled due to reading too many entries from the DB, including the realm selector.

In my test one realm used ~20 entries in the realm cache, but better use 50 entries once you start doing more work on them / start authenticating users.

I suggest you start with a very large cache, then look at the metric vendor_statistics_approximate_entries to see what you ratio or cache entries per realm is depending on your usage. See https://www.keycloak.org/observability/metrics-for-troubleshooting-embedded-caches for more metrics.

wcote-kz Dec 1, 2025

Hi @ahus1 can you explain to me what you are referring to as "in-step" here:

As long as you keep the realm cache in-step with about 50 entries per realm

I don't know if or how we can configure cache entries as for the moment I use

KC_CACHE_EMBEDDED_REALMS_MAX_COUNT: 200000
KC_CACHE_EMBEDDED_USERS_MAX_COUNT: 20000
KC_CACHE_EMBEDDED_AUTHORIZATION_MAX_COUNT: 20000

as suggested above.

thank you

ahus1 Dec 1, 2025
Collaborator

@wcote-kz - please read "in-step" as "when you increase the number of realms, increase the realms cache".

Given my previous comment the 200000 entries in the realms cache should be the lower bound for an initial sizing if you have ~4000 realms. As always, please use a load test and the metrics we provide to determine if you need to increase your caches.

wcote-kz Dec 2, 2025

@ahus1 Ok I see, do entries refer to realms count, users count and auth count? I'm trying to understand how to apply the 50 entries per realm suggestion with the KC_CACHE_EMBEDDED_REALMS_MAX_COUNT, KC_CACHE_EMBEDDED_USERS_MAX_COUNT and KC_CACHE_EMBEDDED_REALMS_MAX_COUNT keys.

Improved scalability over number of realms #11074

Uh oh!

Uh oh!

davoustp Apr 1, 2022

Reproducing the issues

Running Keycloak

Increasing access token lifespan

Measuring tenant creation duration

Measuring Admin console load time

Measures with version 17.0.0

Realm creation time with version 17.0.0

Admin console load time with version 17.0.0

Findings and proposals

Adding/removing children roles to/from parent role triggers full JPA collection load

Role composition suffers from N+1 load issue

Bulk-loading large collections by ids to optimize caching layer hit ratio

Accessing entities just for grabbing their ids

Loading large JPA dataset clutters the persistence context and degrades flush-time efficiency

Investigation outcome

Realm creation time with optimisations (forked from main 18.x branch)

Admin console load time with optimisations (forked from main 18.x branch)

Code changes

Replies: 24 comments · 83 replies

Uh oh!

Uh oh!

thomasdarimont Apr 2, 2022 Collaborator

Uh oh!

davoustp Apr 3, 2022 Author

Uh oh!

Uh oh!

thomasdarimont Apr 3, 2022 Collaborator

Uh oh!

Uh oh!

davoustp Apr 3, 2022 Author

Uh oh!

stianst Aug 25, 2022 Maintainer

Uh oh!

Uh oh!

laskasn Apr 4, 2022

Uh oh!

davoustp Apr 4, 2022 Author

Uh oh!

Uh oh!

davoustp Apr 4, 2022 Author

Uh oh!

pedroigor Apr 7, 2022 Collaborator

Uh oh!

pedroigor Apr 7, 2022 Collaborator

Uh oh!

davoustp Apr 7, 2022 Author

Uh oh!

pedroigor Apr 7, 2022 Collaborator

Uh oh!

ahus1 Apr 8, 2022 Collaborator

Uh oh!

davoustp Apr 8, 2022 Author

Uh oh!

davoustp Apr 8, 2022 Author

Uh oh!

pedroigor Apr 8, 2022 Collaborator

Uh oh!

davoustp Apr 8, 2022 Author

Uh oh!

pedroigor Apr 8, 2022 Collaborator

Uh oh!

Uh oh!

saguntumkar Apr 19, 2022

Uh oh!

davoustp Apr 19, 2022 Author

Uh oh!

Uh oh!

sreekesh93 Apr 21, 2022

Uh oh!

stianst Apr 22, 2022 Maintainer

Uh oh!

mposolda Apr 27, 2022 Maintainer

davoustp
Apr 1, 2022

Replies: 24 comments 83 replies

thomasdarimont
Apr 2, 2022
Collaborator

davoustp Apr 3, 2022
Author

thomasdarimont Apr 3, 2022
Collaborator

davoustp Apr 3, 2022
Author

stianst Aug 25, 2022
Maintainer

laskasn
Apr 4, 2022

davoustp Apr 4, 2022
Author

davoustp Apr 4, 2022
Author

pedroigor
Apr 7, 2022
Collaborator

pedroigor Apr 7, 2022
Collaborator

davoustp Apr 7, 2022
Author

pedroigor Apr 7, 2022
Collaborator

ahus1 Apr 8, 2022
Collaborator

davoustp Apr 8, 2022
Author

davoustp
Apr 8, 2022
Author

pedroigor Apr 8, 2022
Collaborator

davoustp Apr 8, 2022
Author

pedroigor Apr 8, 2022
Collaborator

saguntumkar
Apr 19, 2022

davoustp Apr 19, 2022
Author

stianst
Apr 22, 2022
Maintainer

mposolda
Apr 27, 2022
Maintainer