Performance tuning recommendations? (Quarkus vars or elsewhere) #39233

xgp · 2025-04-25T20:20:07Z

xgp
Apr 25, 2025

( cross posting from https://keycloak.discourse.group/t/quarkus-performance-tuning/29935 )

Does anyone have Quarkus performance tuning recommendations for Keycloak (v26)? I’ve been running the keycloak-benchmark AuthorizationCode scenario on a well-provisioned cluster, and it frequently falls over with the below error even with the Keycloak cluster CPU load average below 20% and the database below 15%.

Curious if anyone here has had success with Quarkus performance tuning, as it seems like it’s just vertx resource exhaustion. E.g.

# Event loop threads: 2 per core (non-blocking IO, HTTP, etc.) - Keycloak default 2 per core
quarkus.vertx.event-loops-pool-size=16
# Core blocking threads: 2 per core - Keycloak default 1 per core
quarkus.thread-pool.core-threads=16
# Max blocking threads: 5–8 per core, if needed - Keycloak default 20
quarkus.thread-pool.max-threads=64
# Queue size for blocking tasks: prevent unbounded growth under load - Keycloak is unbounded (which isn't good for backpressure)
quarkus.thread-pool.queue-size=4096

For reference, this is the error I’m getting:

2025-04-25 19:11:27,380 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-422) Uncaught server error: java.io.IOException: java.nio.channels.ClosedChannelException
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.readBlocking(VertxInputStream.java:256)
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.readIntoBuffer(VertxInputStream.java:123)
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:85)
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:73)
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.read(VertxInputStream.java:64)
        at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler$CapturingInputStream.read(FormBodyHandler.java:132)
        at java.base/java.io.InputStream.read(InputStream.java:287)
        at java.base/java.io.InputStream.read(InputStream.java:220)
        at org.jboss.resteasy.reactive.server.core.multipart.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:235)
        at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:95)
        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:635)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1505)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.nio.channels.ClosedChannelException
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream$VertxBlockingInput.<init>(VertxInputStream.java:184)
        at org.jboss.resteasy.reactive.server.vertx.VertxInputStream.<init>(VertxInputStream.java:39)
        at org.jboss.resteasy.reactive.server.vertx.VertxResteasyReactiveRequestContext.createInputStream(VertxResteasyReactiveRequestContext.java:249)
        at org.jboss.resteasy.reactive.server.core.ResteasyReactiveRequestContext.getInputStream(ResteasyReactiveRequestContext.java:719)
        at org.jboss.resteasy.reactive.server.handlers.FormBodyHandler.handle(FormBodyHandler.java:92)
        ... 10 more

xgp · 2025-04-25T20:35:33Z

xgp
Apr 25, 2025
Author

Also, reading in the HTTP section of the Config doc, there is a http-pool-max-threads, which seems similar to quarkus.thread-pool.max-threads. Does it actually just use the same variable under the covers? Since Keycloak does a lot of blocking things (e.g. Hibernate), why is this only 50 by default?

http-pool-max-threads

The maximum number of threads.

If this is not specified then it will be automatically sized to the greater of 4 * the number of available processors and 50. For example if there are 4 processors the max threads will be 50. If there are 48 processors it will be 192.

CLI: --http-pool-max-threads
Env: KC_HTTP_POOL_MAX_THREADS

6 replies

xgp Apr 28, 2025
Author

@shawkins Thanks for taking the time to respond. I think it was the change of the default that caused the change in behavior of the my benchmarks, because I had not originally set the http-pool-max-threads (or maybe it wasn't available in a previous version). I'm curious what your assumptions were and how you quantified/validated this choice. Did I miss something I clicking through the linked issues?

200+ connections / node is a lot of database connections in a clustered setup.

In my case allocating 1000-2000 database connections to Keycloak was not an issue. I did a lot of permutations of configuration, node size and number, and I found that (at least for the AuthorizationCode benchmark) having the same number of http-pool-max-threads and db-pool-max-size configured , gave the best performance.

200 seems stupidly large ;)

Yet another dogmatic statement, totally free of context. (sigh) Did you do testing to validate that 50 was the non-"stupid" number, given that there are up to 100 database connections in the pool by default, or was this statement alone the guiding light for changes?

shawkins Apr 28, 2025
Collaborator

I'm curious what your assumptions were and how you quantified/validated this choice. Did I miss something I clicking through the linked issues?

The recommandation was inline with the keycloak performance tests that are run, and is somewhat referenced in the scaling guide https://www.keycloak.org/high-availability/concepts-threads It isn't fully articulated there, but the downsides of higher thread / connection counts are: increased context switching overhead, increased memory / stack space usage, and as mentioned in the prior response potentially too many connections to the database from a keycloak cluster.

Keep in mind also that this change was also made in the context of limiting the memory footprint of argon2 hashing - we can only let a relatively small number of threads perform hashing at once. Additional threads won't lead to higher hash rates.

In my case allocating 1000-2000 database connections to Keycloak was not an issue.

That's great, it may not be the situation for all deployments.

I did a lot of permutations of configuration, node size and number, and I found that (at least for the AuthorizationCode benchmark) having the same number of http-pool-max-threads and db-pool-max-size configured , gave the best performance.

That is expected and also mentioned in the docs - most keycloak work will hit the database, if there are too few connections you'll just introduce contention / 503 errors on the worker threads. In that regard the old default of 200 workers should certainly seem too high, given the default of 100 connections.

Yet another dogmatic statement, totally free of context. (sigh)

cc @stianst

Did you do testing to validate that 50 was the non-"stupid" number, given that there are up to 100 database connections in the pool by default, or was this statement alone the guiding light for changes?

I recall there being discussion as well about possibly changing / lowering the default database connections, but I don't believe this was done since that was seen as a more likely causing a breaking change. However no follow-up was captured to look more into this.

cc @vmuzikar @ahus1

ahus1 Apr 28, 2025
Collaborator

Please share additional details on how you configured you DB connections - note that initial = min = max connections is recommended, see https://www.keycloak.org/high-availability/concepts-database-connections for the reasoning.
Also it would be good if you added other specific JVM options around GC and/or heap.

It would be important to find the current bottleneck of the system. Unfortunately looking only at CPU of KC or the DB isn't sufficient.

So I'd recommend to start with:

Gradually increase the load and see if the DB connections are actually used - you would normally not need more DB connections than you have http workers.
The bottleneck might be on the load driver system. Depending on the network stack we usually see a limit of 200 connections per second when the network stack of the load load driver is overloaded with CLOSE_WAIT and no new sockets can be opened. If you have proxies in in the middle, something similar can happen with the proxies unless they tune their network stacks differently. The key to that is horizontally scaling your load drivers. There is --share-connections that can work around that, but while you then have more RPS with a single driver, you won't be able to measure the TLS overhead.
Check what Keycloak is actually doing, by pulling either thread dumps or enabling tracing of a (small) percentage of calls.
Even on a well-provisioned cluster, ensure that you have cpu request/limit set properly. Also double-check the CPU throttling ... the more threads you configured for the Pod, the more like it is that Kubernetes is throttling your Pod when all threads become active at the same time (and Kubernetes doesn't like this if you have a CPU limit).

xgp Apr 28, 2025
Author

@shawkins @ahus1 Thank you for the thoughtful answers.

recommandation was inline with the keycloak performance tests that are run

Did you document full setups for the performance tests you run/ran? And publish the results? Per a lot of the points/questions from @ahus1 I'm curious how your setup compares to mine (and others), and what conclusions, or further questions can be inferred from the differences.

In that regard the old default of 200 workers should certainly seem too high, given the default of 100 connections.

Right. I'm asking how you quantified that they shouldn't be the same defaults. Sounds like a part of the justification is argon hashing, which makes sense, assuming that's a big part of the load profile you're setting the defaults for.

It isn't fully articulated there, but the downsides of higher thread / connection counts are: increased context switching overhead, increased memory / stack space usage

Understood. I'm trying to quantify these against time spent waiting for the database. I'd suggest adding that to the document you linked, as a lot of people with a load issue are going to try to understand the tradeoffs.

Please share additional details on how you configured you DB connections - note that initial = min = max connections is recommended,

initial = min = max = 200 and workers = 200

Gradually increase the load and see if the DB connections are actually used

Yes. This is what I did in my iteration. Load (again, for that AuthorizationCode scenario) tends to ramp up the DB connections and http worker threads used in rough parity.

you would normally not need more DB connections than you have http workers.

Right. The disparity in the defaults was confusing to me (100 > 50).

Thanks for all of the other suggestions, as I hadn't considered some of them. In general, it is hard to divine the meaning and underlying justification of some of the suggestions that ultimately make it into the "Guides" section of the documentation. If you are circulating it internally (at Red Hat, or among the maintainers), I'd encourage you to publish more so the community can have a fuller understanding of context.

ahus1 Apr 28, 2025
Collaborator

I usually see that Keycloak is CPU bound due to password hashing, and DB connection bound. Maybe there is a special situation when all workers are busy, and it needs an extra DB connection to write persistent sessions to the database, or use a nested transaction for some case. To be safe, please increase the initial = min = max = 220 and monitor that the DB connections are not saturated.

If the DB becomes a bottleneck, I would be curious what the DB activities are. How many concurrent users / clients are you testing with, and how did you configure the local cache sizes?
Either using tracing, or thread dumps would reveal the slow activities. If a DB connection is opened during a request, they are kept open until the end of the request. If Keycloak is accessing a slow LDAP, this can lead to those symptoms.

Navigation Menu

Repository navigation

Performance tuning recommendations? (Quarkus vars or elsewhere) #39233

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Performance tuning recommendations? (Quarkus vars or elsewhere) #39233

Uh oh!

xgp Apr 25, 2025

Replies: 1 comment · 6 replies

Uh oh!

xgp Apr 25, 2025 Author

Uh oh!

xgp Apr 28, 2025 Author

Uh oh!

shawkins Apr 28, 2025 Collaborator

Uh oh!

ahus1 Apr 28, 2025 Collaborator

Uh oh!

xgp Apr 28, 2025 Author

Uh oh!

ahus1 Apr 28, 2025 Collaborator

xgp
Apr 25, 2025

Replies: 1 comment 6 replies

xgp
Apr 25, 2025
Author

xgp Apr 28, 2025
Author

shawkins Apr 28, 2025
Collaborator

ahus1 Apr 28, 2025
Collaborator

xgp Apr 28, 2025
Author

ahus1 Apr 28, 2025
Collaborator