Repository navigation
Perfromance improvement recommendations for keycloak deployment in EKS #39313
Replies: 3 comments 1 reply
Can you provide more of your configuration, in particular what have you configured for your datasource max connections? And have you customized the number of http worker thread? It is normal for the connection pool to grow towards the max concurrent usage under load.
Do you mean the realm loading that happens due to a migration, import, or something else?
This would normally only happen if keycloak is unable to establish a connection to the source - the default timeout I beleive is 5 seconds. Have you changed that configuration? |
We have nor configured ant Database max connections. Default will be applied. We have configured minimum pools size as 5.
Realm loading at the time of scaling. When the statefulset scales from 2->3
We haven't changed any config related to that. This is observed to be happening mainly at the time of keycloak pod scale up and when new pod is created and pod is handling API requests and realm loading at the same time. |
|
The default max conections are 100. If you are only getting up to 44 connections before seeing acquisition timeouts, that would imply Keycloak is unable to establish new connections in a timely manor. Can you try setting the min connections = initial = max instead - as described here #24997 That ideally should highlight if you truly have a load depedent issue. |
|
@shawkins could you please help me to resolve this as i am also getting this issue in keycloak 26.3.0. org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection [Acquisition timeout while waiting for new connection] [n/al |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary of the problem:
We are using Keycloak in our deployment on EKS cluster and observe a few performance issues under load. We would like to get suggestions or recommendations on how to improve the keycloak performance.
Configuration
Keycloak Version: 25.0.3
Bitnami chart verion: 23.0.0
Replicas:
min: 2
max: 10
Resources Reservation:
cpu: requests: 400m, limit 500m
memory: requests: 1GB , limit 2GB
Number of realms in keycloak: >500
Issues Observed:
1. Instability while scaling:
When keycloak statefulset scales under load, new pod is not able to handle the load and we get lot of JDBC connection failures. One thing we have observed is connection pool is growing from 5 upto 44 connections. Also we have verified that there are no throttling hit from DB side. Another observation is that newly created pod does realm loading at the same time as it handles API requests and it takes around few minutes to complete the realm loading.
025-04-25 02:32:44,217 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1044) Uncaught server error: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection [Acquisition timeout while waiting for new connection] [n/a]
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:63)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:108)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:94)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:116)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:143)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:54)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:153)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:183)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:155)
at org.hibernate.sql.exec.spi.JdbcSelectExecutor.lambda$list$0(JdbcSelectExecutor.java:85)
at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.executeQuery(DeferredResultSetAccess.java:231)
at org.hibernate.sql.results.jdbc.internal.DeferredResultSetAccess.getResultSet(DeferredResultSetAccess.java:167)
at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.advanceNext(JdbcValuesResultSetImpl.java:218)
at org.hibernate.sql.results.jdbc.internal.JdbcValuesResultSetImpl.processNext(JdbcValuesResultSetImpl.java:98)
at org.hibernate.sql.results.jdbc.internal.AbstractJdbcValues.next(AbstractJdbcValues.java:19)
at org.hibernate.sql.results.internal.RowProcessingStateStandardImpl.next(RowProcessingStateStandardImpl.java:66)
at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:202)
at org.hibernate.sql.results.spi.ListResultsConsumer.consume(ListResultsConsumer.java:33)
at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.doExecuteQuery(JdbcSelectExecutorStandardImpl.java:209)
at org.hibernate.sql.exec.internal.JdbcSelectExecutorStandardImpl.executeQuery(JdbcSelectExecutorStandardImpl.java:83)
at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:76)
at org.hibernate.sql.exec.spi.JdbcSelectExecutor.list(JdbcSelectExecutor.java:65)
at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.lambda$new$2(ConcreteSqmSelectQueryPlan.java:137)
at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.withCacheableSqmInterpretation(ConcreteSqmSelectQueryPlan.java:381)
at org.hibernate.query.sqm.internal.ConcreteSqmSelectQueryPlan.performList(ConcreteSqmSelectQueryPlan.java:303)
at org.hibernate.query.sqm.internal.QuerySqmImpl.doList(QuerySqmImpl.java:509)
at org.hibernate.query.spi.AbstractSelectionQuery.list(AbstractSelectionQuery.java:427)
at org.hibernate.query.Query.getResultList(Query.java:120)
at org.keycloak.models.jpa.JpaUserProvider.getUserByEmail(JpaUserProvider.java:537)
at org.keycloak.storage.UserStorageManager.getUserByEmail(UserStorageManager.java:416)
at org.keycloak.models.cache.infinispan.UserCacheSession.getUserByEmail(UserCacheSession.java:419)
at org.keycloak.models.utils.KeycloakModelUtils.findUserByNameOrEmail(KeycloakModelUtils.java:247)
at org.keycloak.authentication.authenticators.directgrant.ValidateUsername.authenticate(ValidateUsername.java:65)
at org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(DefaultAuthenticationFlow.java:442)
at org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:246)
at org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:1071)
at org.keycloak.protocol.oidc.grants.ResourceOwnerPasswordCredentialsGrantType.process(ResourceOwnerPasswordCredentialsGrantType.java:111)
at org.keycloak.protocol.oidc.endpoints.TokenEndpoint.processGrantRequest(TokenEndpoint.java:140)
at org.keycloak.protocol.oidc.endpoints.TokenEndpoint$quarkusrestinvoker$processGrantRequest_6408e15340992839b66447750c221d9aaa837bd7.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1512)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.sql.SQLException: Acquisition timeout while waiting for new connection
at io.agroal.pool.ConnectionPool.handlerFromSharedCache(ConnectionPool.java:326)
at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:253)
at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
at io.quarkus.hibernate.orm.runtime.customized.QuarkusConnectionProvider.getConnection(QuarkusConnectionProvider.java:23)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:46)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:113)
... 45 more
Caused by: java.util.concurrent.TimeoutException
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
at io.agroal.pool.ConnectionPool.handlerFromSharedCache(ConnectionPool.java:303)
APM shows real slowness in keycloak handling the API requests
This happens while there is a sudden spike in API requests.
Client receiving connection reset by peer from Keycloak
I haven't got exact root cause of this as there are no error logs in keycloak while this happens. This is also observed under load conditions.
All reactions