You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We run keycloak in AWS Fargate. We have 4 tasks, the default infinispan cache sizes, we use the now default JDBC_PING2 discovery protocol, and each cache entry has two owners. So a pretty standard setup.
What we have noticed, I believe since version 24 approximately, is that whenever we make a rolling update and the old tasks are killed, there is window of time in which the thread usage increases up to the max limit, causes the caches queue to fill up and therefore block, making the whole system unresponsive... until it eventually goes back to normal after aroudn 5 min.
We suspect this is happens whenever a node leaves the cluster. Not exactly sure what causes it, but the leading theory is either one of these or a combination of all is causing:
When a node leaves, infinispan needs to rebalance the cache entries in the remaining nodes, casuing a lot of extra work that causes the thread starvation.
When a node leaves, and the cache entries are redsitributed, there is a lot of memory for the GC to clean up. If I undesrtand corrrectly, this work is done in the background, and it might also be using a lot of threads.
There might be a period in time in which the nodes tries to fetch the non-owned session from the cache entry owner, but the owner has already been stopped, timing out on those requests.
We would like to sort out this issue and deploy with the confidence that it will not lead to this virtual 5 min downtime, but we are not sure exactly what to try next. We have experimented with slower deployments, adding and killing a single node at a time, we have reduced the cache sizes to minimze the rebalancing work, we have varied the amount of owners, etc.
We think that in the past we didn't have this issue. It could have been because we had a fully replicated cache instead of distributed, but it could also be because the GC algorithm used in previous versions was ParallelGC instead of G1GC.
Anyhow, has anybody encountered the same issue? Anybody has any idea on what to tweak or what we could try next?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We run keycloak in AWS Fargate. We have 4 tasks, the default infinispan cache sizes, we use the now default JDBC_PING2 discovery protocol, and each cache entry has two owners. So a pretty standard setup.
What we have noticed, I believe since version 24 approximately, is that whenever we make a rolling update and the old tasks are killed, there is window of time in which the thread usage increases up to the max limit, causes the caches queue to fill up and therefore block, making the whole system unresponsive... until it eventually goes back to normal after aroudn 5 min.
We suspect this is happens whenever a node leaves the cluster. Not exactly sure what causes it, but the leading theory is either one of these or a combination of all is causing:
We would like to sort out this issue and deploy with the confidence that it will not lead to this virtual 5 min downtime, but we are not sure exactly what to try next. We have experimented with slower deployments, adding and killing a single node at a time, we have reduced the cache sizes to minimze the rebalancing work, we have varied the amount of owners, etc.
We think that in the past we didn't have this issue. It could have been because we had a fully replicated cache instead of distributed, but it could also be because the GC algorithm used in previous versions was ParallelGC instead of G1GC.
Anyhow, has anybody encountered the same issue? Anybody has any idea on what to tweak or what we could try next?
All reactions