Repository navigation
|
Hi, I would like to know if Keycloak can handle rolling upgrades on Kubernetes when scaled to multiple instances. We are in the process of migrating from Keycloak 19 (legacy Wildfly version) to the latest Quarkus version of Keycloak (21.1.1). If it is not advised to use rolling upgrades, then we will stick with a full redeploy of the Keycloak servers. |
Replies: 4 comments 9 replies
|
It's mentioned (somewhere) in the docs, that Keycloak currently does NOT support zero-downtime deployments between major versions. This is due to possible incompatible database changes between major versions. |
|
Thank you for the feedback Niko. I think I missed that part in the documentation but it makes sense as you explain it. This was also my first guess but now it is confirmed! |
|
I need to do something similar - rolling upgrade from Keycloak 17 to Keycloak 21. So, I checked the DB schema difference between these 2 versions and it does not look like there are any incompatible changes. Just as an experiment, I started a Keycloak 21 instance and pointed it to a DB that was previously used by a Keycloak 17 instance and it upgraded the schema just fine. Then I stopped the Keycloak 21 instance and pointed the Keycloak 17 instance to the same DB (which has an upgraded schema) and it seemed to work fine as well. My bigger concern is the difference in infinispan and JGroups versions between Keycloak 17 and Keycloak 21. Keycloak 17 has infinispan 12.x and Jgroups 4.x while Keycloak 21 has infinispan 14.x and JGroups 5.x. Does anyone know of any solutions to this issue ? |
|
For anyone landing here later wondering if there’s a way to deploy incompatible keycloak versions, if you’re okay with losing old sessions but don’t want to kill all your pods, this is for you! I wanted to post here because after seeing this issue my team was pretty scared to do a major version upgrade due to concerns with Infinispan versions being incompatible and the downtime we would incur from that. We ended up with a pretty successful deployment where we only lost data that was exclusively cached in the older version. We did a rolling update from version 21.0.1 to 23.0.2 - the trick was to have a separate headless service that only the new pods knew about, so the old and new versions couldn’t talk to each other. Yes, we lost sessions from the old pods but other than that once we cut traffic over to the new version everything worked great. We did this by tagging our deployment and naming our headless service with the keycloak version that we’re upgrading to (along with the jgroups.dns.query arg). So yes we lost some sessions but we didn’t need to take everything down. Examples for helm: |
Yes, the major concern is the database. If there are a breaking changes, the older version of KC would stop working. |
|
Looks like a nice workaround! Could it be that with the new "persistent-sessions" feature the problem of loosing sessions could also be solved in that case? (granted there are no breaking changes in the db schema...) |
|
If you use a separate headless service, you would have two different Keycloak clusters. As they would use a different work cache, cache invalidation due to a changed user, permissions or realm information wouldn't be available in the other cluster. |
|
Could this problem be mitigated when scaling down to only two nodes before the update? This would at least keep the timespan of the coexistence of both ispn clusters as short as possible... |
|
By scaling down to two, the effects could be reduced, but can't be eliminated. At the same time, reducing the capacity to two instances isn't always possible for everyone, and might break a high throughput environment. So I don't consider this a generic solution. We're working on preparing rolling updates for Infinispan in upcoming releases. The new upgrade command prepared here will be one of the milestones towards this: #36305 While at first it will support rolling updates while you stay on the same Keycloak release, in the next iteration we'll be working to make rolling updates of patch releases tested and supported. |
|
Some rolling functionality is now available as described in #36305 (comment) and will be available as of Keycloak 26.2 It allows for rolling updates when using the Keycloak operator when using the Automatic rolling updates for patch releases are not yet implemented, but we're looking into that for a follow-up release. |
|
Keycloak 26.2 was released today with the #36305 functionality that allows rolling updates on Operator image changes as long as the KC version stays the same. Those who are brave and want to experiment can try the We're now tracking the work to support zero downtime patch release upgrades in #38881. Please subscribe to that issue to get the latest updates and peek into the design planned. |
It's mentioned (somewhere) in the docs, that Keycloak currently does NOT support zero-downtime deployments between major versions. This is due to possible incompatible database changes between major versions.
So, if you want to migrate from 19 to the most recent version, you better should shut down all nodes and then start (sequentially!) with the first node in current version.