Repository navigation
|
Upon introduction of Keycloak 23, a new documentation has been introduced to discuss multi-site deployments: https://www.keycloak.org/high-availability/introduction The page says "Keycloak supports deployments that consist of multiple Keycloak instances that connect to each other using its embedded Infinispan; load balancers can distribute the load evenly across those instances. Those setups are intended for a transparent network on a single site." but discusses mainly about active-passive deployment model. Does Keycloak support active-active deployments (ie. multiple nodes that serve requests simultaneously)? |
Replies: 3 comments 5 replies
|
I can confirm that active-active deployments can indeed be achieved... as long as the Infinispan instances do not desynchronize. Basically, for a cross-datacentre configuration you need to configure Keycloak to use external Infinispan servers, and the external Infinispan servers must back up cache data to the other site. Cross-datacentre configurations are not entirely new to Keycloak. They have been around since before Keycloak 15 (in an experimental state), when it was still using the old WildFly framework, but after moving to Quarkus the documentation for it was not added back, likely because it was still not fully supported. To make it work, you must make sure to forward all requests for a client (during the authentication flow) to the same backend. This can be done by either using sticky sessions through the Infinispan route cookies, or other means like IP hashing. Sticky session are used to make sure that the intermediate caches used during the authentication flow are always visible to the Keycloak server that receives the request. In theory at least, any server should still be able to fetch the cookies even if a request is not received by the same backend, since Infinispan can fetch the data from any owner in the network. Since an active-active deployment may send requests to either datacentre for a single client, to make sure that caches for all backends are the same, synchronous backups are recommended. The issue with this is that Infinispan does not support automatic recovery from failed backups for synchronous backups (auto recovery for asynchronous backups was recently added, but they may be less reliable in some cases), so from time to time you may find that the sites are considered offline if the network connection encounters errors for whatever reason. To make matters worse, this can happen for a subset of the caches (e.g., the |
|
Thanks for the thoughtful answer @darius-m ! There are a few points I want to clarify a bit.
Do you know why it is required to use external Infinispan servers in this case? Any particular reason in mind why Infinispan cache hosted as it's bundled to Keycloak wouldn't be enough? |
|
To be honest, I've never tried to do the replication through the embedded Infinispan. In theory it may be possible, since most of the configuration is done through JGroups. However, there might be some other issues with how Infinispan is used internally, but I cannot tell. I'm also not sure whether you can access the embedded Infinispan server's CLI to perform the migration steps if you need to do a manual recovery. Regardless, if you want cross-datacenter functionality, you will still need to use the backups system. The issues with the synchronous backups are very likely not caused by the fact that an external Infinispan is used, but rather, how Infinispan is designed to handle failed backups. Currently, for synchronous backups, sites are moved to an offline state and are not re-synchronized automatically. The guide on moving back to the primary cluster is likely the best that can be done currently (manually setting everything offline, removing the cache entries and pushing the changes from one cluster to the other), so an active-active deployment can be subject to temporary failures in the long term. This being said, I think we've had moderate success with multi-cluster deployment, but errors did come up from time to time (I think it was over a month between issues, so not horrible, but far from ideal). Using an external Infinispan has some advantages, however. Infinispan can export opentelemetry data on the caches, so you can monitor and have alerts if the caches go offline, so if you have something like Prometheus with Alertmanager you can get alerts for errors. Other than that, you can have just the external server handle things like persistence, and even if you don't use persistence, you can still use the fact that Infinispan does not necessarily restart whenever Keycloak restarts, so you can keep the caches up even when you need to restart the Keycloak service (e.g., when upgrading Keycloak). |
|
@darius-m : do you know if for stake of simplicity it's possible to use only one infinispan (same for both instance) in active-active mode ? |
|
I honestly don't have a definitive answer, but I don't see a reason why it would not be possible. However, I don't think this setup would qualify as a cross-datacenter implementation, and instead it would be more like a multi-node Keycloak cluster. Multi-node Keycloak clusters are possible already and they should work in active-active mode regardless, as far as I'm aware. However, you don't have any redundancy in the Infinispan service, if the link fails. From my experience, multi-cluster deployments that use Infinsipan to back up the data from one cluster to another appear to work as expected, but there's no real mechanism to completely disable one of the sides in case the caches are desynchronized (you get a split-brain condition with both ends chugging along, which is not great). The new feature to store user sessions in the database (as discussed in #28271) is a step in the direction of fully supporting active-active deployments, but more work is still required. In the current setup I can't truly recommend active-active deployments unless you have monitoring for Infinispan to see when the synchronization has failed and are ready to deal with the headache of manually dealing with the issue. |
|
Hi @Arnaud-FELTZ, @darius-m, @Laurenzi , if I configure my caches to only be local caches, I guess I get rid of the infinispan requirement to be replicated in a multi-site setup, isn't it? Also do you know if there will be any id conflicts in the active- active setup? Is it needed to change the sequences in the database for that to work well? Thank you. Cheers! |
|
While I'm not familiar with the inner workings of Keycloak, judging from various discussions in the repo, I believe Keycloak requires having distributed Infinispan caches, as some caches are used for synchronization between Keycloak instances. Please read the https://www.keycloak.org/high-availability/concepts-multi-site guide for details on how it's supposed to work. |
|
The docs at https://www.keycloak.org/high-availability/introduction have been updated with two scenarios:
Both setups support active-active. With that, I'm calling this one resolved. |
The docs at https://www.keycloak.org/high-availability/introduction have been updated with two scenarios:
Both setups support active-active. With that, I'm calling this one resolved.