Can't run keycloak 17 on Quarkus with cross DC mode #10521

koldson · 2022-03-01T14:56:34Z

koldson
Mar 1, 2022

Hello,
At the beginning I will mention that running the configuration within one subnetwork works.
I prepared necessary option in keycloak.conf like a db info and e.g. I didn't change the configuration for cache-ispn.xml

I ran it in order:
bin/kc.sh -Dkc.db-tx-type=enabled -Dkc.db-driver=com.mysql.cj.jdbc.Driver build --db=mysql
and after
bin/kc.sh start

My next step was to run keycloak in another DC with different network. I tried to put jgroups configuration at the beginning cache.xml

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">


<jgroups>
   <stack name="global-cluster" extends="tcp">
       <!-- Remove MPING protocol from the stack and add TCPPING -->
       <TCPPING initial_hosts="keycloak001.dc3.intra.example.com[7800],keycloak002.dc3.intra.example.com[7800],keycloak001.dc2.intra.example.com[7800],keycloak002.dc2.intra.example.com[7800]"
                stack.combine="REPLACE" stack.position="MPING"/>
   </stack>

   <stack name="xsite" extends="udp">
      <relay.RELAY2 site="dc2-2"
                    max_site_masters="100"
                    can_become_site_master="true"/>
      <remote-sites default-stack="global-cluster">
         <remote-site name="dc2-2"/>
         <remote-site name="dc2-1"/>
         <remote-site name="dc3-1"/>
         <remote-site name="dc3-2"/>
      </remote-sites>
   </stack>

</jgroups>


    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="global-cluster"/>
        <replicated-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
	    <memory max-count="10000"/>
            <backups>
              <backup site="dc2-1" strategy="SYNC" />
              <backup site="dc3-1" strategy="SYNC" />
              <backup site="dc3-2" strategy="SYNC" />
            </backups>
        </replicated-cache>
...

but it does't work ;/
I have 2 keycloak server per DC.
I tired to do:

bin/kc.sh -Dkc.db-tx-type=enabled -Dkc.db-driver=com.mysql.cj.jdbc.Driver build --db=mysql --cache-config-file=cache-my.xml
bin/kc.sh start

Now I am getting such errors:

2022-03-01 12:53:01,895 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `global-cluster`
2022-03-01 12:53:03,995 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak001-34719: no members discovered after 2005 ms: creating cluster as coordinator
2022-03-01 12:53:04,010 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak001-staging-34719|0] (1) [keycloak001-staging-34719]
2022-03-01 12:53:04,015 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak001-34719`, physical addresses are `[10.X.X.X:7800]`
2022-03-01 12:53:04,244 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000080: Disconnecting JGroups channel `ISPN`
2022-03-01 12:53:04,424 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2022-03-01 12:53:04,425 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Error details:: java.lang.RuntimeException: Failed to start caches
	at org.keycloak.quarkus.runtime.storage.infinispan.CacheManagerFactory.getOrCreate(CacheManagerFactory.java:53)
	at org.keycloak.quarkus.runtime.storage.infinispan.CacheManagerFactory_6ec50d2dc78e759865b1f14e1c157f33d9676e70_Synthetic_ClientProxy.getOrCreate(Unknown Source)
	at org.keycloak.quarkus.runtime.storage.infinispan.QuarkusCacheManagerProvider.getCacheManager(QuarkusCacheManagerProvider.java:32)
	at org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory.lazyInit(DefaultInfinispanConnectionProviderFactory.java:139)
	at org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory.create(DefaultInfinispanConnectionProviderFactory.java:81)
	at org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory.create(DefaultInfinispanConnectionProviderFactory.java:65)
	at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:316)
	at org.keycloak.models.sessions.infinispan.InfinispanActionTokenStoreProviderFactory.initActionTokenCache(InfinispanActionTokenStoreProviderFactory.java:52)
	at org.keycloak.models.sessions.infinispan.InfinispanActionTokenStoreProviderFactory.postInit(InfinispanActionTokenStoreProviderFactory.java:67)
	at org.keycloak.quarkus.runtime.integration.QuarkusKeycloakSessionFactory.init(QuarkusKeycloakSessionFactory.java:96)
	at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.initializeKeycloakSessionFactory(QuarkusKeycloakApplication.java:75)
	at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.startup(QuarkusKeycloakApplication.java:56)
	at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver.onStartupEvent(QuarkusLifecycleObserver.java:37)
	at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver_Observer_onStartupEvent_8292a286c9778cb4a07bc3314fcbfb61e7fe81a0.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:320)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:302)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:73)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:128)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:97)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:104)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
	at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:84)
	at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:34)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:84)
	at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:79)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)
Caused by: java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000571: RELAY2 not found in ProtocolStack. Unable to do cross-site requests.
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.keycloak.quarkus.runtime.storage.infinispan.CacheManagerFactory.getOrCreate(CacheManagerFactory.java:51)
	... 42 more
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000571: RELAY2 not found in ProtocolStack. Unable to do cross-site requests.

Can anyone help or send a correct example to do it cross site replication?

Thanks a lot for your help and tips!

darius-m · 2022-03-03T11:16:12Z

darius-m
Mar 3, 2022

Not sure if I understand the new configurations properly, but for distributed caches you may need to set the cache to type to ISPN (as mentioned in the cache configuration page), using --cache ispn.

0 replies

darius-m · 2022-03-03T11:34:50Z

darius-m
Mar 3, 2022

Looking closer at your configuration, I think there is an error. You have the following configuration:

<transport lock-timeout="60000" stack="global-cluster"/>

This means that you are not using the xsite stack that handles replication. The stack parameter should be changed to xsite.

Also, if you prefer, you could use a <replicated-cache-configuration name="cache-config"> element that defines common cache parameters, such as encoding, memory, and others, and then reuse it when declaring the cache, like <replicated-cache name="realms" configuration="cache-config"/>. More details are in the infinispan documentation, but unfortunately you will have to dig for them a bit.

0 replies

lacek · 2022-03-31T05:40:46Z

lacek
Mar 31, 2022

Is cross DC support even ready for Quarkus distributions yet?

Below are what I currently observe:

In the guide for legacy WildFly distribution, it was mentioned that for cross DC mode:

we assume that either both data centers talk to the same database or that every data center has its own database node and both database nodes are synchronously replicated across the data centers
In the obsoleted storage design for Keycloak X, it is mentioned that:

Infinispan is responsible for all replication cross DC, which would provide a redundant copy of the database without the need to configure replication of the database itself
In the server guide, it is mentioned that:

Keycloak uses JGroups and Infinispan under the covers to provide a reliable, HA-ready stack to run in a clustered scenario.
In the issue list, Enable Cross DC support for quarkus based keycloak distribution #9831 is pending for version 19.0.0

I am a bit confused with the above observations for a cross DC scenario in Keycloak 17. Shall we:

use WildFly distribution with cross DC clusters of database and cache (as described in 1 above - the legacy guide)
use Quarkus distribution with cross DC clusters of database and cache (would it be just similar to 1?)
use Quarkus distribution with cross DC cluster cache and individual database for each DC (if so, why do we have item 4 then? for testing and docs only?)
...

0 replies

nathmo · 2022-08-23T17:17:27Z

nathmo
Aug 23, 2022

This discussion might be related : #10694

Anyway, both subject looks important to me and should be addressed soon as this is quite important when deploying with redundancy in mind.

another idea would be to propose a way of manually specifying the node in a list instead of relying on the UDP multicast as it seems to have a default TTL of 2 and i can't find where to change that so that it works.

0 replies

Navigation Menu

Repository navigation

Can't run keycloak 17 on Quarkus with cross DC mode #10521

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Can't run keycloak 17 on Quarkus with cross DC mode #10521

Uh oh!

koldson Mar 1, 2022

Replies: 4 comments

Uh oh!

darius-m Mar 3, 2022

Uh oh!

Uh oh!

darius-m Mar 3, 2022

Uh oh!

lacek Mar 31, 2022

Uh oh!

Uh oh!

nathmo Aug 23, 2022

koldson
Mar 1, 2022

darius-m
Mar 3, 2022

darius-m
Mar 3, 2022

lacek
Mar 31, 2022

nathmo
Aug 23, 2022