Configuring distributed caches #12022

rmjstn · 2022-05-16T16:37:17Z

rmjstn
May 16, 2022

Hello,

I managed to do this configuration in the Wildfly distribution. I can't do it in the Quarkus distribution.

I have added this to the file keycloak.conf :

cache=ispn
cache-stack=tcp

An extract of the cache.xml :

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

      <jgroups>
        <stack name="tcpping" extends="tcp">
         <TCP bind_port="7800" />
         <TCPPING
              timeout="3000"
              initial_hosts="10.247.17.56[7800],10.247.17.57[7800]"
              port_range="0"
              num_initial_members="2"
           />
      </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport cluster="mngdkeycloak06t" lock-timeout="60000"  stack="tcpping"/>
        <local-cache name="realms">
        ...

I run this commands :

/opt/keycloak/bin/kc.sh build --cache-config-file=/opt/keycloak/conf/cache.xml
/opt/keycloak/bin/kc.sh --verbose start

Port 7800 is open between both servers.

I don't see the discovery of the members in the logs.

The logs are as follows :

2022-05-16 18:26:55,826 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2022-05-16 18:26:55,887 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2022-05-16 18:26:55,936 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2022-05-16 18:26:56,169 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.8.Final
2022-05-16 18:26:56,283 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `mngdkeycloak05t` with stack `tcpping`
2022-05-16 18:26:58,398 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) mngdkeycloak05t-5291: no members discovered after 2002 ms: creating cluster as coordinator
2022-05-16 18:26:58,408 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel mngdkeycloak05t: [mngdkeycloak05t-5291|0] (1) [mngdkeycloak05t-5291]
2022-05-16 18:26:58,413 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `mngdkeycloak05t` local address is `mngdkeycloak05t-5291`, physical addresses are `[10.247.17.56:7800]`

When I run this command on the second node :

> /dev/tcp/10.247.17.56/7800

I've got this :

2022-05-16 18:29:43,374 WARN  [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-2,mngdkeycloak05t-5291) JGRP000006: failed accepting connection from peer Socket[addr=/10.247.17.57,port=41082,localport=7800]: java.io.EOFException

Thanks,

Rémy

rmjstn · 2022-05-21T08:57:15Z

rmjstn
May 21, 2022
Author

More information.

I noticed that if the file is referenced by the name cache-ispn.xml in keycloak.conf, it is ignored (introducing an error in the XML syntax doesn't disrupt the startup). So I rename it cache-ispn-tcpping.xml.

If I set the port in the file like this :

 <TCP bind_port="8888" />

This has no effect, Keycloak is listening on default port 7800.

If I run this command :

 /opt/keycloak/bin/kc.sh -Djgroups.bind.port=8888 start

Keycloak is listening on port 8888.

Thanks,

Rémy

0 replies

Nascire · 2022-05-21T13:07:14Z

Nascire
May 21, 2022

I got it working for me after some digging and try&error.

In the keycloak.conf I´ve added:

cache=ispn

Than, I copied cache-ispn.xml to cache-ispn-tcpping.xml, and made it look like this:

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">

    <!-- custom stack goes into the jgroups element -->
    <jgroups>
        <stack name="tcpping" extends="tcp">
            <TCP bind_port="7800"/>
            <TCPPING
                initial_hosts="172.999.999.10[7800],192.999.999.10[7800]"
                port_range="0"
                stack.combine="REPLACE"
                stack.position="MPING"
                />
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="tcpping"/>
        <local-cache name="realms">

Afterwards run a build and specify the new cache file, so values get updated.
kc.sh build --cache-config-file=cache-ispn-tcpping.xml

timeout/join_timeout and num_members are not included, because of an error I received:
ERROR: JGRP000001: configuration error: the following properties in TCPPING are not recognized: {num_initial_members=2, join_timeout=3000}

To show that it´s indeed working:

May 21 13:00:22 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:22,381 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
May 21 13:00:22 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:22,535 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
May 21 13:00:22 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:22,675 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
May 21 13:00:23 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:23,174 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.8.Final
May 21 13:00:23 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:23,301 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `tcpping`
May 21 13:00:23 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:23,602 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak02-45874|1] (2) [keycloak02-45874, keycloak01-51236]
May 21 13:00:23 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:23,759 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak01-51236`, physical addresses are `[192.999.999.10:7800]`
May 21 13:00:26 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:26,518 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak01-51236, Site name: null
May 21 13:00:35 keycloak01.ipa.... kc.sh[6090]: 2022-05-21 13:00:35,134 INFO  [io.quarkus] (main) Keycloak 18.0.0 on JVM (powered by Quarkus 2.7.5.Final) started in 21.875s. Listening on: https://0.0.0.0:8443

0 replies

rmjstn · 2022-05-21T16:48:34Z

rmjstn
May 21, 2022
Author

Hi @Nascire

It works...
I was able to find my mistake thanks to your comment.
I have commented cache-stack=tcp in the keycloak.conf file which allowed to correct the problem.

I don't know what the default value ss : https://www.keycloak.org/server/caching#_transport_stacks

Thanks

Rémy

5 replies

kkcmadhu Jun 2, 2022

hope same thing will work for dns ping too, any idea?

ghost Jun 14, 2022

Thanks a lot @rmjstn for the explanation! This helped me a lot.

It seems the filename makes a lot of difference?

When I make the changes to cache-ispn.xml and rebuild keycloak then tcpping will not work.
If I copy cache-ispn.xml to another file, eg. tcpping.xml and make exact the same changes, rebuild keycloak, then it will work.

Is that also your experience?
I am running keycloak 18.0.0 on JVM 17.

Nascire Jun 17, 2022

@jurgeng-ot I´m not sure if it is really the change of the filename - during my troubleshooting it seemed that the settings in the file only got read when changing the filename - I than just stuck to a different filename, also to make it clear what it´s used for (chache-ispn-tcpping.xml)

kkcmadhu Jun 17, 2022

i think keycloak released a new verson 18.0.1 yesterday/today ,which possible contains a fix for this https://www.keycloak.org/2022/06/keycloak-1801-released.html
#11302 and #11645

Nascire Jun 17, 2022

#11302 only is cosmetic, telling you that the path is relative to /conf
#11645 is only a minor version upgrade on the infispan version

So I don´t think the new version will change anything ...

kkcmadhu · 2022-06-16T15:19:10Z

kkcmadhu
Jun 16, 2022

my present keycloak configiruation is to use dns ping as below in wildfly. i just copy pasted the same in my cache-ispn-k8s.yml and could not get my server started. can some one suggest whats the equivalent to be used with quarkus

<stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="dns.DNS_PING">
                        <property name="dns_query">keycloak18-headless.kc.svc.cluster.local</property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS">
                        <property name="join_timeout">30000</property>
                    </protocol>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>

0 replies

kkcmadhu · 2022-06-16T15:32:42Z

kkcmadhu
Jun 16, 2022

i would like to use this kind of config in quarkus https://infinispan.org/docs/stable/titles/server/server.html#discovery-dns-ping_cluster-transport

and possible pass the dns_query as a runtime/configuration parameter something like

kc.[bat|sh] start --dns_query mykeycloakk9sservice-headless.local.cluster

I think this will be very useful enhancment to keycloak on quarkus

jgroups actually support -D values for query param refer here https://infinispan.org/docs/stable/titles/server/server.html#kubernetes

1 reply

kkcmadhu Jun 20, 2022

i could get it working by config param --cache=ispn and --cache-stack=kuberneter

and as far as the dns query is concerned i loaded it using env varibale jgroups.dns.query (pointing to my headless service)

I was able to get more than 1 keycloak node added to infinispan cluster on quarkus ( with all oob setting and the additional env variable)

so...

build time

command to build

kc.bat build --db=mysql --features="token-exchange,scripts,preview" --http-relative-path=/auth --health-enabled=true --metrics-enabled=true --cache ispn --cache-stack=kubernetes

config map :

        - name: jgroups.dns.query
          value: "keycloak18-headless"

  - name: KEYCLOAK_ADMIN
          value: "myadminuser" (load from secrets)
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: "password" (load from secrets)
        - name: KC_PROXY
          value: "edge"
        - name: KC_DB
          value: "mysql"
        - name: KC_DB_URL
          value: "jdbc:mysql://mysqlurl"
        - name: KC_DB_USERNAME
          value: "username"   (load from secrets)
        - name: KC_DB_PASSWORD
          value: "password" (load from secrets)
        - name: KC_HEALTH_ENABLED
          value: "true"
        - name: KC_METRICS_ENABLED
          value: "true"
        - name: KC_FEATURES
          value: "token-exchange,preview,scripts"
        - name: KC_HTTP_RELATIVE_PATH
          value: "/auth"
other environment variable which you need..

Starting keycloak ::
kc.bat |sh [dev-]start

eggyknap · 2022-07-14T20:11:31Z

eggyknap
Jul 14, 2022

My two-node cluster, not currently in production, runs on Azure where multicast isn't an option so eventually we went with DNS_PING. I have --cache=ispn and --cache-stack=kubernetes in my build options. I include this environment variable when starting Keycloak: JAVA_OPTS_APPEND="-Djgroups.dns.query=my_dns_name". That my_dns_name name is configured in /etc/hosts, like this:

10.1.0.1  my_dns_name
10.1.0.2  my_dns_name

... where 10.1.0.1 and 2 are the IP addresses of my two nodes. This appears to work fine.

2 replies

Arulaln-AR Jul 28, 2022

@eggyknap , thanks. We don't need the --cache ispn right. Because it is the default value. rest all looks good. thanks will try that

eggyknap Jul 28, 2022

All I can say is that it worked for me with that option explicitly set. I don't think I experimented with turning it off, nor paid attention to what the default actually is.

kkcmadhu · 2022-07-28T16:21:00Z

kkcmadhu
Jul 28, 2022

yes --cache ispn not required for default settings. On Thursday, 28 July, 2022 at 09:21:08 pm IST, Arulaln-AR ***@***.***> wrote: @eggyknap , thanks. We don't need the --cache ispn right. Because it is the default value. rest all looks good. thanks will try that — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

0 replies

bravoman · 2022-12-23T15:41:05Z

bravoman
Dec 23, 2022

<jgroups>
    <stack name="tcpping" extends="tcp">
        <TCP bind_port="7800"/>
        <TCPPING
            initial_hosts="172.999.999.10[7800],192.999.999.10[7800]"
            port_range="0"
            stack.combine="REPLACE"
            stack.position="MPING"
            />
    </stack>
</jgroups>

<cache-container name="keycloak">
    <transport lock-timeout="60000" stack="tcpping"/>

That was exactly what I needed @Nascire Thanx! I was so close to this solution already, just didn't know how to override the MPING in the tcp stack.

Weird though that UDP multicast is not working (or so hard to get working) on our end. But I used something like this as a workaround before in older keycloak version configurations.

1 reply

raphisuter Jan 19, 2023

Same here, there was always MPING active for discovery instead of the configured TCPPING. By adding

stack.combine="REPLACE"
stack.position="MPING"

keycloak finally switched away from UDP MPING and used TCPPING my configuration.
This definitely needs to be placed in the documentation!

kavehazizi · 2023-09-12T14:25:48Z

kavehazizi
Sep 12, 2023

hi, I am a little frustrated here as no matter what I am doing, I cannot make the HA on our swarm, logs below is the same on all the other two nodes. It seems that none of them can see each other (while their container is being created flawlessly), I am using tcpping and mariadb. Any help would be much appreciated.

2023-09-12 15:32:51,599 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <request>, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: -1, Proxied: true 2023-09-12 15:32:52,853 INFO [org.keycloak.common.crypto.CryptoIntegration] (main) Detected crypto provider: org.keycloak.crypto.def.DefaultCryptoProvider 2023-09-12 15:32:54,794 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal 2023-09-12 15:32:54,913 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled 2023-09-12 15:32:54,932 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller' 2023-09-12 15:32:55,151 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.9.Final 2023-09-12 15:32:55,294 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel ISPNwith stacktcpping 2023-09-12 15:32:57,415 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) 3a4fc384a7ed-28317: no members discovered after 2005 ms: creating cluster as coordinator 2023-09-12 15:32:59,050 INFO [io.quarkus] (main) Profile prod activated.
tcpping config:

<jgroups> <stack name="tcpping" extends="tcp"> <TCP bind_port="7800" />  <TCPPING initial_hosts="x.x.x.x.[7800],y.y.y.y[7800],z.z.z.z[7800]" port_range="0" stack.combine="REPLACE" stack.position="MPING" /> </stack> </jgroups>

0 replies

rajesh206 · 2023-09-27T11:02:04Z

rajesh206
Sep 27, 2023

how can i test clustering works with tcp ping, can anyone help

1 reply

kavehazizi Sep 27, 2023

your question is not clear, when you are done with your clsuter, in the docker logs you should see that different containers can see each other and join the cluster.

prettypot · 2023-11-06T17:15:14Z

prettypot
Nov 6, 2023

I have started a new thread here. Seems it could be neater given this thread might not be relevant to my problem, or not. The URL is : Keycloak distributed caches - TCPPING and variable replacement error #24584

HI,

I tried to get TCPPING configured but cannot get it to replace the variables. I start to think that the build is ignoring my xml file, or the varibles in my start.sh file.

Here is my cache file. I deleted the original cache-ispn.xml.

$ cat cache-uat.xml
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
        xmlns="urn:infinispan:config:14.0">

<jgroups>
    <stack name="tcpping" extends="tcp">
    <TCP external_addr="${KC_EXTERNAL_ADDR}" bind_addr="${env.KC_BIND_ADDR}" bind_port="${env.KC_BIND_PORT}" />
    <TCPPING initial_hosts="${env.KC_INITIAL_HOSTS}"
    max_dynamic_hosts="0"
    port_range="0"
    stack.combine="REPLACE"
    stack.position="MPING"/>
    </stack>
</jgroups>

<cache-container name="keycloak">
<transport lock-timeout="60000" stack="tcpping" />
        <local-cache name="realms" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>

I am building the config with:

$ kc.sh build --cache-config-file=cache-uat.xml

And then start it with this script:

$ cat start.sh
export JAVA_HOME=/usr
export PATH=$PATH:$JAVA_HOME/bin
export KEYCLOAK_ADMIN=admin
export KEYCLOAK_ADMIN_PASSWORD=password
export HTTPS_PROXY=http://cat.dog.local:3128
export NO_PROXY=*.dog.local,*.bird.local
export KC_CACHE_CONFIG_FILE=cache-uat.xml
export KC_EXTERNAL_ADDR=10.1.1.11
export KC_BIND_ADDR=10.1.1.11
export KC_BIND_PORT=7600
export KC_INITIAL_HOSTS="10.1.1.12[7600],10.1.1.13[7600]"
/data/keycloak/bin/kc.sh start --optimized

Yet I cannot get around this replace error :

INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: <request>,Strict HTTPS: false, Path:<request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1,Proxied: true
WARN  [org.infinispan.commons.util.StringPropertyReplace
r] (keycloak-cache-init) ISPN000901: Property KC_EXTERNAL_ADDR could not be replaced as intended!
WARN  [org.infinispan.commons.util.StringPropertyReplacer] (keycloak-cache-init) ISPN000901: Property env.KC_BIND_ADDR could not be replaced as intended!
WARN  [org.infinispan.commons.util.StringPropertyReplacer] (keycloak-cache-init) ISPN000901: Property env.KC_BIND_PORT could not be replaced as intended!
WARN  [org.infinispan.commons.util.StringPropertyReplacer] (keycloak-cache-init) ISPN000901: Property env.KC_INITIAL_HOSTS could not bereplaced as intended!
WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
ERROR [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000660: DefaultCacheManager start failed, ... 
[SNIP]

Does anybody know?

0 replies

Configuring distributed caches #12022

Uh oh!

Replies: 11 comments · 10 replies

Uh oh!

rmjstn May 21, 2022 Author

Uh oh!

Uh oh!

rmjstn May 21, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 11 comments 10 replies

rmjstn
May 21, 2022
Author

rmjstn
May 21, 2022
Author