Keycloak && IS cluster #12239

AlexeiKlimenko · 2022-05-27T14:12:32Z

AlexeiKlimenko
May 27, 2022

Describe the bug

Hello.
Keycloak - 18.0.0
Infinispan - 13.0.8

Was deployed one instance KC bare metal and trying to configure it with IS cluster (2 nodes in different datacenters).

Config of IS:
`

  </replicated-cache-configuration>

  <socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
     <socket-binding name="default" port="${infinispan.bind.port:11222}"/>
     <socket-binding name="memcached" port="11221"/>
  </socket-bindings>

  <security>
     <credential-stores>
        <credential-store name="credentials" path="credentials.pfx">
           <clear-text-credential clear-text="secret"/>
        </credential-store>
     </credential-stores>
     <security-realms>
        <security-realm name="default">
           <!-- Uncomment to enable TLS on the realm -->
           <!-- server-identities>
              <ssl>
                 <keystore path="application.keystore"
                           password="password" alias="server"
                           generate-self-signed-certificate-host="localhost"/>
              </ssl>
           </server-identities-->
           <properties-realm groups-attribute="Roles">
              <user-properties path="users.properties"/>
              <group-properties path="groups.properties"/>
           </properties-realm>
        </security-realm>
     </security-realms>
  </security>

  <endpoints socket-binding="default" security-realm="default">
    <endpoint>
      <hotrod-connector name="hotrod">
        <authentication>
          <sasl mechanisms="DIGEST-MD5"
                server-name="infinispan" />
        </authentication>
      </hotrod-connector>
      <rest-connector name="rest"/>
    </endpoint>
  </endpoints>

`

Config cache on KC:

`

<cache-container name="keycloak">
<transport lock-timeout="60000"/>

    <local-cache name="realms">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>


    <local-cache name="users">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>


    <distributed-cache name="sessions" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="sessions" 
                      fetch-state="false"
                      purge="false"
	          preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
	<security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!-- encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>	
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="authenticationSessions" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
            <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>         
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="offlineSessions" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
            <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="clientSessions" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
        <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="offlineClientSessions" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
        <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="2">
    <expiration lifespan="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="loginFailures" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
        <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
    <local-cache name="authorization">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <replicated-cache name="work">
    <expiration lifespan="-1"/>
    <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="work"
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
        <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                            type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </replicated-cache>
    <local-cache name="keys">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="3600000"/>
        <memory max-count="1000"/>
    </local-cache>
    <distributed-cache name="actionTokens" owners="2">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="-1" lifespan="-1" interval="300000"/>
    <memory max-count="-1"/>
        <remote-store xmlns="urn:infinispan:config:store:remote:13.0" cache="actionTokens" 
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      segmented="false"
	          shared="true">
            <remote-server host="infinispan01" port="11222" />
	<remote-server host="infinispan02" port="11222" />
        <security>
              <authentication>
	    <digest username="X" password="Y" realm="default"/>
          </authentication>
              <!--encryption>
                <truststore filename="/opt/keycloak/truststore.jks"
                            password="Z"
                          
8000
  type="JKS"/>
          </encryption-->
	</security>
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
        </remote-store>
    </distributed-cache>
</cache-container>

`

After building
"sudo bin/kc.sh start --auto-build --cache=ispn --cache-config-file=cache.xml --https-port 443"

faced with issue:

INFO [org.infinispan.HOTROD] (blocking-thread--p3-t1) ISPN004021: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.8.Final INFO [org.infinispan.HOTROD] (Thread-0) ISPN004006: Server sent new topology view (id=5, age=0) containing 2 addresses: [10.104.X.X:11222, 10.4.Y.Y:11222] INFO [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(10.104.X.X:11222), adding to the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(10.4.Y.Y:11222), adding to the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan02:11222), removing from the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan01:11222), removing from the pool. INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: %node_name%-65174, Site name: null INFO [org.keycloak.models.sessions.infinispan.InfinispanUserLoginFailureProviderFactory] (main) Remote store configured for cache 'loginFailures' INFO [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel ISPNERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Multiple datacenters available, but site name is not configured! Check your configuration

How to resolve it?

Version

18.0.0

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Anything else?

No response

AlexeiKlimenko · 2022-05-27T14:21:15Z

AlexeiKlimenko
May 27, 2022
Author

1 reply

AlexeiKlimenko May 31, 2022
Author

Any ideas?

darius-m · 2022-05-31T19:10:05Z

darius-m
May 31, 2022

In the WildFly distribution you have to define the site name as the jboss.site.name variable (see the configuring remote cache stores section of the installation guide), since the Keycloak server becomes part of a site's Infinispan cluster. You should also probably configure only the local datacenter's Infinispan servers in the remote store section for each Keycloak server.

If you have cross-site replication using two Infinispan servers, one in each datacenter, and two Keycloak servers in each datacenter, there will be a local cluster in each datacenter (one Infinispan server + two Keycloak servers), and a cross-datacenter cluster composed of the two Infinispan servers. I may have misunderstood this part, but it is what I understand from the Communication details section of the installation guide, and from other sources.

7 replies

AlexeiKlimenko Jun 7, 2022
Author

Hello.
Was succeeded to run KC by directly specify site name in kc.sh:
SERVER_OPTS="-Dkc.home.dir='$DIRNAME'/../ -Djboss.site.name='site1' -Djboss.server.config.dir='$DIRNAME'/../conf -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dquarkus-log-max-startup-records=10000"

But actually (despite on running status) it still is not able to use IS cluster as remote cache:
INFO [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(XXX:11222), adding to the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004014: New server added(YYY:11222), adding to the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan02:11222), removing from the pool. INFO [org.infinispan.HOTROD] (Thread-0) ISPN004016: Server not in cluster anymore(infinispan01:11222), removing from the pool. INFO [org.keycloak.connections.infinispan.RemoteCacheProvider] (main) Remote store security for cache loginFailures is disabled. If server fails to connect to remote JDG server, enable it. INFO [io.quarkus] (main) Keycloak 18.0.0 on JVM (powered by Quarkus 2.7.5.Final) started in 6.112s. Listening on: https://0.0.0.0:443 INFO [io.quarkus] (main) Profile prod activated. INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
WARN [org.infinispan.HOTROD] (Thread-0) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: ISPN000936: Class 'org.keycloak.cluster.infinispan.WrapperClusterEvent' blocked by deserialization allow list. Adjust the configuration serialization allow list regular expression to include this class.
ERROR [org.keycloak.services] (Timer-0) KC-SERVICES0089: Failed to run scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$1056/0x0000000840b28440: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=810 returned server error (status=0x85): org.infinispan.commons.CacheException: ISPN000936: Class 'org.keycloak.cluster.infinispan.WrapperClusterEvent' blocked by deserialization allow list. Adjust the configuration serialization allow list regular expression to include this class.

luccascbeltrame Jul 16, 2022

I solved this error followings this steps bellow:

In your infinispan cluster set configurations caches key and value application/x-jboss-marshalling
In your infinispan args set this parameters -Dinfinispan.deserialization.allowlist.classes=java.time.Instant,org.keycloak.cluster.infinispan.WrapperClusterEvent
-Dinfinispan.deserialization.allowlist.regexps=.*
In you cache.xml on keycloak unset enconding for replicated caches.

satanshiro Sep 20, 2022

Hi I tried @luccascbeltrame solution but for some reason my infinispan pods won't start after I add the allow list classes in the cluster config
<infinispan> <cache-container> <serialization marshaller="org.infinispan.commons.marshall.JavaSerializationMarshaller"> <allow-list> <class>org.keycloak.cluster.infinispan.WrapperClusterEvent</class> <class>java.time.Instant</class> <regex>.*</regex> </allow-list> </serialization> </cache-container> </infinispan>
the error is :
15:49:13,130 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:770) at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:734) at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:69) at org.infinispan.security.Security.doPrivileged(Security.java:56) at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40) at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:72) at org.infinispan.server.Server.run(Server.java:398) at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:165) at org.infinispan.server.tool.Main.run(Main.java:98) at org.infinispan.server.Bootstrap.main(Bootstrap.java:50) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.server.loader.Loader.run(Loader.java:106) at org.infinispan.server.loader.Loader.main(Loader.java:51) Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:714) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:660) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:549) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:517) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14) at org.infinispan.security.Security.doPrivileged(Security.java:61) at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30) at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39) at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.start(GlobalConfigurationManagerImpl.java:104) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:60) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:48) at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:766) ... 15 more Caused by: java.util.concurrent.CompletionException: org.infinispan.persistence.spi.PersistenceException: java.io.StreamCorruptedException: invalid stream header: 9801ED07 at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1739) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.infinispan.persistence.spi.PersistenceException: java.io.StreamCorruptedException: invalid stream header: 9801ED07 at org.infinispan.persistence.file.SingleFileStore.blockingStart(SingleFileStore.java:231) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736) ... 5 more Caused by: java.io.StreamCorruptedException: invalid stream header: 9801ED07 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at org.infinispan.commons.marshall.CheckedInputStream.<init>(CheckedInputStream.java:18) at org.infinispan.commons.marshall.JavaSerializationMarshaller.objectFromByteBuffer(JavaSerializationMarshaller.java:52) at org.infinispan.commons.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:82) at org.infinispan.marshall.core.impl.DelegatingUserMarshaller.objectFromByteBuffer(DelegatingUserMarshaller.java:63) at org.infinispan.marshall.protostream.impl.MarshallableUserObject$Marshaller.read(MarshallableUserObject.java:139) at org.infinispan.marshall.protostream.impl.MarshallableUserObject$Marshaller.read(MarshallableUserObject.java:100) at org.infinispan.protostream.impl.ProtobufTagMarshallerDelegate.unmarshall(ProtobufTagMarshallerDelegate.java:32) at org.infinispan.protostream.WrappedMessage.readMessage(WrappedMessage.java:539) at org.infinispan.protostream.WrappedMessage.read(WrappedMessage.java:351) at org.infinispan.protostream.ProtobufUtil.fromWrappedByteArray(ProtobufUtil.java:129) at org.infinispan.marshall.protostream.impl.AbstractInternalProtoStreamMarshaller.objectFromByteBuffer(AbstractInternalProtoStreamMarshaller.java:97) at org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:395) at org.infinispan.persistence.file.SingleFileStore.blockingStart(SingleFileStore.java:205) ... 6 more
it's such a weird error I tried making sure that the configuration is the culprit and it is,
removing it lets infinispan pods start normally.

darius-m Sep 22, 2022

We are using the following configurations for Infinispan and Keycloak - these are Ansible templates for the files, but hopefully they are not too confusing.

There is a replicated-cache-configuration for the caches in Infinispan, and all caches use this configuration, while the Keycloak caches are configured individually. The encoding is not defined in either Infinispan or Keycloak.

nunoOliveiraqwe Oct 31, 2022

Hi I tried @luccascbeltrame solution but for some reason my infinispan pods won't start after I add the allow list classes in the cluster config <infinispan> <cache-container> <serialization marshaller="org.infinispan.commons.marshall.JavaSerializationMarshaller"> <allow-list> <class>org.keycloak.cluster.infinispan.WrapperClusterEvent</class> <class>java.time.Instant</class> <regex>.*</regex> </allow-list> </serialization> </cache-container> </infinispan> the error is : 15:49:13,130 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start org.infinispan.manager.EmbeddedCacheManagerStartupException: .................. org.infinispan.persistence.file.SingleFileStore.**rebuildIndex**(SingleFileStore.java:395) at org.infinispan.persistence.file.SingleFileStore.blockingStart(SingleFileStore.java:205) ... 6 more it's such a weird error I tried making sure that the configuration is the culprit and it is, removing it lets infinispan pods start normally.

Just encountered this exact same issue when messing around with infinispan server.
After looking at the offending code, SingleFileStore#blockingStart, it seems that .dat files are being loaded and marshalled trough the configuration specified marshaller, if the marshaller is changed to another one, the files are not deleted, instead the code tries to load them as is, which leads to the stack trace above if the .dat are stored using something other than java.ObjectInputStream.

In order to solve it, i just deleted all .dat files. Exactly what it's the .dat files, I cannot say, since I still haven't had time to look at them.

Here's the code for the blockingStart.

Here's the code for getStoreFile

martin-kanis · 2022-11-01T11:59:54Z

martin-kanis
Nov 1, 2022
Collaborator

Hi, I tried current main branch (should be the same with Keycloak 19.0.3) with external Infinispan 14 server (No cross-cd mode, just sessions persistence to a remote ISPN server). Feel free to try it #13926 (comment)

0 replies

Keycloak && IS cluster #12239

Uh oh!

AlexeiKlimenko May 27, 2022

Describe the bug

Version

Expected behavior

Actual behavior

How to Reproduce?

Anything else?

Replies: 3 comments · 8 replies

Uh oh!

Uh oh!

AlexeiKlimenko May 27, 2022 Author

Uh oh!

AlexeiKlimenko May 31, 2022 Author

Uh oh!

darius-m May 31, 2022

Uh oh!

AlexeiKlimenko Jun 7, 2022 Author

Uh oh!

luccascbeltrame Jul 16, 2022

Uh oh!

Uh oh!

satanshiro Sep 20, 2022

Uh oh!

darius-m Sep 22, 2022

Uh oh!

Uh oh!

nunoOliveiraqwe Oct 31, 2022

Uh oh!

martin-kanis Nov 1, 2022 Collaborator

AlexeiKlimenko
May 27, 2022

Replies: 3 comments 8 replies

AlexeiKlimenko
May 27, 2022
Author

AlexeiKlimenko May 31, 2022
Author

darius-m
May 31, 2022

AlexeiKlimenko Jun 7, 2022
Author

martin-kanis
Nov 1, 2022
Collaborator