Keycloak in kubernetes cluster #12594

Parth6288 · 2022-06-20T11:52:19Z

Parth6288
Jun 20, 2022

HI Team,

I am new to keycloak and deployed it on a kubernetes cluster with standalone ha cluster mode. I created a new realm and integrated it with AD. However when I redeployed with a new image, all my changes were gone. I had to re-create the realm and integrate it again with the AD.

Where are this config stored with in the pods? How do I ensure that this changes are persisted across my releases?

kkcmadhu · 2022-06-21T08:45:36Z

kkcmadhu
Jun 21, 2022

the stadalone.xml , standalone-ha.xml kind of configurations are only for wildfy deployments, if i understand it right you are using quarks based distribution.

You can put the jdbc details in you config map

I use the below yaml for bring up my keycloak on quarkus with mysql.. you can cange the db url to use postgres
Note :: if your keycloak is not behind reverse proxy with does ssl termination, you can remove the proxy env variable and add relavelt certificattes here.. the configuration details are availble in config guie https://www.keycloak.org/server/all-config

apiVersion: v1
kind: Service
metadata:
  name: keycloak18-headless
spec:
  clusterIP: None
  selector:
    app: keycloak18
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: http
    - name: jgroups
      port: 7600
      protocol: TCP
      targetPort: jgroups
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: keycloak18
  labels:
    app: keycloak18
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak18
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak18
  labels:
    app: keycloak18
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak18
  template:
    metadata:
      labels:
        app: keycloak18
    spec:
      containers:
      - name: keycloak18
        image: quay.io/keycloak/keycloak:18.0.0
        args: ["start-dev"]
        env:
        - name: KEYCLOAK_ADMIN
          value: "admin"
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: "admin"
        - name: KC_PROXY
          value: "edge"
        - name: KC_DB
          value: "mysql"
        - name: KC_DB_URL
          value: "jdbc:mysql://mysqlhost:3306/keycloakonquarkus"        
        - name: KC_DB_USERNAME
          value: "mysqluser"
        - name: KC_DB_PASSWORD
          value: "mysqlpassword"
        - name: KC_HEALTH_ENABLED
          value: "true"
        - name: KC_METRICS_ENABLED
          value: "true"
        - name: KC_FEATURES
          value: "token-exchange,preview,scripts"
        - name: KC_HTTP_RELATIVE_PATH
          value: "/auth"
        - name: KC_SPI_ADMIN_REALM
          value: "yourmasterrealmname"
        ports:
        - name: http
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /auth/realms/yourmasterrealmname
            port: 8080

0 replies

kkcmadhu · 2022-06-20T13:59:22Z

kkcmadhu
Jun 20, 2022

if you are using oob keycloak then it will be using h2 database, which will store stuff in local pod.
better use a proper db (recommended is postgres) and configure the same in standalone.xml

Also, keycloak is moving away from wildfly based depoloyment, i would recommend using the quarkus distribution of keycloak (there too you will have to configure database)

0 replies

Parth6288 · 2022-06-20T15:53:35Z

Parth6288
Jun 20, 2022
Author

Thanks @kkcmadhu. We are using the postgresSQL and have provided the details in our helm charts however we see the realms configs are being stored in the local h2 database. so when a pod, that has this DB, dies the config is gone. Looking the docs we need to update the standalone.xml file for this config however we don't see this file in our pods. What path can we find it?

0 replies

Parth6288 · 2022-06-20T15:54:25Z

Parth6288
Jun 20, 2022
Author

We are already using the quarkus deployment(version 17.0)

0 replies

Parth6288 · 2022-06-21T15:34:29Z

Parth6288
Jun 21, 2022
Author

Thanks @kkcmadhu . Here is my deployment file. Am I using the wrong env variables? I would also want this deployed via helm charts in future and I am working on creating the values file.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "19"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"keycloak"},"name":"keycloak","namespace":"kc"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"selector":{"matchLabels":{"app":"keycloak"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app":"keycloak"}},"spec":{"containers":[{"args":["start","--hostname=kc.k8.nonprod.abc.com","--https-certificate-file=/opt/pem/cert-pem/cert.pem","--https-certificate-key-file=/opt/pem/key-pem/key.pem","--log-level=DEBUG"],"env":[{"name":"KEYCLOAK_ADMIN","value":"admin"},{"name":"KEYCLOAK_ADMIN_PASSWORD","value":"test@123"},{"name":"PROXY_ADDRESS_FORWARDING","value":"true"},{"name":"DB_ADDR","value":"database-2.c657irl4zbya.us-east-1.rds.amazonaws.com"},{"name":"DB_DATABASE","value":"postgres"},{"name":"DB_USER","value":"postgres"},{"name":"DB_PASSWORD","value":"admin1234"},{"name":"DB_SCHEMA","value":"public"},{"name":"DB_VENDOR","value":"POSTGRES"},{"name":"JGROUPS_DISCOVERY_PROTOCOL","value":"dns.DNS_PING"},{"name":"JGROUPS_DISCOVERY_PROPERTIES","value":"dns_query=keycloak"},{"name":"CACHE_OWNERS_COUNT","value":"2"},{"name":"CACHE_OWNERS_AUTH_SESSIONS_COUNT","value":"2"}],"image":"quay.io/keycloak/keycloak:17.0.0","imagePullPolicy":"IfNotPresent","name":"keycloak","ports":[{"containerPort":7600,"name":"jgroups","protocol":"TCP"},{"containerPort":8080,"name":"http","protocol":"TCP"},{"containerPort":8443,"name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/realms/master","port":8443,"scheme":"HTTPS"},"initialDelaySeconds":30,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":30},"resources":{},"securityContext":{"runAsUser":0},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/opt/pem/key-pem","name":"key-pem"},{"mountPath":"/opt/pem/cert-pem","name":"cert-pem"}]}],"dnsPolicy":"ClusterFirst","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"name":"key-pem"},"name":"key-pem"},{"configMap":{"name":"cert-pem"},"name":"cert-pem"}]}}}}
  creationTimestamp: "2022-04-18T12:53:02Z"
  generation: 27
  labels:
    app: keycloak
  name: keycloak
  namespace: kc
  resourceVersion: "28999775"
  uid: 714bba17-f9c2-4272-a85f-08b798a320dd
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: keycloak
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: keycloak
    spec:
      containers:
      - args:
        - start
        - --hostname=kc.k8.nonprod.abc.com
        - --https-certificate-file=/opt/pem/cert-pem/cert.pem
        - --https-certificate-key-file=/opt/pem/key-pem/key.pem
        - --log-level=DEBUG
        env:
        - name: KEYCLOAK_ADMIN
          value: admin
        - name: KEYCLOAK_ADMIN_PASSWORD
          value: test1234
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        - name: DB_ADDR
          value: database-1.c657irlijuio.us-east-1.rds.amazonaws.com
        - name: DB_DATABASE
          value: postgres
        - name: DB_USER
          value: postgres
        - name: DB_PASSWORD
          value: keycloak1234
        - name: DB_SCHEMA
          value: public
        - name: DB_VENDOR
          value: POSTGRES
        - name: JGROUPS_DISCOVERY_PROTOCOL
          value: dns.DNS_PING
        - name: JGROUPS_DISCOVERY_PROPERTIES
          value: dns_query=keycloak
        - name: CACHE_OWNERS_COUNT
          value: "2"
        - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
          value: "2"
        image: quay.io/keycloak/keycloak:17.0.0
        imagePullPolicy: IfNotPresent
        name: keycloak
        ports:
        - containerPort: 7600
          name: jgroups
          protocol: TCP
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /realms/master
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        resources: {}
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/pem/key-pem
          name: key-pem
        - mountPath: /opt/pem/cert-pem
          name: cert-pem
        - mountPath: /opt/keycloak/data
          name: keydata
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: key-pem
        name: key-pem
      - configMap:
          defaultMode: 420
          name: cert-pem
        name: cert-pem
      - emptyDir: {}
        name: keydata

5 replies

kkcmadhu Jun 21, 2022

you can compare with the config map i have shared, your parameter names do not seem to be standard ones.. the convention for quarkus keycloak is KC_DB KC_XXX etc instead of is DB_ADDR , DB_PASSWORD etc ... you double env section of your yaml..

You can go through the config documents for details (i have alrady shared in above post)

Parth6288 Jun 23, 2022
Author

So I was able to set the env variables as KC_DB and run our pods and it fails. Looking at the logs it is not finding the correct jdbc driver for postgresql. Below are the logs from my pods:

"ERROR: Failed to start server in (production) mode
2022-06-23 16:40:14,895 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
2022-06-23 16:40:14,895 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: No suitable driver found for jdbc:postgresql://keycl.c657irl4zbya.us-east-1.rds.amazonaws.com:5432/keycloakdbtest
2022-06-23 16:40:14,895 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command."

Is there any additional setting required for JDBC?

kkcmadhu Jun 23, 2022

please be aware to if you start using .kc.sh start then that the quarkus optimization/configuration is apriori, and needs to be setup.
you may want to try with .kc/sh start --auto-build param..

I think your crux of problem is solved here (not able to configure postgres url). please refer the documentation for further configurtions.

Parth6288 Jun 23, 2022
Author

thanks @kkcmadhu . I tried with the auto build option and getting below error now.

"[org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
2022-06-23 17:56:00,463 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
2022-06-23 17:56:00,464 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: connect timed out
2022-06-23 17:56:00,464 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command."

Parth6288 Jul 4, 2022
Author

I was able to get the keycloak working with my postgresql. However now I am seeing some issues using infinispan distributed caching with it. As per the doc with multi node eks cluster infispan is the default caching and I set the "CACHE_OWNERS_COUNT to 2". But when I check the pod logs I don't see the nodes joining the infinispan cluster. Here is my deployment yaml file: https://pastebin.com/kpMz4Yvt

Here are the pod logs:
"2022-07-04 17:51:35,003 DEBUG [org.infinispan.interceptors.impl.AsyncInterceptorChainImpl] (main) Interceptor chain size: 8
2022-07-04 17:51:35,003 DEBUG [org.infinispan.interceptors.impl.AsyncInterceptorChainImpl] (main) Interceptor chain is:
>> org.infinispan.interceptors.impl.BatchingInterceptor@5f1f0ee6
>> org.infinispan.interceptors.impl.InvocationContextInterceptor@326706d
>> org.infinispan.interceptors.impl.CacheMgmtInterceptor@76fc5687
>> org.infinispan.interceptors.impl.TxInterceptor@53fbb2b4
>> org.infinispan.interceptors.locking.PessimisticLockingInterceptor@7434ee13
>> org.infinispan.interceptors.impl.NotificationInterceptor@44a14de0
>> org.infinispan.interceptors.impl.EntryWrappingInterceptor@50fd739d
>> org.infinispan.interceptors.impl.CallInterceptor@5e1d1524
2022-07-04 17:51:35,005 DEBUG [org.infinispan.commons.util.ServiceFinder] (main) Loading service impl: org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator
2022-07-04 17:51:35,007 DEBUG [org.infinispan.persistence.manager.PreloadManager] (main) Preloaded 0 keys in 0 milliseconds
2022-07-04 17:51:35,007 DEBUG [org.infinispan.cache.impl.CacheImpl] (main) Started cache authorizationRevisions on keycloak
2022-07-04 17:51:35,008 DEBUG [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Using container managed Infinispan cache container, lookup=DefaultCacheManager keycloak-594794b847-wx85c-26716
2022-07-04 17:51:35,008 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-594794b847-wx85c-26716, Site name: null
2022-07-04 17:51:35,298 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) Will preload sessions with transaction timeout 300 seconds
2022-07-04 17:51:35,322 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) No remote store configured for cache 'sessions'
2022-07-04 17:51:35,323 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) No remote store configured for cache 'clientSessions'
2022-07-04 17:51:35,323 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) No remote store configured for cache 'offlineSessions'
2022-07-04 17:51:35,324 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) No remote store configured for cache 'offlineClientSessions'
2022-07-04 17:51:35,326 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) Skipping pre-loading of userSessions from persistent storage
2022-07-04 17:51:35,331 DEBUG [org.keycloak.cluster.infinispan.CrossDCAwareCacheFactory] (main) No configured remoteStore available. Cross-DC scenario is not used
2022-07-04 17:51:35,346 DEBUG [org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory] (main) Initialized cluster startup time to Mon Jul 04 17:51:29 GMT 2022
2022-07-04 17:51:35,348 DEBUG [org.keycloak.cluster.infinispan.InfinispanNotificationsManager] (main) Added listener for infinispan cache: work
2022-07-04 17:51:35,350 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserSessionProviderFactory] (main) Registered cluster listeners
2022-07-04 17:51:35,351 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanAuthenticationSessionProviderFactory] (main) Registered cluster listeners
2022-07-04 17:51:35,354 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserLoginFailureProviderFactory] (main) No remote store configured for cache 'loginFailures'
2022-07-04 17:51:35,355 DEBUG [org.keycloak.models.sessions.infinispan.InfinispanUserLoginFailureProviderFactory] (main) Registered cluster listeners
2022-07-04 17:51:35,361 DEBUG [org.keycloak.models.cache.infinispan.InfinispanCacheRealmProviderFactory] (main) Registered cluster listeners
2022-07-04 17:51:37,555 DEBUG [org.keycloak.models.cache.infinispan.InfinispanUserCacheProviderFactory] (main) Registered cluster listeners
2022-07-04 17:52:35,331 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.PersisterLastSessionRefreshStore] (Timer-0) Updating 0 userSessions with lastSessionRefresh: 1656957095
2022-07-04 17:53:35,331 DEBUG [org.keycloak.models.sessions.infinispan.changes.sessions.PersisterLastSessionRefreshStore] (Timer-0) Updating 0 userSessions with lastSessionRefresh: 1656957155"

Also attached is the infinispan config file from the pod.
latest infinispan (1).docx

kkcmadhu · 2022-07-04T18:15:11Z

kkcmadhu
Jul 4, 2022

name: JGROUPS_DISCOVERY_PROPERTIES
value: dns_query=keycloak

you may want to try
```
      - name: jgroups.dns.query
  value: "keycloak18-headless"
```
ensure you have a head less service which could retrun all node addresses

metadata:
name: keycloak18-headless
spec:
clusterIP: None
selector:
app: keycloak18
ports:
- name: http
port: 8080
protocol: TCP
targetPort: http
- name: jgroups
port: 7600
protocol: TCP
targetPort: jgroups
sessionAffinity: None
type: ClusterIP

4 replies

Parth6288 Jul 5, 2022
Author

You mean replace
name: JGROUPS_DISCOVERY_PROPERTIES
value: dns_query=keycloak

with

name: jgroups.dns.query
value: "keycloak18-headless"

after creating headless service?

kkcmadhu Jul 5, 2022

yes

Parth6288 Jul 11, 2022
Author

Thanks @kkcmadhu. We were able to create an embedded infinispan cluster in the existing keycloak pods, so do we need to secure the cache communication? I am assuming this is secure by default.

I am pretty new to keycloak and infinispan so apologizes if my questions sound little silly!

Parth6288 Jul 11, 2022
Author

Also do we need to create a separate deployment for infinispan? If so how do we integrate it with the keycloak deployment in kubernetes?

mazbeh · 2022-11-11T10:48:32Z

mazbeh
Nov 11, 2022

Hello every body,
@Parth6288 i get alos the same error for the DB ( postgres ) connection ( in my case the postgresql DB is local ). could you please give some tips , how did you solve it? I may need a separate service for the DB and poit to it in the deployment file, is that so?
Thanks in advance !

0 replies

Keycloak in kubernetes cluster #12594

Uh oh!

Replies: 7 comments · 9 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Parth6288 Jun 20, 2022 Author

Uh oh!

Parth6288 Jun 20, 2022 Author

Uh oh!

Uh oh!

Parth6288 Jun 21, 2022 Author

Uh oh!

Uh oh!

Parth6288 Jun 23, 2022 Author

Uh oh!

Uh oh!

Parth6288 Jun 23, 2022 Author

Uh oh!

Parth6288 Jul 4, 2022 Author

Uh oh!

metadata: name: keycloak18-headless spec: clusterIP: None selector: app: keycloak18 ports: - name: http port: 8080 protocol: TCP targetPort: http - name: jgroups port: 7600 protocol: TCP targetPort: jgroups sessionAffinity: None type: ClusterIP

Uh oh!

Parth6288 Jul 5, 2022 Author

Uh oh!

Uh oh!

Parth6288 Jul 11, 2022 Author

Uh oh!

Parth6288 Jul 11, 2022 Author

Uh oh!

Replies: 7 comments 9 replies

Parth6288
Jun 20, 2022
Author

Parth6288
Jun 20, 2022
Author

Parth6288
Jun 21, 2022
Author

Parth6288 Jun 23, 2022
Author

Parth6288 Jun 23, 2022
Author

Parth6288 Jul 4, 2022
Author

metadata:
name: keycloak18-headless
spec:
clusterIP: None
selector:
app: keycloak18
ports:
- name: http
port: 8080
protocol: TCP
targetPort: http
- name: jgroups
port: 7600
protocol: TCP
targetPort: jgroups
sessionAffinity: None
type: ClusterIP

Parth6288 Jul 5, 2022
Author

Parth6288 Jul 11, 2022
Author

Parth6288 Jul 11, 2022
Author