Repository navigation
Implement Distributed Cache with Kubernetes Cache Stack #37647
|
Hi everyone, I would like to share my experience about how to implement distributed cache on Keycloak version 23.x with Kubernetes Cache. In this post, I'm also share the errors, the cause of errors, and how to fix it. I deploy Keycloak on GKE. I need your feedback if there's a thing that I'm missing in this implementation. This post is a part of my posts in my personal blog: https://dev.kresna.me/tags/keycloak/. Thank you and happy reading! Increase Replica to NSo far, we just use 1 replica in our deployment file. But, we will update the replica to N. The value of N may vary, in this case I set the value of N to 3. Here's the snippet of deployment.yaml file. # deployment.yaml
spec:
replicas: 3
template:
spec:
containers:
# ... Same as previous guidelineThat's it! Then, apply the change with command Error 1: Redirect LoopI bet you will see the and error page from you browser. I take from Mozilla as example.
I was confused a bit when I see the error. At that time, I don't know what's going on! I make guesses at that time:
The second guess is the answer. When you increase your replica to N like 2 or above, Keycloak will need a way to figure out which pod currently user access the Keycloak and Keycloak need to stick their sessions across the pods in order to prevent the redirect loop issue. You may need to read Enable sticky sessions from Keycloak Docs in order to get the full picture. In this post, I will take a piece of docs and I put in here.
If I access the Admin Console login page then open the browser console in tab Storage > Cookies. I see the AUTH_SESSION_ID has value like this:
Keycloak creates the cookie AUTH_SESSION_ID with the format .. The session id is In summary, we need to tell our loadbalancer which is ingress-nginx to stick over the AUTH_SESSION_ID cookie in order to solve the redirect loop issue. So, let's get started! Fix Error 1: Redirect LoopLet's update the apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak-ingress-public
namespace: keycloak
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "AUTH_SESSION_ID"
spec:
ingressClassName: nginx
# ... Same as previous guideline
tls:
- hosts:
- sso.domain.tld
secretName: keycloak-tls-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
8000
name: keycloak-ingress-admin
namespace: keycloak
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "AUTH_SESSION_ID"
spec:
ingressClassName: nginx
# ... Same as previous guideline
tls:
- hosts:
- iam.domain.tld
secretName: keycloak-tls-secretThen, apply the change by run Please visit github.com/senkulabs/keycloak-playground then search Enable Distributed CacheKeycloak support two cache stacks: The However, production-ready deployments should always consider session affinity and favor redirecting users to the node where their sessions were initially created. By doing that, you are going to avoid unnecessary state transfer between nodes and improve CPU, memory, and network utilization.
Ok! Let's update the deployment.yaml file. # deployment.yaml
spec:
template:
spec:
containers:
- name: keycloak
image: asia-southeast2-docker.pkg.dev/project-id/project-id/keycloak:latest
args:
- "start"
env:
- name: KC_CACHE
value: "ispn"
- name: KC_CACHE_STACK
value: "kubernetes"
- name: JAVA_OPTS_APPEND
value: "-Djgroups.dns.query=keycloak-service.keycloak.svc.cluster.local"
# ... Same as previous guidelineFirst, we add Let's break down the format of Kubernetes DNS:
So, in other words looks like this:
Ok, let's apply the update by run Error 2: 401 UnauthenticatedSo, maybe you can get inside the Admin Master Console after successful login and maybe not. Then, you will see the 401 HTTP code in your Admin Master Console page or in browser console. This happen because the We can also check the logs with command Appending additional Java properties to JAVA_OPTS: -Djgroups.dns.query=keycloak-service.keycloak.svc.cluster.local
2025-02-19 04:38:37,640 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel ISPN with stack kubernetes
2025-02-19 04:38:37,644 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: f829b1c3-5389-4aea-9bb7-8ea9541af869, name: keycloak-deployment-c65548c97-sjhf4-16255
2025-02-19 04:38:37,671 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.57800
2025-02-19 04:38:39,741 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-deployment-c65548c97-sjhf4-16255: no members discovered after 2003 ms: creating cluster as coordinator
2025-02-19 04:38:39,756 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-deployment-c65548c97-sjhf4-16255|0] (1) [keycloak-deployment-c65548c97-sjhf4-16255]
2025-02-19 04:38:39,941 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel ISPN local address is keycloak-deployment-c65548c97-sjhf4-16255, physical addresses are [10.28.3.20:7800]
2025-02-19 04:56:47,602 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,keycloak-deployment-c65548c97-sjhf4-16255) JGRP000006: 10.28.3.20:7800: failed accepting connection from peer Socket[addr=/10.28.4.27,port=41611,localport=7800]: java.io.EOFExceptionIn the log above, we see an important message: keycloak-deployment-c65548c97-sjhf4-16255: no members discovered after 2003 ms: creating cluster as coordinatorThis means the pod keycloak-deployment-c65548c97-sjhf4 is starting its own cluster instead of joining an existing one, which breaks the distributed cache. The other message is: JGRP000006: 10.28.3.20:7800: failed accepting connection from peer Socket[addr=/10.28.4.27,port=41611,localport=7800]: java.io.EOFExceptionThis suggests connectivity issues between Keycloak nodes over JGroups. The root cause is in
Fix Error 2: 401 UnauthenticatedLet's update the deployment.yaml and service.yaml files. In deployment.yaml file, we update the value of # deployment.yaml
spec:
template:
spec:
containers:
- name: keycloak
image: asia-southeast2-docker.pkg.dev/project-id/project-id/keycloak:latest
args:
- "start"
env:
# ... Same as previous section
- name: JAVA_OPTS_APPEND
value: "-Djgroups.dns.query=keycloak-discovery.keycloak.svc.cluster.local"
# ... Same as previous guideline
ports:
- name: http
containerPort: 8080
- name: jgroups-bind
containerPort: 7800
- name: jgroups-fd
containerPort: 57800In service.yaml, we add new line separated with apiVersion: v1
kind: Service
metadata:
name: keycloak-service
namespace: keycloak
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: keycloak
---
apiVersion: v1
kind: Service
metadata:
name: keycloak-discovery
namespace: keycloak
spec:
clusterIP: None
ports:
- port: 7800
targetPort: 7800
protocol: TCP
name: jgroups-bind
- port: 57800
targetPort: 57800
protocol: TCP
name: jgroups-fd
selector:
app: keycloakRun Received new cluster view for channel ISPN: [keycloak-deployment-5556b54567-wgdsr|1] (3) [keycloak-deployment-wgdsr, keycloak-deployment-xyz, keycloak-deployment-abc]Please visit github.com/senkulabs/keycloak-playground then search ExtraWhen you integrate Keycloak SSO with many client apps, there will be a case that you will get 502 Bad Gateway Nginx. If we check the log with command 2025/02/20 08:04:12 [error] 713#713: *202851 upstream sent too big header while reading response header from upstreamThis means that the backend server (upstream) in this case is Keycloak, is sending a response header that is larger than Nginx's configured buffer size. To solve this issue, we will increase the proxy-buffer-size greater than 4k which is the default value. # ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak-ingress-public
namespace: keycloak
annotations:
# ... Same as previous section
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
spec:
ingressClassName: nginx
# ... Same as previous guideline
tls:
- hosts:
- sso.domain.tld
secretName: keycloak-tls-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak-ingress-admin
namespace: keycloak
annotations:
# ... Same as previous section
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
spec:
ingressClassName: nginx
# ... Same as previous guideline
tls:
- hosts:
- iam.domain.tld
secretName: keycloak-tls-secretApply the change with command ClosingThat's it! We learn how to implement distributed cache on Keycloak in Kubernetes environment. Thanks for reading! |
Replies: 5 comments 16 replies
|
@kresnasatya the keycloak discussions are not a good place to post a blog entry. @jonkoops @vmuzikar should this be locked or simply deleted? |
|
I don't have any strong opinion to be honest, as long as there are no commercial interests it doesn't bother me in particular. |
|
Ok, we'll leave it up then. It's good to also mention that the operator handles all of what is described except for creating the admin ingress. |
|
Hi, thank you for your warning. I’m apologize for this case. If the next time I want to share about the things I have try on Keycloak in production mode like GKE, which forum should I use? Because, with the share things I have try or anybody try on Keycloak, we can learn from each other and maybe someone who expert with Keycloak in production mode may give suggestions. Usually, in some open source projects on GitHub, they open topic like “Share what you have achieved by using our open source projects”. |
Is the operator that mentioned in this statement is a Keycloak Operator? |
|
@kresnasatya A better format would be like an announcement - just link to your content with a description of who it would be of interest for.
That seems good - a category like https://github.com/keycloak/keycloak/discussions/categories/announcements but for non-maintainers. Let's see what others think as well.
Yes |
I'm agree with this! How about the category name is "Showcase 😎" or "Sharing 💁🏻"? I think this category name is suitable for anyone who try Keycloak on production mode for their projects or for their companies, so anyone can see and learn from the real world use case. For Keycloak Operator, I'm still bit not familiar with it. But, I have intuition that it will summarize or simplify my Keycloak deployment setup with Kubernetes. Hmm, I think I need to learn and understand how to use Kubernetes Operator first, right? Please correct me if I'm wrong. Thank you. |
Yes, have a look at the operator. See https://www.keycloak.org/oper 8000 ator/basic-deployment to get an idea of what the Keycloak custom resource looks like. I should also mention that the proxy-buffer-size is not something that is handled automatically by the operator, but you can add that to the Keycloak CR spec.ingress.annoations Also see the overview of scaling https://www.keycloak.org/getting-started/getting-started-scaling-and-tuning - we're still recommending StatefulSet instead of a Deployment. |
|
Thank you for your guidance, Steven. |
|
Thank you for this post. There is one more thing: We would recommend using a stateful set for Keycloak, as this allows for a better handling for scaling up/down, and avoid loosing sessions. |
|
@kresnasatya - can you please share which documentation you used to get started? Was it https://www.keycloak.org/getting-started/getting-started-kube? I see that this getting-started guide might be setting people on a wrong track as the quickstart is missing a lot of the crucial elements for people who want to run things on Kubernetes. @shawkins - I would suggest for https://github.com/keycloak/keycloak-quickstarts/blob/main/kubernetes/keycloak.yaml to include all the missing bits about caching and clustering including the service. For that, it would also make sense to include a database Pod. Let me know your thoughts on this from the two of you. |
There is a note in the getting started guide referring users to the scaling guide, which then also refers users to the operator. I see we are missing a direct referral to the operator from the kube getting started. The scaling guide covers the statefulset consideration that we both mentioned here.
A concern would be that we're not even keeping a simple example up-to-date: keycloak/keycloak-quickstarts#667 I'm fine with keeping this quickstart as simple as possible as long as we're effectively directing users to this advice
That is certainly doable. I would wonder if there was some hesitency to include that given the implications of the older operator seemingly managing the database. Can you capture a quickstart issue for that? |
Hi Alex, yes I'm using this as starting point especially in Start Keycloak section: https://www.keycloak.org/getting-started/getting-started-kube#_start_keycloak I see the Honestly, I'm seeking a spare time for explore the Kubernetes Operator and Keycloak Operator. If I see the benefits then I would like to implement it on my Keycloak deployment setup. Also, I'm being fortunate to give an access from my company (university) to explore Keycloak deployment into Kubernetes (GKE for example). I spend around 3 weeks to try and error then documented what I learn into a blog posts: https://dev.kresna.me/tags/keycloak/. Why 3 weeks? Because, Kubernetes is a new area for me. I have gooling and debate with LLM all day. XD So, If I'm a beginner in topics of deploy Keycloak into Kubernetes maybe the guidelines looks like this:
I have a plan to make a post about upgrading Keycloak from Keycloak 23 to 24 in Kubernetes environment. Two things that I notice since upgrade to 24 are: proxy mechanism change and I don't need to set sticky session or affinity session |
|
Hi @kresnasatya do you mind sharing more details on why AUTH_SESSION_ID would no be be needed any more? Trying now to set up session affinity i see that nginx would not use the backend cookie but create its own cookie: |
Hi, I’m sorry for late reply. If I remember correctly, this is because since Keycloak version 24, they refactor Keycloak Cookie. That’s a reason I omit the AUTH_SESSION_ID. Link: https://www.keycloak.org/docs/latest/upgrading/index.html#updates-to-cookies (Go to section Migration Changes > Migrate to 24.0.0). Hope it helps. |
|
I have the same problem, however i use image quay.io/keycloak/keycloak:25.0.6 |
|
Hi, could you please explain what are the same problems you are facing in Keycloak deployment? |
apiVersion: v1
|
|
Hi! I see the problem here. The problem is you haven't made the headless service ( |
|
Thanks for your article, it helped me solve it |
|
Glad to hear that! |
@kresnasatya A better format would be like an announcement - just link to your content with a description of who it would be of interest for.
That seems good - a category like https://github.com/keycloak/keycloak/discussions/categories/announcements but for non-maintainers. Let's see what others think as well.
Yes