Repository navigation
Replies: 2 comments 21 replies
|
Thank you for starting this discussion. My 2 cents on this:
About checking if the database is readonly or not: I think I remember that you mentioned PostgreSQL as the database. This supports the I'm not sure what the instance should do when the database is no longer reachable, and how long it should take to kill/exit the Pod. Usually an overloaded database or a temporary network glitch are no good reasons to restart Pod, as it would lead to cascading failures. Let's defer that part. cc: @pruivo - what are your thoughts on this? |
|
@ahus1 I checked the Deployment behavior. A new pod is created before the old one is terminated. So unlike StatefulSets, the rollout doesn't get stuck when a pod is NotReady because the surge mechanism bypasses the maxUnavailable budget. However, this doesn't actually help in the warm standby case. All pods are intentionally NotReady because they're waiting for a writable database. A new surge pod will also wait and never become Ready, so the controller won't remove the old one. The rollout is stuck either way, Deployments just waste an extra pod while waiting. The core issue is that Kubernetes ties rollout progress to pod readiness. As long as standby pods report NotReady, neither workload type can roll them out. This would need to be solved at the Keycloak level, e.g. by decoupling the "waiting for DB" state from readiness so that Kubernetes still considers the pod replaceable. I will update my POC implementation with a new proposal. |
|
To your questions: I think we have enough to scope an enhancement request. I'd keep it as preview/experimental for now, agreed. The StatefulSet problem from my earlier comment is actually solved in this PoC by keeping readiness UP during the wait, so rolling updates aren't blocked. I used a feature flag (--features=warm-standby) in my PoC, but happy to switch to another CLI option if that's preferred. On database genericity: the current implementation catches the PostgreSQL-specific targetServerType=primary rejection. For other databases we'd need equivalent checks, but I think starting with PostgreSQL covers the main use case. Main change compared to my earlier proposals: the pod reports ready immediately, even before bootstrap completes. Traffic routing is controlled separately via a new health group endpoint (/health/group/active) that only returns UP after bootstrap is done. What the PoC does:
Two things worth considering longer term:
|
@msander I don't feel comfortable ignoring the definition of ready: |
|
@pruivo The Kubernetes docs say I looked into how other projects handle this. The closest match I found is HashiCorp Vault's DR replication. Vault DR secondaries are warm standby clusters that do not accept any client requests until they are promoted:
(https://developer.hashicorp.com/vault/tutorials/enterprise/disaster-recovery) Vault has the same readiness problem on Kubernetes: a DR secondary node returns HTTP 472 by default on the health endpoint, which fails a standard readiness probe. Vault's (https://developer.hashicorp.com/vault/api-docs/system/health , https://support.hashicorp.com/hc/en-us/articles/20953204088083-How-To-Setup-Readiness-Liveness-Probes-with-Replication) Note: I haven't tested this with Vault myself, this is based on their docs only. The alternative would be to keep standby pods NotReady and use the OnDelete StatefulSet update strategy, with the Keycloak Operator handling pod deletion in the right order. That would work, but it moves update orchestration into the operator. |
|
Pinging @vmuzikar @shawkins for awareness and expertise here.
@msander, that's true. However, everything is manually configured. If this is the path we want to move forward with, then your suggestion is fine, and you don't need to change the liveness probe. I like the idea of using the parameters in the health query, but I haven't checked the Quarkus configuration to see if it is supported. As a side note, replies from the team may slow down as we have our task list to finish for the Keycloak 26.6 release. |
|
We're currently exploring a active-hot-standby where the Keycloak instance in the other datacenter is actually running, see #49469 With that, the standby would connect in primary database and use it to receive for example invalidations. This would be sufficient for synchronous database failovers. For asynchronous database failovers, some data might be lost. For that, one would want to clear caches. For that, #46814 would need to be implemented. With one of those, or both, implemented, I don't think there is any additional need for additional functionality as long as the instances in the secondary DC can connect to the DB in the primary DC. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Problem
Keycloak currently expects the configured database to be writable during startup.
That becomes a problem in failover setups where Keycloak is already running in a standby region, but the local database is still a read-only replica. In that case Keycloak cannot finish startup, because startup still includes steps that need write access, for example schema migration and JDBC-Ping.
Also the database might become read-only again after Keycloak has already started.
Use case
The setup I have in mind is a warm standby across regions:
The main goal here is to keep failover time low once the database is writable.
What seems to be missing
What seems to be missing is a supported way to keep Keycloak blocked until the configured database is actually writable.
A command like
kc.sh wait-for-writable-dbwould already help. It could wait until the configured database accepts writes and then exit successfully.My first thought was to use that in a Kubernetes init container, but that does not really solve the whole problem, because init containers are not run again when the pod restarts.
So at the moment I see two viable options:
A) Wait in the container entrypoint
Run
kc.sh wait-for-writable-dbbeforekc.sh start.That would be simple and would not require changing Keycloak's startup model much. The downside is that while the container is waiting, Keycloak itself is not up yet, so its normal health endpoints are not available. Kubernetes would eventually kill the pod. To deal with that, we would need some lightweight health endpoint during the waiting phase.
B) Start partially, then finish initialization later
Start the server and expose health endpoints first, keep readiness DOWN, and wait in the background until the database becomes writable. After that, continue the remaining initialization and only then become ready.
Operationally this feels cleaner, but it is also a much bigger change because it would split initialization into an earlier and a later phase.
Runtime behavior
There is also the runtime side of this.
If the database turns read-only again later, I do not think readiness alone is enough. Keycloak cannot really keep operating in that state. It would probably need to leave the cluster and restart into the same waiting mode until the database is writable again.
Questions
All reactions