Repository navigation
Split brain due to PG outage #43192
Unanswered
naumanahmedkhan
asked this question in
Q&A
Split brain due to PG outage
#43192
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi all,
I ran into a problem in a 2-pod Keycloak cluster (running on Kubernetes with Postgres as the DB). During a node restart, one of the Postgres pods went down. The Keycloak pods were not on that node, but the DB outage triggered a chain reaction:
One Keycloak pod became unresponsive while waiting on Postgres.
Infinispan marked it as an “abrupt leaver,” and the other pod started rejecting its messages (ASYM_ENCRYPT).
The cluster kept trying to merge (ISPN000093 MERGED cluster view), but only one pod recovered its cache state.
Even after Postgres came back, the two Keycloak pods never fully re-established the cluster.
From what I can tell, a temporary DB outage caused one pod to appear “dead,” which led to ongoing split-brain behavior.
Questions:
Is this expected behavior when Postgres is unavailable?
Are there best practices to prevent Keycloak pods from being treated as leavers during DB issues?
Would running Postgres in HA mode be enough, or are there cluster/timeout configurations that should be tuned as well?
All reactions