failed cluster: failed sending message to 127.0.0.1:0: java.net.ConnectException: Connection refused #44773

awoimbee · 2025-12-09T10:40:36Z

awoimbee
Dec 9, 2025

I had this issue last night where my keycloak cluster (3 replicas) was down until I ran a rollout of the statefulset.
I don't have all the info necessary to open an issue but still I want to leave this somewhere.

Using https://quay.io/phasetwo/phasetwo-keycloak 26.3.5.1759832119 without the keycloak operator.

Extract form the logs:

2025-12-08 18:35:14,311 ERROR [org.jgroups.protocols.TCP] (pd-bundler-13,keycloak-keycloakx-1-6428) keycloak-keycloakx-1-6428: failed sending message to keycloak-keycloakx-2-16433: java.net.ConnectException: Connection refused
2025-12-08 18:35:45,538 ERROR [org.jgroups.protocols.TCP] (pd-bundler-15,keycloak-keycloakx-1-6428) keycloak-keycloakx-1-6428: failed sending message to (starting): java.net.ConnectException: Connection refused
2025-12-08 18:35:45,857 ERROR [org.jgroups.protocols.TCP] (pd-bundler-14,keycloak-keycloakx-0-23055) keycloak-keycloakx-0-23055: failed sending message to 127.0.0.1:0: java.net.ConnectException: Connection refused
2025-12-08 18:35:45,938 ERROR [org.jgroups.protocols.TCP] (pd-bundler-15,keycloak-keycloakx-1-6428) keycloak-keycloakx-1-6428: failed sending message to (starting): java.net.ConnectException: Connection refused
2025-12-08 18:35:46,138 ERROR [org.jgroups.protocols.TCP] (pd-bundler-15,keycloak-keycloakx-1-6428) keycloak-keycloakx-1-6428: failed sending message to (starting): java.net.ConnectException: Connection refused
2025-12-08 18:35:47,501 WARN  [org.jgroups.protocols.pbcast.GMS] (main) keycloak-keycloakx-2-62989: JOIN(keycloak-keycloakx-2-62989) sent to keycloak-keycloakx-1-6428 timed out (after 2000 ms), on try 3
2025-12-08 18:35:51,507 WARN  [org.jgroups.protocols.pbcast.GMS] (main) keycloak-keycloakx-2-62989: JOIN(keycloak-keycloakx-2-62989) sent to keycloak-keycloakx-1-6428 timed out (after 2000 ms), on try 5
2025-12-08 18:42:54,074 ERROR [org.jgroups.protocols.TCP] (pd-bundler-22,keycloak-keycloakx-2-62989) keycloak-keycloakx-2-62989: failed sending message to 127.0.0.1:0: java.net.ConnectException: Connection refused

Meaningful kubernetes events:

keycloak-keycloakx-1-6428:
    15:42:46.832: started
    19:41:17.000: evicted
    19:41:35.000: recreated

keycloak-keycloakx-0-23055
    15:43:13.000: started
    
keycloak-keycloakx-2-62989
    15:42:03.000: started
    19:35:14.000: evicted
    19:35:16.000: recreated

(eviction was from node under-utilization)

awoimbee · 2026-01-20T14:57:49Z

awoimbee
Jan 20, 2026
Author

This keeps happening

0 replies

shawkins · 2026-01-23T15:10:48Z

shawkins
Jan 23, 2026
Collaborator

cc @pruivo @ryanemerson

0 replies

pruivo · 2026-01-23T16:00:24Z

pruivo
Jan 23, 2026
Collaborator

@awoimbee Those warnings are usually harmless. They were logged only at trace level before, but we changed them to warnings to increase visibility when debuggi 8000 ng network problems. (commit: belaban/JGroups@1928ee6#diff-7e18f6d55051953fa7d332946acbab809f40b0ccffab21ed2d4579399b8e8afc)

Are you observing any other errors or exceptions?

3 replies

ryan-odyssey Jan 27, 2026

i think the problem is that it's logged as an error, not a warning. If it were a warning, they it wouldn't trigger alarms in our system

awoimbee Jan 27, 2026
Author

Hello, I have observed that keycloak can get stuck trying to reach 127.0.0.1:0, preventing it from forming a cluster, preventing it from being Ready to respond to requests.

I understand failed sending message to keycloak-keycloakx-2-14888, I understand failed sending message to (starting), but failed sending message to 127.0.0.1:0 ?

pruivo Jan 27, 2026
Collaborator

The 127.0.0.1:0 is the (starting) instance, which does not have an IP assigned yet.
I'll open a PR for JGroups to reduce the log level to warning, except if one of you wants to contribute to JGroups.

Navigation Menu

Repository navigation

failed cluster: failed sending message to 127.0.0.1:0: java.net.ConnectException: Connection refused #44773

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

failed cluster: failed sending message to 127.0.0.1:0: java.net.ConnectException: Connection refused #44773

Uh oh!

Uh oh!

awoimbee Dec 9, 2025

Replies: 3 comments · 3 replies

Uh oh!

awoimbee Jan 20, 2026 Author

Uh oh!

shawkins Jan 23, 2026 Collaborator

Uh oh!

pruivo Jan 23, 2026 Collaborator

Uh oh!

ryan-odyssey Jan 27, 2026

Uh oh!

awoimbee Jan 27, 2026 Author

Uh oh!

pruivo Jan 27, 2026 Collaborator

awoimbee
Dec 9, 2025

Replies: 3 comments 3 replies

awoimbee
Jan 20, 2026
Author

shawkins
Jan 23, 2026
Collaborator

pruivo
Jan 23, 2026
Collaborator

awoimbee Jan 27, 2026
Author

pruivo Jan 27, 2026
Collaborator