Repository navigation
Help with Clustering using JDBC_PING with Mysql and Keycloak 17 Quarkus on 2 AWS ECS Containers Running EC2 Instances behind ELB Load Balancer #23400
|
Hello everyone, I've been working on upgrading our Keycloak instances from 16 Wildfly to 17 Quarkus, and enabling clustering while I'm at it. I've currently gotten the upgrade working and reachable with one instance, but I'm really struggling to get two instances to actually communicate and cluster properly. I want to preface that I'm much more back end server code oriented and very much out of my element with a lot of this, so I've been piecing a lot of new information together and apologize beforehand for any odd conventions or obvious mistakes. I've based a lot of my current configuration off of the suggestions from the Discourse discussion here. Here's my Dockerfile mykc.sh is a simple shell script to grab the ecs container's Port and IP for the JDBC_PING, since these change on every deployment, then starts the kc.sh script after setting these variables. And then my cache-ispn-jdbc-ping.xml file was pieced together from the Discourse discussion mentioned above. Here's the jgroup definition In container A, I can see the startup and correctly creating the jgroup channel and the corresponding entry in the database is created with the proper container IP set as the bind address.
But when I stand up the second instance, it fails to connect 10 times until it gives up and makes its own singleton.
I have the security group in AWS set up to allow all TCP traffic from Container 1's IP CIDR to all ports. I logged in to Container 2 and tried to telnet via curl to Container 1 but the connection gets refused, but each Container can telnet to itself via localhost. Oddly enough I can telnet to the other ports set up on Container 1 (this instance is not the only one on the box) but trying to hit the port specified for the JDBC_ping specifically refuses the connection. With the above example that would be Container1IP:33116 being denied. I'm leaning towards this being an issue with the AWS configuration of the container's networks potentially since it seems like the Keycloak instance is properly getting the ip and port specified to attempt the connection and each container can connect to itself, but any insight would be massively appreciated. |
Replies: 1 comment
|
In case anyone stumbles upon this in the future, the issue was the fact that the AWS Target Group was set as Bridge mode instead of VPC mode, which was preventing each Keycloak instance from having its own IP. Switching everything over to use VPC mode with the target group being IP Address instead of Instance based let the Keycloak Instances communicate with each other and properly cluster. Leaving the target IP Addresses on the target group as blank allows the Load Balancer to assign the IPs. We also had to turn off the public subnets for our VPC and only assign the service to use the private subnets. |
In case anyone stumbles upon this in the future, the issue was the fact that the AWS Target Group was set as Bridge mode instead of VPC mode, which was preventing each Keycloak instance from having its own IP. Switching everything over to use VPC mode with the target group being IP Address instead of Instance based let the Keycloak Instances communicate with each other and properly cluster. Leaving the target IP Addresses on the target group as blank allows the Load Balancer to assign the IPs. We also had to turn off the public subnets for our VPC and only assign the service to use the private subnets.