Repository navigation
Replies: 15 comments 23 replies
|
I have been deploying Keycloak in bare deployments in customer environments, so no being able to use Kubernetes features and commonly used jdbc_ping for setting up the infinispan cache. Having documentation about how to properly setup jdbc_ping inside Keycloaks project documentation would be very much appreciated. Currently it is often very hairy to set this 10BC8 up correctly and debugging it not easy. Even better would be to have the db ready for this out-of-the box and also show in the UI if Keycloak is running in clustered mode. Thanks! |
Hi @chrisbra - this is a great input, we'll have a look how we can do this. |
|
It is possible to reuse the existing JDBC datasource provided by quarkus in infinispan instead of using a dedicated connection: <JDBC_PING stack.combine="REPLACE" stack.position="MPING"
datasource_jndi_name="jdbc/datasource"
initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL,cluster_name varchar(200) NOT NULL,updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP,ping_data BYTEA,constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))"
insert_single_sql="INSERT INTO JGROUPSPING (own_addr, cluster_name, ping_data) values (?, ?, ?)"
delete_single_sql="DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?"
select_all_pingdata_sql="SELECT ping_data FROM JGROUPSPING WHERE cluster_name=?" /> |
|
Thank you, this gets us one step closer for configuring the database! |
|
I just checked this and it seems that it no longer works, since the JNDI support is disabled by default in current quarkus versions for security reasons. This was to mitigate log4shell and similar attacks. It would be cool if the jdbc_ping would allow to specify something like a "datasource_resolver_class_name". hypotetical DataSourceResolver interface: interface DataSourceResolver {
DataSource getDataSource();
}Simple CDI based implementation that would work right away with Keycloak/Quarkus: package my.pkg;
public class CdiDataSourceResolver implements DataSourceResolver {
public DataSource getDataSource() {
return CDI.current().select(DataSource.class).get();
}
}or as suggested by Bela: package my.pkg;
import ...
public class CdiDataSourceResolver implements Supplier<DataSource> {
public DataSource get() {
return CDI.current().select(DataSource.class).get();
}
}hypotetical configuration: <JDBC_PING stack.combine="REPLACE" stack.position="MPING"
datasource_resolver_class_name="my.pkg.CdiDataSourceResolver"
...
/> |
|
Hello I set up multiple environments and always used the JDBC_PING mechanism. I have to admit I could not understand why this mechanism is not part of the documentation, so definitely I support adding it on the official documentation. |
|
Thank you for sharing this. Can you share with us an example configuration you are using? Did you find problems in configuring it, or problems running it in the past? |
|
I took inspiration from this example: https://gist.github.com/xgp/768eea11f92806b9c83f95902f7f8f80 |
|
A few years ago, we used JDBC_PING for the mentioned reasons. At that time, it worked so-so. The biggest problem was that for some reason old entries were not removed correctly. Over time this led to large startup times as each joining node tried to connect the list of nodes in the DB unsuccessfully. However, it could be we did something wrong or the problem is fixed in the meantime. We switched to DNS_PING since we run in Kubernetes and didn't have any problems with that ever since. |
|
Therefore I'm writing Let's use Note btw that a JGroups stack (5.x) can have multiple discovery protocols, e.g. |
|
That sounds very promising. Thanks for the detailed look behind the scenes! Would you mind taking a look at this idea to make the datasource resolution pluggable? |
|
Injecting a |
|
The next Infinispan 15 release will support injecting the data source: https://issues.redhat.com/browse/ISPN-15961 |
|
Even with the occasional zombie entries as mentioned by @sschu above, we've been using |
|
I'm not going to deprecate JDBC_PING; it will get some new features, e.g. [1]. At the same time, JDBC_PING2 should be simpler to run, but let the community decide which one will win in the long run. |
|
@xgp - could you please elaborate on "with very few issues" on what those issues were? Thanks! |
|
From memory, we had
|
|
JDBC_PING makes a lot of sense to me for two reasons:
|
|
JDBC_PING or JDBC_PING2 sound great. I'd just add:
|
|
@pauliuspetk : if you refer to an existing datasource, then - yes - that can be injected into JDBC_PING{2}, either via JNDI or |
|
Nice! Since when is the @ahus1 I think it would be great if we had a default implementation for a |
|
You need to inject the data source into Infinispan configuration: pruivo@5f75f5f#diff-fe85ff9d09b49c428eb6a66276d3f68fc0d8c329cbb906afd2b59c81ca4e83fcR191 |
|
Thanks @pruivo that's also a nice solution, as we wouldn't need a new class to inject the database. |
|
The idea is to use Keycloak to create the table and generate the queries in the right dialect. But, I'm stuck (not a DB expert). ps. |
|
Providing a link to some branch here "as is" is IMHO all it needs for someone to pick it up. We'll need to wrap up our current tasks and see if this one end up on top in our next planning. If so, we can continue. In the meantime, someone else can give it a go. |
|
Hi @thomasdarimont : since 5.3.7 (https://issues.redhat.com/browse/JGRP-2795). If you need this to be available in a different version, it should be pretty painless to backport... |
|
Thanks! Keycloak 25.0.1 is currently using 5.3.6.Final So we would either manually upgrade to 5.3.7 or wait for the next quarkus / infinispan upgrade, what ever happens first for 26.x (or perhaps a 25.x intermediate release) |
|
Documentation: http://www.jgroups.org/manual5/index.html#JDBC_PING2 |
|
@thomasdarimont 5.3.7 is backwards-compatible to 5.3.6, so it should be simple to manually replace the JGroups JAR for testing this feature. |
|
Thank you Thomas! @pruivo was giving it a try as well. We would eventually have the table for JGroups managed by Liquibase, and that would require some more wiring to get the right startup order. See https://github.com/pruivo/keycloak/tree/t_29399 for the branch. We're currently busy with some other bits and pieces... if you want to give it a try with a proper PR, let us know. We might add some more items to the bill :-) |
|
I'll give it a try. I looked at @pruivo 's branch with the hardcoded JDBC_PING configuration. I think this will work, but if customers want to use a different JGroups configuration they still need to replace the whole infinispan configuration file. How about moving the jgroups config from the infinispan cache configuration into a separate file? I think the following could work and make it easier for customers to replace the jgroups configuration with their own, by refering to a custom jgroups stack config file via ispn-cache.xml: <infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd"
xmlns="urn:infinispan:config:15.0">
<jgroups>
<stack-file name="keycloak" path="${keycloak.jgroups.stackfile:jgroups-jdbc.xml}"></stack-file>
</jgroups>
<cache-container name="keycloak">
<transport lock-timeout="60000" stack="keycloak"/>
...jgroups-jdbc.xml: <!--
JDBC_PING2 for Postgresql
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_addr="${jgroups.bind_addr:site_local}"
bind_port="${jgroups.bind_port:7800}"
port_range="50"
recv_buf_size="150000"
send_buf_size="640000"
sock_conn_timeout="300ms"
/>
<JDBC_PING2
remove_all_data_on_view_change="true"
register_shutdown_hook="true"
return_entire_cache="false"
<!--
insert_sp="CREATE PROCEDURE deleteAndInsert
(addr varchar(200), name varchar(200), cluster varchar(200), ip varchar(200), coord boolean)
LANGUAGE SQL
BEGIN ATOMIC
DELETE from jgroups where address = addr;
INSERT INTO jgroups VALUES(addr, name, cluster, ip, coord);
END"
call_insert_sp="call deleteAndInsert(?,?,?,?,?);"
-->
/>
<MERGE3 min_interval="10000"
max_interval="30000"/>
<FD_SOCK2/>
<FD_ALL3 timeout="40s" interval="5s" />
<VERIFY_SUSPECT2 />
<pbcast.NAKACK2
use_mcast_xmit="false"
xmit_interval="500ms"/>
<UNICAST3
xmit_interval="500ms"/>
<pbcast.STABLE
desired_avg_gossip="5s"
max_bytes="1000000"/>
<pbcast.GMS
print_local_addr="false"
join_timeout="1s"
max_join_attempts="1"/>
<UFC max_credits="2M"
min_threshold="0.40"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG3 frag_size="60000" />
<pbcast.STATE_TRANSFER/>
</config>@belaban is it possible to use inheritance of jgroups configurations in jgroups config files? This could help reduce the config quite a bit. jgroups-jdbc.xml: <config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-5.3.xsd"
extends="tcp">
<JDBC_PING stack.combine="REPLACE" stack.position="MPING"/>
</config> |
|
@thomasdarimont if the user wants to use any other defaults shipped by Infinispan, the user can use |
|
Thanks for the reminder. That's used quite often, however I wondered if we can be more flexible here with the groups stack configuration. |
|
IMHO we shouldn't be more flexible with the JGroups stack configuration, as people always struggle here. This thread originally aims to find a solution to resolve 80% problems users have around the disovery, and making the remaining 20% simpler. If we don't think JDBC Ping can provide this, we'll need to re-plan. It should get the discovery on the same level of convenience as DNS Ping is proving on the Kubernetes level. |
|
I am fine if JDBC_PING is the default in the future but I would not like to remove support for different mechanisms. We have been running with DNS_PING for the last four years without any problem. It works really well in Kubernetes. For us, there is really no reason to change given our experiences so far. We have seen strange issues in Azure SQL (e.g. worker pool exhaustion causing selected queries to fail) so I would rather keep a JGroups discovery mechanism that was 100% reliable for us so far. |
|
@thomasdarimont No, JGroups config files don't support inheritance |
|
Keylcoak 26.1 is using the cache-stack Thank you for the collaboration! I'm now archiving this issue. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Current status of this discussion
We're trying to understand if JDBC_PING would be a solution, and what people struggle with in this are of Keycloak's configuration. Please provide your feedback below in the comments. We'll then update the the summary below.
Problem description
The current cache configuration describes multiple JGroups discovery and transport configurations. While for example the Keycloak Operator hides that configuration complexity from users by configuring a reliable setup, there are regularly questions on how to configure discovery in other setups.
The additional transport stacks for ec2, google and azure require additional libraries that aren't shipped with Keycloak. Users will need to install these in the providers folder to make use of them.
Solution idea
Common for all setups is that Keycloak stores its data in the database. So JDBC_PING seems to be a common denominator for all environments.
Current gaps
The current Keycloak documentation doesn't describe JDBC ping at all, and the necessary tables are missing in Keycloak's database schema.
The current configuration doesn't re-use Keycloak's database settings to set up JDBC ping, but requires a separate configuration. Editing Keycloak's Infinispan XML shouldn't be required for such a setup.
Some configurations used by the community:
Problems reported by users using JDBC Ping:
Additional difficulties to users are to bind JGroups to the correct host address, and setting the external address for a node:
Related work:
All reactions