Repository navigation
Replies: 10 comments 15 replies
|
Another upside of the profile-based approach is that we could even pre-load all the storage factories, instantiating one for each profile and then computing the |
|
Thanks for proposing this change. Once mapStorage gets traction, I see that there will be multiple instances of a variant like JPA or LDAP most of the time. My comments on this: I like the solution where a configuration is stated centrally in mapStorage and is then referenced in other places as it prevents duplicating the configuration by an ID. To name this ID a profile is fine for me, or it could be just named "id". One thing I didn't see in the examples you provided: How would I configure a store per realm? I can image that users want to map a realm to a database, or in LDAP scenarios would map a realm to a specific LDAP directory or subtree. While this question is quite important for me from an LDAP point of view, feel free to explain if this is not related to this question. |
|
Thanks for the comments, @ahus1! For the moment I will stick to having a default profile just to avoid enforcing users to explicitly configure a profile in cases where they only have one possible config at the provider root config, but this is of course subject to changes. The problem with the configuration of the providers as components in a realm is one that is still not solved and is still a work in progress. There are a few points that need to be addressed on that front, so as of now we simply don't know how exactly that will look like. |
|
The realm configuration will be done via declaring a component for map storage / map provider (to be decided) inside that realm, and then requesting a map storage using a component lookup prior to using a system-wide one. |
|
I've crafted a draft PR with the changes proposed here - #10429. Testing locally I was able to have some models (role) in one DB, and others (client, clientScope) in a different DB. It needs some tidying up but it gives an idea of what would be needed to make it work. |
|
Thank you @sguilhen for the proposal. I very much like the idea of ability to declare profiles within a system configuration. I would like to stress the scope of this proposal: The proposal is limited for use in system configuration; configuration profile declaration is not suitable for realm components (since component configuration is not a tree but a map of configuration options, and profile declaration effectively builds a tree structure). There are few comments / clarifications needed, and I will add those as separate answers to this discussion so that they can be discussed independently. |
Definitely, thanks for stressing this. The proposal's scope is indeed limited to the system configuration. |
Default configurationThe default profile configuration should be indistinguishable from the same configuration of provider when there is no profile defined, i.e. the following two configs should be equivalent for achieving "postgres-dev" configuration: "mapStorage": {
...
"jpa-map-storage": {
"driver": "XYZ",
"datasource": "FOO"
# configuration of the postgres-dev database.
} },and "mapStorage": {
...
"jpa-map-storage": {
"driver": "XYZ",
"datasource": "FOO",
// configuration same as of the postgres-dev profile.
"profile": {
"postgres-dev": {
// "datasource": "FOO" <-- this has been here originally and should not have been, moved above
// this profile configuration is intentionally left empty
},
"postgres-prod": {
"datasource": "BAR"
// configuration of the postgres prod database.
} } } },From that it seems to me that there is no need for WDYT? |
|
I've updated the original answer.
Yes, exactly. In other words, the profile config inherits the root config and overrides / adds config options of that. |
|
Ok, so you're proposing that the "empty" profile (i.e. one that gets all all the config options from the root config) is to be considered the default one? |
|
If by "empty" profile you mean no-profile-referenced-or-declared, then yes. The default profile is in this proposal should be just the same as the system-wide configuration. |
I think I need more clarification here, not sure what you meant by "no-profile-referenced-or-declared". Currently, the algorithm to select the default profile would be as follows: Now, when we have to fetch the scope to init the factory with the right configuration options: If I understand right, you want to get rid of (1) above and not have an explicit config attribute for the default profile, but rather try to locate that based on some other rule, like an empty profile. So the algo would be something like I just want to understand what the algorithm to determine the profile when none is declared in one of the areas that use the map storage should look like. In other words, if the area specifies a profile it is straightforward to locate that profile in the root config and use its config. But when the profile is not mentioned, we need to figure out what should be the default profile. I imagine you would like that to be resolved to |
|
There is no "empty" profile per se. The "default profile" is driven by the default configuration done in the same way as today for providers when we lack any profile support: "mapStorage": {
...
"jpa-map-storage": {
"driver": "XYZ",
"datasource": "FOO"
# configuration of the postgres-dev database.
} },Any profile defined in a provider config would inherit all the values specified in the main provider configuration, and would be able to override those. In other words, the profile
|
Distinguishing when to cope with configuration as reference and when as a new componentHow would the code mutually distinguish the following different use cases of the configuration?
|
|
I'm addressing this in context of the discussion bellow, in which the config for a specific storage factory has to happen in the scope of that factory's name: 1- In this case, code builds a 2- The config would actually be (according to what I've shown in the discussion bellow) So the code here checks if the config scope for the storage factory is not empty in the area configuration. In this case, it is not, it is defining the datasource, so we treat it as a new config that will end up having a 3- Again based on the discussion bellow, the |
Compatibility with component configurationRe: Area-specific storage provider configuration
provider=jpa-map-storage
provider-profile=postgres-prod
jpa-map-storage.option1=value1
jpa-map-storage.option2=value2
...This means that for the respective provider factory, rather than the That is also the reason why the storage component configuration is flattened in map storage and behaves as if that was like this: "role": {
"provider": "map",
"map": {
"storage": {
"provider": "jpa-map-storage",
"provider-profile": "postgres-prod",
"option1": "value1"
// alternative configuration
} } } }
How exactly would the algorithm for generation of this |
|
That doesn't sound right to me. Indeed, the storage config is flattened in the map storage, but when we try to fetch the storage provider factory the code in For example, this is how the sguilhen@6b6abe3#diff-0396a8b7dd0fd795fe8e23759f0b301cd2496fcc8c558d61c5766352372f227aR72 So it does indeed adjust the scope and the property
I can't really get to the Thinking about it now, we could simply use the fully-qualified class name of the model as the key, so we would have a |
This is right, I want to stress that this means that any provider factory that offers such an inline configuration must also provide a contextual identifier ( |
|
I like the suggestion to use the model type mapped to model name. |
Profile referencingOne idea to consider: Would it be reasonable to reference the profile by standard path convention, i.e. "role": {
"provider": "map",
"map": {
"storage": {
"provider": "jpa-map-storage",
"provider-profile": "postgres-prod"
} } } }to use rather: "role": {
"provider": "map",
"map": {
"storage": {
"provider": "jpa-map-storage/postgres-prod",
} } } }? |
|
+1 to that. Actually, this was my first idea but I couldn't find any other code that uses this convention so I thought it would be preferrable to use a separate config attribute, but I'm totally in favor of this approach. |
|
@stianst @pedroigor @mposolda I think it would be good if you guys could weigh in on this |
|
@hmlnarik @sguilhen The proposal about using profiles sounds like a good idea to solve the original problem with the From a distribution perspective, I can think of two possible configuration approaches:
I'm not 100% sure about the #1 above because the storage configuration is dynamic in the sense that users can define their own providers and name profiles accordingly, making it harder to keep it aligned with other options in terms of help message and auto-completion. Another problem with this approach is that we don't have yet discussed the outline for extensions. The #2 approach is what I think is simpler, with a good UX, and flexible enough to support all the configuration aspects. By following this approach we could have a configuration namespace For instance, if you want a single database for all areas we could still leverage the existing options to replace values in the default configuration. It should also be possible to dynamically create data sources based on the storage configuration if users define different profiles for the JPA storage. This one, in particular, should make it easier when you need to use your own storage implementation to integrate with a legacy identity store for users, roles, and groups. The same goes for a top-level LDAP configuration, to which we could provide specific options for configuring LDAP. I'm not 100% sure yet about other potential use cases and combinations that make sense to provide out of the box via de the default configuration. I'm also not 100% about how far people are going to use the top-level configuration to configure the store rather than configuring the storage on a per-realm basis. |
Uh oh!
There was an error while loading. Please reload this page.
This discussion aims to address the issue repored in #10226
The problem
To summarize the issue, we can have areas, such as
role,group,user, etc in the system configuration that use the sameMapStorageProviderimplementation - e.g.jpa-map-storage, but with different configurations. Currently the storage providers can define a root configuration in themapStorageSPI, and areas that need to use a different storage configuration have to specify the storage configuration as part of their own config. Something like this:However, all the storage factories for the different areas are assigned the same
componentId, whose format ismapStorage-f:ROOT:storage_nameand the first storage factory that is instantiated is cached under thecomponentIdkey, which in the end makes all areas use the same configuration. So instead of havingroleandclientusing different DBs they both end up in the same DB - the one specified in the configuration of the area that is requested first.So the issue is that the
componentIddoes not consider the configuration of the storage provider, and we need to think about ways to fix this.Possible Solutions
Include the configuration hashcode in the
componentIdTwo configs for a storage provider are equal if they yield the same values for the set of supported configuration attributes. This means that we could add a
getSupportedAttributesmethod to the factories (perhaps to theAmphibianProviderFactory) that returns the set of attributes the particular implementation supports, and then we could fetch the attribute values using theScopethat is constructed for the particular area storage and store all key/value pair in aHashMap. The final map's hashcode can then be appended to thecomponentId.With this approach, two different areas that have the same config end up sharing the same factory. However, this means that every time a storage is requested we will have to compute this hashcode to determine if we can reuse a previously instantiated factory. In addition, if two or more areas need to share the same alternative configuration, we still need to replicate the config for each area.
Configure the
componentIdmanually for each areaIn this case, one could manually set the
componentIdfor the areas, and those with the samecomponentIdend up sharing the same storage factory. Something like this:This would work but it we still need to replicate the config for all areas that want to share an alternative configuration because we don't know which area will be requested first (and thus have its storage factory initialized and cached for the others to use).
Proposal: add profiles for the storage factory configurations and have areas reference specific profiles
The idea here is to expand the configuration of the storage providers to support multiple profiles, and have each area reference one of the profiles in their own configuration. The
componentIdwould be computed asmapStorage:storage_provider:profile. For example, if we have a profile namedpostgres-devfor thejpa-map-storageprovider, thecomponentIdwould be computed asmapStorage:jpa-map-storage:postgres-dev.The configuration of a root storage factory would look like this (NOTE: in the examples we're using the
jpa-map-storagebut the idea can be applied to any storage provider):default-profileis optional, in the absence of this attribute we could look for a profile nameddefaultand if there is no such profile we could use the factory's root configuration as the scope that is fed toinit. For example,Would be equivalent of a configuration that has a single profile named
default. i.e.Now each individual area would be able to specify the profile in its configuration and reuse the same config. For example,
Would use the
postgres-devprofile of thejpa-map-storageprovider for the roles. In the absence of theprovider-profile, the storage factory's default profile would be used (and for that we would check what default was configured in the root provider configuration). That isWould end up using the default profile for the
jpa-map-storage. If our config looks like the first example in this section, we would end up using thepostgres-devprofile.Every area using the same storage provider and profile would end up having the same
componentIdand thus share the same storage provider factory. In addition, this no longer requires copying and pasting the same config for every area that needs to use a different database configuration. Finally, this solution makes it easier to switch between dev and production profiles by having all areas use the default profile and then change the default to a different profile in the root configuration.Area-specific storage provider configuration
It would still be possible to have a config specific to an area. One way to do it would be to define a separate profile that is used only by that single area. A different approach would be to have the storage config specified in the area itself, much like what we do today:
In this case, we check if there is an inline configuration for the storage and if we have one, we can generate a
componentIdjust for the specific area. Something likemapStorage:jpa-map-storage:role-inline. As a result it would have its own storage factory.The inline config would still fallback to the root configuration as we have today. In other words, if an attribute can't be found in the inline config it checks the root config. The only difference is that in this case we would fallback to one of the profiles. If the
provider-profileisn't specified, we fallback to the default profile as determined by the root configuration. If theprovider-profileis specified, we fallback to the specific profile.So, the example above would fallback to the
postgres-devprofile as that is the default profile in the root configuration. So theroleconfig can, for example, have just a different connection URI that uses a different DB, and get all other attributes, such as user, password, etc, from thepostgres-devprofile.The following inline config
Would use the inline config and fallback to the
postgres-prodprofile.Of course there are a few implementation-specific details that would need to be sorted out to make all this work, but it is flexible in the sense that it expands the storage factories configuration and it allows for the areas to use different storage configurations simply by referencing them instead of copying and pasting the config into each area. Plus, it can still provide a way for an area to have a very specific config by keeping the inline configuration that falls back into one of the profiles.
Thoughts, comments and other considerations are highly welcome!
All reactions