Challenges with custom Federated User Storage Provider implementation #20179

cesdperez · 2023-05-05T09:13:28Z

cesdperez
May 5, 2023

Hi everyone,

I'm working on implementing a custom user storage provider in Keycloak to store users' PII in a separate service and database, while keeping all non-PII data in Keycloak. I've referred to the official documentation and am using AbstractUserAdapterFederatedStorage along with implementing UserStorageProvider, UserRegistrationProvider, UserLookupProvider, and UserQueryProvider.

However, I'm facing a couple of challenges:

Atomicity: Keycloak seems to create users in multiple phases, making it difficult to insert/update users in my external storage atomically and handle errors properly. I could use separate operations but then I would need to implement virtual transactions with manual rollbacks, which isn't ideal.
Hidden update logic: Future user updates appear to happen behind the scenes, and I don't have control over them in my provider. This is done through AbstractUserAdapterFederatedStorage#addAttribute and #removeAttribute methods, which makes error handling and atomicity more complex.

I've reviewed several custom user storage provider examples, but they either:
a. Don't implement UserRegistrationProvider, so users aren't created in external storage.
b. Handle inserts and updates with separate operations without proper error handling and rollbacks.
c. Delaying user creation until close() in the provider (e.g., this example by @dasniko). This seems like a hack, would need manual transactions and rollbacks, and wouldn't handle updates either. (@dasniko 's examples were great to get me started tho! kudos 🙏🏼)

I've also found several open questions on this, with no answer:

Am I missing something? Is there a better way to handle this? It seems like a common use case when integrating with external storage in a production environment. Any help or suggestions would be greatly appreciated!

If I can make this work properly, I'd be happy to contribute an example implementation.

Thank you!

dasniko · 2023-05-06T08:34:25Z

dasniko
May 6, 2023

Hi @cesdperez
thanks for the kudos! 🙏
I'm happy that my examples (and probably my videos?) are of good help for you.

My solution might appear to be "hacky" in the first place, but, it's the only way to deal with it if you have to call something like an external API through HTTP/REST or similar.

IMPORTANT:
The key issue here is, that you have to implement not only your ...UserStorageProvider properly, but also your ...UserAdapter (like my FlintstoneUserAdapter, see e.g. the setAttribute method(s)).

If you implement everything properly, it works seamlessly as expected.

The User Storage SPI was initially designed for use with JPA, so that Keycloak has a direct connection to a database where the users data is being stored. That way, and you can see this also in this quickstart, the storage provider just makes use of the database transaction management and no separate "create" or "update" phase is necessary.

Side note:

My example is not yet completely finished, as I'm expecting some runtime trouble when creating new users. Currently, I'm only sending the create/update request of users data in the close() method. But for a new registered user, you also have to update the password, which happens in updateCredentials() method. This method is executed (of course) before close().
So, for new users, my example currently makes the updateCredential request before the createUser request.

Solution would be to create the user in the addUser() method, with username only and updating the user again in the close() method.

2 replies

cesdperez May 8, 2023
Author

Thank you for your response, @dasniko. Your examples and videos have been quite beneficial during my brief experience with Keycloak, and I appreciate your help!

The User Storage SPI was initially designed for use with JPA, so that Keycloak has a direct connection to a database where the users data is being stored. That way, and you can see this also in this quickstart, the storage provider just makes use of the database transaction management and no separate "create" or "update" phase is necessary.

Somehow I hadn't come across that example before. It helps, but it's not an ideal solution for my situation.

My case has some particular aspects:

a. I need to store data in an external service via an API (gRPC, to be specific, but that hasn't posed any issues).
b. User data is partially stored in this external service and partially stored in Keycloak's database. For example, the firstName and lastName are in the external service, while other information (e.g., passwords) should remain in the Keycloak database.
c. For these reasons, ensuring consistency is crucial for me. I want to avoid scenarios where a user exists in the external service but not in Keycloak's database or vice versa. That's why I consider atomicity so important.

As you mention, it appears that my only option is to create the user in phases:

The user is created in my external service (within addUser).
Keycloak updates all user attributes (this seems to occur in DefaultUserProfileProvider, which will call my UserAdapter#setAttribute(...)). Some of these attributes will be stored in Keycloak's database (default behavior), while others will be held in memory (by my custom logic in setAttribute) until
I update the user in my external storage with the missing attributes, potentially during my UserStorageProvider#close.

But there are still issues:

Because of c, I'll have to manually manage errors and rollbacks. If step 2 or 3 fail, I'll need to manually rollback the previous steps (and that can fail too, etc).
I'm still unsure I'll be able to handle user updates properly. There's no updateUser in the UserStorageProvider interfaces, and I'm unsure the #close() approach will work there. I will give it a try this week. The jpa quickstart example doesn't seem to be doing anything regarding updates, unless I'm missing something. I only see a call to em.persist on addUser and no calls to em.merge or anything else relevant anywhere else.

dasniko May 8, 2023

Because of c, I'll have to manually manage errors and rollbacks.

Welcome to the world of distributed systems! ;)

I'm still unsure I'll be able to handle user updates properly. There's no updateUser in the UserStorageProvider interfaces, and I'm unsure the #close() approach will work there. I will give it a try this week. The jpa quickstart example doesn't seem to be doing anything regarding updates, unless I'm missing something. I only see a call to em.persist on addUser and no calls to em.merge or anything else relevant anywhere else.

That's what I wrote in my first answer about "the storage provider just makes use of the database transaction management" - this is simply handled by the database transaction, as connected entities are automatically being merged on transaction commit.

javac9 · 2023-05-05T18:31:12Z

javac9
May 5, 2023

Hi @cesdperez ,

I still don't have a clean solution for this problem but rather a half-baked one. It's about implementing own UserProfileProvider just like we're doing with UserStorageProvider.

Once that's registered by Keycloak, it will call the overridden create() methods in that class. From within one of them, I'm calling my REST API to create my user in my backend.

The problem is - even though my user gets created in backend, from Keycloak's perspective, the user creation process was unsuccessful and therefore, if the new user was, for example, created by filling the "New user" form in Keycloak, the created user won't be shown and error message will be displayed that user creation failed. That is expected since indeed, Keycloak itself didn't create any user.

So, I'm currently looking for at least a workaround which would allow me to handle gracefully the errors I'm getting after my user got created externally - I'd like to somehow let Keycloak know that my user was created externally and provide it with the new user's id so Keycloak can show it after creation is done.

1 reply

cesdperez May 8, 2023
Author

Thanks for your reply.

I had seen that DefaultUserProfileProvider is the guy calling all the #addAttribute and #removeAttribute from my UserAdapter. I also considered creating a custom UserProfileProvider but that seems even more hacky and also couldn't find much documentation on it (if you found something besides the javadocs, feel free to share).

The approach used by Dasniko in this example is not ideal, but still better I think. See the response by Dasniko below.

Navigation Menu

Repository navigation

Challenges with custom Federated User Storage Provider implementation #20179

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Challenges with custom Federated User Storage Provider implementation #20179

Uh oh!

cesdperez May 5, 2023

Replies: 2 comments · 3 replies

Uh oh!

dasniko May 6, 2023

Uh oh!

Uh oh!

cesdperez May 8, 2023 Author

Uh oh!

dasniko May 8, 2023

Uh oh!

javac9 May 5, 2023

Uh oh!

cesdperez May 8, 2023 Author

cesdperez
May 5, 2023

Replies: 2 comments 3 replies

dasniko
May 6, 2023

cesdperez May 8, 2023
Author

javac9
May 5, 2023

cesdperez May 8, 2023
Author