Repository navigation
Import / Export and related operations in Admin Api V2 #41515
Replies: 2 comments 2 replies
|
On Tue, Jul 29, 2025 at 3:31 PM Steven Hawkins ***@***.***> wrote:
import / export
The import and export of a full realm(s), including users, groups, etc. to
a single file is problematic - the maximum REST payload for a single body
is currently limited to 10 MB and as touched on in #40798
<#40798> we have memory issues
because we parse the entire contents into memory. We may want to limit or
even discourage the usage of such a format.
I don't recall if we have committed in admin api v2 to supporting a realm
respresentation that contains all, or a configurable set, of sub-resources.
To keep things symetrical we should not allow sub-resources to appear in
POST, PUT, or PATCH. Pragmatically they could be fine in POST. However
appearing (or not) in a PUT, which can be interpretted as an overwrite of
the sub-resources will make the management of the top-level resource
difficult or start to muddle PUT and PATCH operations.
update realm (PUT .../realms/name)
Using some of a realm representation payload this currently provides an
operation that is similar to a json merge patch - that is most fields
appear to be modified only if they are specified in the payload body, and
applicable collections (attributes) are overwriten entirely. Also allows
for the realm name to be changed.
Given that this is not well documented and has non-standard semantics we
should not persue additional changes, such as #25326
<#25326>, and instead focus on
deprecating this behavior as soon as possible. This will be superceded by a
realm PUT or PATCH (json merge patch, or json patch) operation in the admin
api v2.
partial import / export
Partial import appears to be a compliment update realm and supports
modifying some collections under the realm, but not the top-level realm
fields.
The payload is similar to a realm representation, but supports only a
subset of fields, and has additional policy and ifResourceExists fields.
It also does no removals, only additions or overwrites - so it is also
similar to a json merge path, but does not do removals of collections.
Similar to update realm this is not well documented and has non-standard
semantics. We should probably not persue additional changes, such as
#41436 <#41436>, and instead
focus on deprecating this behavior as soon as possible.
The applicable set of resources in admin api v2 will be targetable by PUT
and PATCH (json merge patch, or json patch). We may also want to consider
adding end points for bulk operations - for example an endpoint that allows
for upserting a list of users.
Anything that needs to be acted upon in this manner needs to it's own
sub-resource - this at least includes users, groups, clients,
identityProvider, and identityProviderMappers.
UI support is TBD depending upon the exact endpoints supported. A zip
artifact could be considered is this will span multiple payloads and we
don't want to introduce a specialized coordinating file.
cc @pedroigor <https://github.com/pedroigor> @ssilvert
<https://github.com/ssilvert> @vmuzikar <https://github.com/vmuzikar>
@mabartos <https://github.com/mabartos> @Pepo48
<https://github.com/Pepo48> - are there any qualms with closing out the
long-standing issues logged against update realm and partial import /
export?
So there will be new functionality that makes partial import / export
obsolete?
I don't see any reason to close the old issues against current
functionality until the new feature is ready.
… —
Reply to this email directly, view it on GitHub
<#41515>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAD32QWTG6TKF75HFLBDY7L3K7DWRAVCNFSM6AAAAACCU3XTD6VHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZYGY2TENZTGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I would like for them to be obsolete in the current form yes. We should instead focus on what will be offered already as PUT / PATCH behavior and then build upon for performance and transactional considerations only if required.
Other than the recent request for better documentation, the issues have been open for 18 months - 3 years with only a modest amount of community interest. My intention was to at least explain why they will not be looked at further by the keycloak team. If you don't want them actually closed, I'll just add a comment to them pointing to additional commentary on #41436 that references the discussion here. |
|
Thank you for summarizing this. Once thing that we didn't touch upon: If you have a big JSON that you send to KC, you can then assume that it is processed in one transaction. When running multiple requests for multiple resources, things can get stuck in the middle (for example due to invalid data), and then a realm might be in an inconsistent state / no longer working. Given that we won't be able to process a full realm JSON of just any size, this is then a trade-off we might need to accept?! Eventually people might ask for a suggested ordering of updating resources. |
|
Yes, I need to elaborate more on the transactional trade off in the description - I only touched on it in passing in the previous comment #41515 (reply in thread) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
import / export
The import and export of a full realm(s), including users, groups, etc. to a single file is problematic - the maximum REST payload for a single body is currently limited to 10 MB and as touched on in #40798 both via the api and on the filesystem we have memory issues because we parse the entire contents into memory. We may want to limit or even discourage the usage of such formats - making exceptions only in cases where usability, performance, or transactionality would be compromised.
I don't recall if we have committed in admin api v2 to supporting a realm respresentation that contains all, or a configurable set, of sub-resources - see below for more on partial imports.
While pragmatically fine in a POST, to keep things symetrical we should not allow sub-resources to appear in POST, PUT, or PATCH. Appearing (or not) in a PUT, can be interpretted as an overwrite of the sub-resources, which will make the management of the top-level resource difficult or start to muddle PUT and PATCH operations.
update realm (PUT .../realms/name)
Using some of a realm representation payload this currently provides an operation that is similar to a json merge patch - that is most fields appear to be modified only if they are specified in the payload body, and applicable collections (attributes) are overwriten entirely. Also allows for the realm name to be changed.
Given that this is not well documented and has non-standard semantics we should not persue additional changes, such as #25326, and instead focus on deprecating this behavior as soon as possible. This will be superceded by a realm PUT or PATCH (json merge patch, or json patch) operation in the admin api v2.
partial import / export
Partial import appears to compliment update realm and supports modifying some collections under the realm, but not the top-level realm fields.
The payload is similar to a realm representation, but supports only a subset of fields, and has additional policy and ifResourceExists fields.
It also does no removals, only additions or overwrites - so it is also similar to a json merge path, but does not do removals of collections.
Similar to update realm this is not well documented and has non-standard semantics. We should probably not persue additional changes, such as #41436, and instead focus on deprecating this behavior as soon as possible.
The applicable set of resources in admin api v2 will be targetable by PUT and PATCH (initially json merge patch, or json patch). However working with collections from a parent resource PATCH in the prototype is currently difficult. With both json patch and json merge patch there are usability issues wrt to array / collection handling. With json merge patch you must completely specify the array - selectively adding or removing items is not possible. With json patch there is no understanding of identity in a collection beyond an index, so there is no better approach to add / replace / remove than to require stable sorting w/ opputunistic locking and to compute the necessary indexes on the client side when building the patch.
Primarily for performance reasons we may want to consider adding end points for bulk operations - for example a PATCH type / common endpoint that allows for upserting a list of resources. Anything that needs to be acted upon in this manner needs to its own sub-resource - this at least includes users, groups, clients, identityProvider, and identityProviderMappers.
Obviously spanning a partial import over multiple requests runs the risk of an incomplete application. While partial import seems to be for users who are opting for a UI driven (non-declaratively managed) flow to updating resources and propogating that to another environment, it's not yet clear how much we'll expect operators to sync in way that is transactional - or if we can safely assume an ability to reach the desired runtime state via a well-known ordering of updates with necessary ability to retry.
Alternatively we can more generally consider an additional patch format - similar to the Kubernetes strategic /enhanced json merge patch. While that format is non-standard, from a parent resource it does allow for merging arrays / collection based upon merge keys - but things get complicated if there are collections that do not have keys.
UI support is TBD depending upon the exact endpoints supported. A zip artifact could be considered is this will span multiple payloads and we don't want to introduce a specialized coordinating file.
cc @pedroigor @ssilvert @vmuzikar @mabartos @Pepo48 - are there any qualms with closing out the long-standing issues logged against update realm and partial import / export?
All reactions