Migration on Keycloak 17 (keycloak x) using file (it was deprecated?) #10350

SrMouraSilva · 2022-02-21T10:15:12Z

SrMouraSilva
Feb 21, 2022

Hello,

I noticed that apparently are some missing docs related to migration.

In the Wildfly versions, it's possible to load an exported realm using:

# Enable upload of realm-export.json
-Dkeycloak.profile.feature.upload_scripts=enabled

# Load the migration file
-Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/tmp/realm-export.json
-Dkeycloak.migration.strategy=IGNORE_EXISTING

I didn't found its documentation at the website, but I found at the Github.

I know that the features enabling changed between Wildfly and Quarkus versions. At the Server Features page, I found that "upload-scripts" is now deprecated. Also, I noticed that apparently isn't possible to upload an exported realm file using the admin2 theme, at least not in the same way as the original admin: If I send a file, it apparently only loads the Realm configuration, instead of also loading the users, groups, etc.

Note: The previous migration configuration apparently works yet, but I don't know if it really load all the expected configuration

Is the migration no more supported? It is yet supported, but will be fully deprecated? There are plans to a substitute?

Answered by pedroigor

Feb 21, 2022

@SrMouraSilva Did you try using the import command available from the CLI?

After updating your keycloak.conf with the settings you need (e.g.: database vendor/credentials, features) and running a build, you should be able to import realms by:

./kc.sh import --dir /tmp/kc-export

You should also be able to import from a file by using the --file option.

View full answer

pedroigor · 2022-02-21T10:44:21Z

pedroigor
Feb 21, 2022
Collaborator

@SrMouraSilva Did you try using the import command available from the CLI?

After updating your keycloak.conf with the settings you need (e.g.: database vendor/credentials, features) and running a build, you should be able to import realms by:

./kc.sh import --dir /tmp/kc-export

You should also be able to import from a file by using the --file option.

6 replies

Dill-Dall Feb 22, 2022

I did this, but got the script upload is disabled error. How to set it as enabled in 17?

2022-02-22 11:01:55,608 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (import_export) mode 2022-02-22 11:01:55,608 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Error details:: java.lang.RuntimeException: Script upload is disabled at org.keycloak.authorization.policy.provider.js.JSPolicyProviderFactory.updatePolicy(JSPolicyProviderFactory.java:125) at org.keycloak.authorization.policy.provider.js.JSPolicyProviderFactory.onImport(JSPolicyProviderFactory.java:70) at org.keycloak.models.utils.RepresentationToModel.toModel(RepresentationToModel.java:2466) at org.keycloak.authorization.AuthorizationProvider$3.create(AuthorizationProvider.java:351) at org.keycloak.models.utils.RepresentationToModel.importPolicies(RepresentationToModel.java:2392) at org.keycloak.models.utils.RepresentationToModel.toModel(RepresentationToModel.java:2340) at org.keycloak.models.utils.RepresentationToModel.importAuthorizationSettings(RepresentationToModel.java:2287) at org.keycloak.services.managers.RealmManager.setupClientServiceAccountsAndAuthorizationOnImport(RealmManager.java:761) at org.keycloak.exportimport.dir.DirImportProvider$7.runExportImportTask(DirImportProvider.java:174) at org.keycloak.exportimport.util.ExportImportSessionTask.run(ExportImportSessionTask.java:35) at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:239) at org.keycloak.exportimport.dir.DirImportProvider.importRealm(DirImportProvider.java:169) at org.keycloak.exportimport.dir.DirImportProvider.importModel(DirImportProvider.java:76) at org.keycloak.exportimport.ExportImportManager.runImport(ExportImportManager.java:87) at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.createMasterRealm(QuarkusJpaConnectionProviderFactory.java:280) at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.initSchema(QuarkusJpaConnectionProviderFactory.java:226) at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:239) at org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory.postInit(QuarkusJpaConnectionProviderFactory.java:158) at org.keycloak.quarkus.runtime.integration.QuarkusKeycloakSessionFactory.init(QuarkusKeycloakSessionFactory.java:96) at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.initializeKeycloakSessionFactory(QuarkusKeycloakApplication.java:75) at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.startup(QuarkusKeycloakApplication.java:56) at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver.onStartupEvent(QuarkusLifecycleObserver.java:37) at org.keycloak.quarkus.runtime.integration.QuarkusLifecycleObserver_Observer_onStartupEvent_8292a286c9778cb4a07bc3314fcbfb61e7fe81a0.notify(Unknown Source) at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:320) at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:302) at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:73) at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:128) at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:97) at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source) at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source) at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source) at io.quarkus.runtime.Application.start(Application.java:101) at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:104) at io.quarkus.runtime.Quarkus.run(Quarkus.java:67) at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:84) at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:34) at org.keycloak.quarkus.runtime.cli.command.AbstractExportImportCommand.run(AbstractExportImportCommand.java:71) at picocli.CommandLine.executeUserObject(CommandLine.java:1939) at picocli.CommandLine.access$1300(CommandLine.java:145) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358) at picocli.CommandLine$RunLast.handle(CommandLine.java:2352) at picocli.CommandLine$RunLast.handle(CommandLine.java:2314) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) at picocli.CommandLine$RunLast.execute(CommandLine.java:2316) at picocli.CommandLine.execute(CommandLine.java:2078) at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:84) at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:79) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60) at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)

SrMouraSilva Feb 22, 2022
Author

@ThomasDahll, I think that you need to enable it before to try to run

[...] build --features=upload-scripts

LazaroPinheiro Mar 10, 2022

@SrMouraSilva upload_scripts feature is deprecated to prevents that potential harm to the system in case malicious scripts are accidentally executed!
Ability to upload scripts through the Admin Console is deprecated and will be removed in a future version of Keycloak.

khassad May 30, 2022

Sorry but this does not answer the problem of @ThomasDahll , which I also stumble upon :
We are talking about a CLI usage from the administrator, nothing to do with the Admin Console.

So yes there is a really bad behavior here where whatever way I use to import it fails due to this upload_scripts issue.
I tried start-dev --import-realm at server startup (container version), and kc.sh import --dir= ,
both fails because they complain about the fact upload-scripts is disabled, which is stupid since we are in a CLI context here.

Nerzal Aug 4, 2022

@khassad i can confirm this behavior with the latest image (keycloak 19).

pedroigor · 2022-02-21T10:44:47Z

pedroigor
Feb 21, 2022
Collaborator

But yeah, we should probably write a guide about this too.

1 reply

pedroigor Feb 21, 2022
Collaborator

@SrMouraSilva I've created #10355 to fix the descriptions.

khassad · 2022-05-30T13:25:04Z

khassad
May 30, 2022

So far we are badly stuck here with no working way to import our realms to bootstrap a production server in the KC 18 release 😢
Or am I missing something ?

5 replies

khassad May 30, 2022

Let me contribute with my findings after lots of hair pulling :

The container --import-realm option does not work for directory imports, it needs a single file per realm, which does not work when you have a decent number of users.
So a directory import only possibility is using the kc.sh import --dir=... command
You first need to clean up the realm file from the default policies created in previous KeyCloak versions which are of javascript type, those cannot be imported in KC18, it bails out with the cryptic Script upload is disabled. You also need to remove any other policy or feature relying on js or any other capability depending on the removed upload-scripts feature.
I still get some error but overall the import looks like it worked :

 WARN  [com.arjuna.ats.arjuna] (main) ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a000264:aa67:6294d06a:0
 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff0a000264:aa67:6294d06a:0
 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ARJUNA016102: The transaction is not active! Uid is 0:ffff0a000264:aa67:6294d06a:0

This works for KC18

pedroigor May 31, 2022
Collaborator

Thanks @SrMouraSilva, we appreciate the feedback as it should help others.

We did not change the behavior of "importing at startup" and it works the same way as in legacy.

However, both import and export actions/commands span a short-lived instance to do the work. That is by design and to create a more clear separation around doing imports/exports and actually starting the server (mainly in production). You should expect a better experience for your use case with the introduction of the new store too. @hmlnarik Can give more details about how it will work.

As per the upload-scripts feature, I know it sucks. But it has been deprecated for quite some time and now we finally managed to remove it.

W.r.t. to the error, I don't think that is a good log entry, Are you able to share a JSON that I can use to reproduce here?

ghost Jul 18, 2022

Hi @pedroigor
I am experiencing exactly the same issues outlined above when attempting to migrate realms + users from KC v9.0 to KCv18+

I can confirm that export does NOT work for any strategy bar --users strategy: different_files and same_file. Further, only different_files actually succeeds in exporting all users after some housekeeping. The same_file export breaks part way through still.

I believe my issue and possibly issues that other folk are experiencing is related to an earlier Keycloak release: v9.0

Scenario:

I have 10+ realms. Only 2 realms break the export.
Both realms have duplicate parent groups as per the duplicate top level group bug introduced in Keycloak v9.0

I remove the dupes. This resolves for Keycloak application version upgrades as the new version is able to update the source database; however, tech debt, associated with this bug can still persist in the form of Realm users who have been assigned 1 or more groups where, in fact, those groups were dupe groups.

I have strong evidence that correlates users with broken group referential integrity and exports that fail.

Taking one realm as an example:

When exporting via different_files the export broke. The final user json's last user entry happens to be the last user before a problem user who has been identified as having references to 1+ dupe groups that have now been removed to accommodate the update.
For this realm there is only one user that has these broken references.
I therefore remove this user via admin console and re-tested using the same strategy.
This time, all users (bar the deleted user) are exported; however, not without issue still:
E.G.

2022-07-18 11:15:07,705 WARN  [org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorTrackingImpl] (Transaction Reaper Worker 0) HHH000451: Transaction afterCompletion called by a background thread; delaying afterCompletion processing until the original thread can handle it. [status=4]
2022-07-18 11:15:07,706 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a000008:932b:62d53f8f:0

As khassad pointed out earlier - this doesn't stop the export tho; however, like you, I don't like the look of the error.

Using same_file will not work period until the 'rogue' user is removed; however, it is more brittle and still fails part way through.

In your response to khassad you stated:
However, both import and export actions/commands span a short-lived instance to do the work.
Can you elaborate on this please. What do you actually mean by span a short-lived instance to do the work

Can you also point me in the direction of comphrehensive online documentation that describes this new store reference you also mention in your response. Having reviewed the release notes - I can see reference to a new store (albeit not in context to MySQL) but there is not context.

The upshot of multiple tests is that I am forced to use strategy different_files for the realm exports and am about to move on to importing to see if I can get that to work.

Any help here would be greatly appreciated.

Thanks.

ghost Jul 18, 2022

@pedroigor Some further info in case helpful:

KC log output part way through export:

2022-07-18 16:28:15,702 INFO  [org.keycloak.exportimport.dir.DirExportProvider] (main) Users 4300-4349 exported
2022-07-18 16:28:19,766 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check processing TX 0:ffff0a000008:8151:62d588f7:0 in state  RUN
2022-07-18 16:28:19,776 WARN  [org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorTrackingImpl] (Transaction Reaper Worker 0) HHH000451: Transaction afterCompletion called by a background thread; delaying afterCompletion processing until the original thread can handle it. [status=4]
2022-07-18 16:28:19,777 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff0a000008:8151:62d588f7:0
2022-07-18 16:28:22,345 INFO  [org.keycloak.exportimport.dir.DirExportProvider] (main) Users 4350-4399 exported

KC log output once test realm export complete:

2022-07-18 16:38:25,537 INFO  [org.keycloak.exportimport.dir.DirExportProvider] (main) Users 7550-7569 exported
2022-07-18 16:38:25,587 INFO  [org.keycloak.services] (main) KC-SERVICES0035: Export finished successfully
2022-07-18 16:38:25,588 WARN  [com.arjuna.ats.arjuna] (main) ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a000008:8151:62d588f7:0
2022-07-18 16:38:25,807 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (import_export) mode
2022-07-18 16:38:25,808 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: javax.transaction.RollbackException: ARJUNA016102: The transaction is not active! Uid is 0:ffff0a000008:8151:62d588f7:0
2022-07-18 16:38:25,808 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ARJUNA016102: The transaction is not active! Uid is 0:ffff0a000008:8151:62d588f7:0
2022-07-18 16:38:25,808 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

Nerzal Aug 4, 2022

You first need to clean up the realm file from the default policies created in previous KeyCloak versions which are of javascript type, those cannot be imported in KC18, it bails out with the cryptic Script upload is disabled. You also need to remove any other policy or feature relying on js or any other capability depending on the removed upload-scripts feature.

This fixed it for me. Thank u so much.

Migration on Keycloak 17 (keycloak x) using file (it was deprecated?) #10350

Uh oh!

Uh oh!

Replies: 3 comments · 12 replies

Uh oh!

pedroigor Feb 21, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

SrMouraSilva Feb 22, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pedroigor Feb 21, 2022 Collaborator

Uh oh!

pedroigor Feb 21, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pedroigor May 31, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 3 comments 12 replies

pedroigor
Feb 21, 2022
Collaborator

SrMouraSilva Feb 22, 2022
Author

pedroigor
Feb 21, 2022
Collaborator

pedroigor Feb 21, 2022
Collaborator

pedroigor May 31, 2022
Collaborator