Preview - Keycloak Test Framework #34951

lhanusov · 2024-11-14T10:47:27Z

lhanusov
Nov 14, 2024
Collaborator

Dear Keycloak community,

We would like to ask you for your feedback on Keycloak Test Framework, which is already present as a preview in the main branch and released in nightly builds. The features list is not fully implemented.

Background:
The Keycloak team has started an initiative called "Test framework" to replace the current integration test suite solution.

Integration test suite drawbacks:

Arquillian framework is no longer actively supported
complicated test execution
learning curve for beginners is quite steep, since there is a complex structure of dependencies and configurations
missing support for custom extension tests or configs
overall it creates a bad user experience, when writing new tests

The goal of this initiative is to replace the old solution with a new one, based on the test framework, and provide a more user-friendly experience when adding new tests.

It is also one of the first steps towards improving the Keycloak Developer experience by providing a lightweight and Keycloak-oriented testing framework that should make it a lot easier to write and test extensions. For this reason, your feedback is important to help us to provide the best experience possible for our developers.

5 steps on how to start:

Download keycloak repository (main)
Build keycloak-parent project
Read the user guide
Check examples on how to use the framework
Play

If you are an extension developer, please take a look at the custom provider deployment: provider example -> pom.xml test dependency -> test example.

This discussion targets Keycloak developers and also Keycloak extension developers, because the framework will enable the feature to install custom providers.

Thank you for your comments!

Red Hat Keycloak team

michael-clay-mox · 2025-01-09T09:21:05Z

michael-clay-mox
Jan 9, 2025

Some feedback and one question from initial testing and working with the new Keycloak test framework i.e. @KeycloakIntegrationTest

Question: how is the EmbeddedKeycloakServer supposed to be used?

DistributionKeycloakServer.DEBUG (hardcoded set to false) should be configurable. Currently this behavior can only be overridden by copying existing classes and providing them as custom TestFrameworkExtension
I suggest to use the copy options REPLACE + COPY_ATTRIBUTES so that the creation and modification date is preserved and only copy the file if it has changed.

Reason: Feedback cycle performance when running tests because providers + dependencies need to be bundled as fat/shadowed jar and therefore there size must be considered.

private static void copyProvider(Path distPath, String groupId, String artifactId) {
        try {
            Files.copy(Maven.resolveArtifact(groupId, artifactId), distPath.resolve("providers").resolve(artifactId + ".jar"));
        } catch (IOException cause) {
            throw new RuntimeException("Failed to copy JAR file to 'providers' directory", cause);
        }
    }

I suggest to change the default behaviour of the RawKeycloakDistribution#prepareDistribution in the following ways
- allow the user to overwrite the destination of the final dist zip and use java.io.tmpdir only as fallback.
- make the recreate switch configurable
- only copy the dist jar if it does not already exists (this is currently not possible because of the inited switch )
- Reasons:
-- again feedback cycle performance
-- also i would use either the gradle build/tmp directory or the target/ folder for maven as destination for the dist zip, so i can control the recreation by means of the build tool lifecycle task i.e. mvnw/gradlew clean

private Path prepareDistribution() {
        try {
            Path distRootPath = Paths.get(System.getProperty("java.io.tmpdir")).resolve("kc-tests");
            distRootPath.toFile().mkdirs();

            File distFile = new File("../../dist/" + File.separator + "target" + File.separator + "keycloak-" + Version.VERSION + ".zip");
            String distDirName;

            if (distFile.exists()) {
                distDirName = distFile.getName();
            } else {
                distFile = Maven.resolveArtifact("org.keycloak", "keycloak-quarkus-dist").toFile();
                distDirName = distFile.getName().replace("-quarkus-dist", "");
            }
            distRootPath.toFile().mkdirs();
            Path dPath = distRootPath.resolve(distDirName.substring(0, distDirName.lastIndexOf('.')));

            if (!inited || (reCreate || !dPath.toFile().exists())) {
                FileUtil.deleteDirectory(dPath);
                ZipUtils.unzip(distFile.toPath(), distRootPath);

                if (System.getProperty("product") != null) {
                    // JDBC drivers might be excluded if running as a product build
                    copyProvider(dPath, "com.microsoft.sqlserver", "mssql-jdbc");
                }
            }

            // make sure script is executable
            if (!dPath.resolve("bin").resolve(SCRIPT_CMD).toFile().setExecutable(true)) {
                throw new RuntimeException("Cannot set " + SCRIPT_CMD + " executable");
            }
            if (!dPath.resolve("bin").resolve(SCRIPT_KCADM_CMD).toFile().setExecutable(true)) {
                throw new RuntimeException("Cannot set " + SCRIPT_KCADM_CMD + " executable");
            }

            inited = true;

            return dPath;
        } catch (Exception cause) {
            throw new RuntimeException("Failed to prepare distribution", cause);
        }
    }

11 replies

stianst Jan 30, 2025
Maintainer

Env vars are problematic as it would only work for distribution server mode, and not the embedded mode. What are you configuring with env variables? Could it be done with SPI provider config options instead?

michael-clay-mox Jan 30, 2025

dynamic configuration which can only be created at runtime and with BEFORE_KEYCLOAK_SERVER. think about dynamic port allocation which is done in OauthClientCallbackServer but for other backend systems which are need to be mocked during the integration tests.

michael-clay-mox Feb 10, 2025

any thoughts on how to provide env vars for the use case above?

is there a mode to recycle the keycloak/quarkus process instance for a whole unit i.e. test class instead of creating a process for each test method execution?

stianst Feb 17, 2025
Maintainer

If you set the env variable with export VAR=VALUE before running the tests it will be available to the server; as it will inherit the environment, but that is a bit of a limitation for sure.

We could add an option to add additional arguments to the startup, that can be set using an env variable, or using for instance the .test.env file.

Let's say we add something like KC_TEST_CUSTOM_ARGS or something then you can update that in any of the ways the testsuite allows being configured. Your custom code/provider would have to use SPI config options for that to work rather than using env variables directly.

michael-clay-mox Feb 17, 2025 8000

thanks for your feedback. I am pragmatic about the way i the configuration is consumed as long there is a way to do this. How would you provide this configuration option? via KeycloakServerConfigBuilder?

NobelNerdBenne · 2025-01-19T18:54:53Z

NobelNerdBenne
Jan 19, 2025

Hello Keycloak community,

First of all I think its a very good direction with the new test-framework! I think its a fantastic aspect for all extension developers to test there code with the same framework as the main keycloak.

Time for some feedback from the first tests and work with the new test-framework in our Keycloak extension.

For us it is not ready for use in our extension at the moment. It is too complicated to use.

As @stianst already mentioned in the following issue #34188 there is no easy way for extension developers without doing a Maven build. So we hope for a quick and good solution for this problem.

Cheers
Benne

6 replies

stianst Jan 22, 2025
Maintainer

Could you provide some more detail on what is too complicated to use? I can see the requirement to do a Maven build separately for sure, and that's something we plan to update, but wondering if there's any other feedback you have?

8000

NobelNerdBenne Jan 30, 2025

Hi @stianst , thanks for your quick reply.

In our old test setup we used Niko Köbler's Testcontainers, where the extension jar from the target folder is used. This had the great advantage that we could use one module with one pom. Now we had to split our extsion with more modules, more poms and more overhead for a small extension. This makes it more "complicated" for us.

Some other feedback from the first test I wrote. I want to test our authenticator and configure and develop the flow with the authenticator. It seemed logical to me to do this with the RealmBuilder, as flows are something I use in the context of a realm, but there is no option. The only solution I have found is to use the AdminClient and then bind it to the realm.

stianst Jan 30, 2025
Maintainer

Why would you need to split an extension into multi-modules? The general idea here is that you should be able to have a single module (src/main/java with your provider code, and src/test/java with your tests). However, there's a limitation right now that you have to build the module with Maven first, and specify the module as a dependency to include. This is something we want to improve in #34188, but not sure when we get around to that one.

stianst Jan 30, 2025
Maintainer

RealmBuilder is not complete yet, and only covers some stuff. It will eventually be expanded to support anything you can do with a realm like adding a custom auth flow.

michael-clay-mox Jan 30, 2025

@stianst regarding the separation into two modules ..one use case would be to separate prod and test code when test code contains extension providers that should only active during integration test

Riciardos · 2025-01-23T13:26:22Z

Riciardos
Jan 23, 2025

Hi all,

Just wanted to first say thank you for setting this up, I always love a good containerized framework that can run full integration tests without too much hassle.

I do want to share some things I've come across trying to set up tests for a slightly customized TokenExchangeProvider, that I'm sure are being worked on or would be fixed in the future anyway but here it goes:

in v26.1, the features(TOKEN_EXCHANGE) in KeycloakServerConfigBuilder doesn't work, the logs say "token-exchange" is not a recognized feature. Probably it's missing the v1 in it. My current workaround was just to do option("features","preview") instead.
I think there was a previous version in v25 where you could inject a RealmRepresentation.class, which allowed me to setup a configuration in the dev server locally, export it to a JSON file and read it in the test. I have gotten around that with the partialImport() function on the admin object, but it seems like the update(RealmRepresentation) function didn't seem to do anything? All this is probably on purpose as you can't export a full RealmRepresentation in v26 anyway, but maybe an example in the docs would be nice (I might open some PR for this later this week if you'd like).
Not being able to pass in custom properties into the test. My customized TokenExchangeProvider needs a baseUrl property, which I can set by using an environment variable, but it doesn't seem to work by just adding it to keycloak-test.properties where you can change the log level etc. This is the final hurdle I'm trying to get by, as I'd like to get our CI/CD pipeline to (and anyone pulling in our repo) to just do mvn test and not worry about setting the environment variable correctly first.
Lastly, and this is more of a PEBKAC issue anyway, but I've ran into the problem that when I run the tests while I've accidentally left a dev server running at the same time, the tests start up but fail quickly with a 401 Not authorized and not much else. It turns out the test admin starts firing requests that are picked up by the running instance instead of the test framework, so maybe some detection that checks if there is already something listening on the port would be a nice to have. (I sadly spent longer than I'd like to admit trying to figure out if I had somehow permanently broken the framework before I noticed I had a instance running that was causing it to fail)

Anyway, thanks for getting the framework going, I'm certain it'll be of great use for many!

3 replies

stianst Jan 30, 2025
Maintainer

Tried to reproduce the token exchange issue, but the following works for me:

    public static class CustomServerConfig implements KeycloakServerConfig {

        @Override
        public KeycloakServerConfigBuilder configure(KeycloakServerConfigBuilder config) {
            return config.features(Profile.Feature.TOKEN_EXCHANGE);
        }

    }

stianst Jan 30, 2025
Maintainer

For your TokenExchangeProvider perhaps use SPI provider config instead of an environment variable? That would make it configurable using CLI arguments, which you could set with a custom server config (option("spi-oauth2-token-exchange-myprovider-myoption", "value")). We could perhaps also consider adding some support for defining options for the server in the test config itself.

One challenge with environment variables is that we support both an embedded and a distribution mode, where the first doesn't work with setting environment variables as config options.

stianst Jan 30, 2025
Maintainer

We could probably add support for injecting a realm with a json as the config instead of directly specifying it using the realm builder. Could you create an issue for that and link it here?

LucasPratesOriginal · 2025-01-27T19:09:08Z

LucasPratesOriginal
Jan 27, 2025

Hi guys! I'm trying to figure out how to test one of my providers, it's an class that extends the Authenticator interface, any example of how i test the main methods like authenticate() and action()? I would appreciate the help.

2 replies

lhanusov Jan 30, 2025
Collaborator Author

Hello @LucasPratesOriginal, thank you for your question, right now, there is only a small PoC of OAuth client -> https://github.com/keycloak/keycloak/blob/main/test-framework/oauth-nimbus-poc/src/main/java/org/keycloak/testframework/oauth/nimbus/OAuthClient.java. Right now, there are only few methods, but is planned to extend it. We will discuss it within a team meeting and get back to you with more proper answer. :-)

stianst Jan 30, 2025
Maintainer

At the moment we are focusing on integration tests; as such you don't have direct access to calling methods on authenticators, but rather would drive this through the login pages to test.

We're adding support for RunOnServer to the new test framework this will allow running code on the server side. However, you'd need to mock up the authentication context, etc. to be able to test an authenticator.

At some point in the future we're hoping to do a provider test of sorts; that should allow testing a single provider directly.

BenLad17 · 2025-01-27T22:51:35Z

BenLad17
Jan 27, 2025

Hi, thanks for the new framework. I think it has a lot of potential.
I'm trying to get the test coverage of my tests but it seems to be difficult because the keycloak instance is spawned in a new process.
I would like to see a way to get the test coverage of this process.

Test coverage tools like JaCoCo use a Java agent to collect the data they need.
Could you perhaps pass the -javaagent input argument of ManagementFactory.getRuntimeMXBean().getInputArguments() to the child process or provide an option to add arguments to the start command?

Or do you know of a way to solve the problem that I have not yet discovered?

4 replies

lhanusov Jan 30, 2025
Collaborator Author

Hello @BenLad17, thank you for your feedback, did you try to use embedded server mode (distribution, is set as default)?
You can switch it by using .env-test file: https://github.com/keycloak/keycloak/tree/main/test-framework#configuring-the-test-framework and define KC_TEST_SERVER=embedded in it: https://github.com/keycloak/keycloak/tree/main/test-framework#config-options. This should help, since the distribution mode is in a separate process and the embedded is not.

stianst Jan 30, 2025
Maintainer

As @lhanusov suggests using embedded mode is simper for this purpose, and also any cases where you want to debug your provider code as you don't need to do any attaching of the debugger to a remote interface.

PeterMue Oct 15, 2025

I'm trying to get jacoco coverage working in a multi-module maven project with no success.
My first attempt was to switch from distribution to embedded test-server but this makes no difference at all.
I stumbled across this quarkus documentation which seems to be a bit tricky to get jacoco working with quarkus at all.

Has anybody successfully configured a multi-module project with code-coverage, where integration-tests are in a separate maven module?

michael-clay-mox Oct 15, 2025

Using the default option RawKeycloakDistribution spawns a process which inherits the environment variable of the parent process which allows to just use JAVA_OPTS_APPEND to pass through the javaagent:jacoco... settings.

michael-clay-mox · 2025-01-30T16:42:31Z

michael-clay-mox
Jan 30, 2025

@stianst what is the suggested way to deploy a custom realm when using integration tests pls

4 replies

lhanusov Feb 3, 2025
Collaborator Author

@michael-clay-mox Hello, here, you can check the manual for it:
https://github.com/keycloak/keycloak/tree/main/test-framework#configuring-resources
or the example:
https://github.com/keycloak/keycloak/blob/main/test-framework/examples/tests/src/test/java/org/keycloak/test/examples/FancyRealmTest.java
If you have already an imported realm in a database, you can attach the ManagedRealm to it: https://github.com/keycloak/keycloak/blob/main/test-framework/examples/tests/src/test/java/org/keycloak/test/examples/AttachToTest.java#L22C18-L22C26

Does it answer your question?

michael-clay-mox Feb 3, 2025

thanks for the links...i remember i already saw the fancyrealmtest when you released the testframework. The background of my question was the creation and deployment of a custom authentication flow. What is your opinion on that ..should it be setup on the client side (rest representation) or on the server side? Is there anything ready which integrates smoothly with the new testframework?

michael-clay-mox Feb 17, 2025

@lhanusov any thoughts on that?

lhanusov Feb 26, 2025
Collaborator Author

Apologize for my late replay, right now it's not implemented, but it should be part of RealmConfigBuilder class. Please, can you open an issue for it and assign a label test-framework? Thank you

hho · 2025-04-07T09:30:25Z

hho
Apr 7, 2025

First of all: Thank you for the new test framework, it is absolutely an improvement! ❤️

I have only one issue I couldn't resolve so far: When starting a test with a customized distribution (i.e. testing my extension), org.keycloak.it.utils.RawKeycloakDistribution uses an internal Maven resolver to resolve some artifacts. How can I configure, which Maven settings this resolver uses? It doesn't use the settings from the Maven build running the test (that's expected, how would it get the info?), but I need it do point to a private repo…

1 reply

lhanusov Apr 16, 2025
Collaborator Author

Hello @hho, thanks for trying it out, I hope it has better user experience then the Arquillian one.
Our Maven resolver works this way - link, you must specify your dependency in the pom.xml, initial it tries to look for it in the local repo, if not present it calls remote repos. Right now we don't plan to extend our util class. I propose to set internal repositories in pom.xml - link. Good luck!

Nahuel92 · 2025-04-21T14:13:42Z

Nahuel92
Apr 21, 2025

Hi everyone!

If this doesn't belong here, please let me know and I'll delete the comment.

The new framework looks promising, but I ran into the following issue, and I couldn't find out a way to solve it yet:
I have a custom JPA user federation, and I want to test it. My understanding is that, currently, it's complicated to do so.

So far, I have:

Configured TestContainers to spin up a disposable DB container
- DB port and database name are hardcoded to match what's defined on quarkus.properties
Configured @KeycloakIntegrationTest to use my implementation of KeycloakServerConfig, where I define my custom federation

I haven't found a way to:

Programmatically configure my federation in a realm defined via a custom implementation of RealmConfig. I assume I should export a realm as a JSON from a real instance and then import it on my test Keycloak instance.
Automatically wire things up (I believe the @KeycloakIntegrationTest is not leveraging on Quarkus to autoconfigure things such as env. variables, DB configuration, DB initialization scripts execution, etc.)

I would like to know if someone has already solved this and can give me some pointers on how to wire things up.

Thanks in advance!

4 replies

lhanusov May 16, 2025
Collaborator Author

Hello @Nahuel92,

I apologize for a late replay. I will try to explain it this way:

@KeycloakIntegrationTest is based on JUnit 5 framework only, but we use Smallrye to configure some things, like selecting db or server type, logging, etc.: https://github.com/keycloak/keycloak/blob/main/test-framework/core/src/main/java/org/keycloak/testframework/server/AbstractKeycloakServerSupplier.java#database
right now, there is no option on how to connect external db, so it is not possible to specify db connection string, user, password, port, etc., but we plan to implement it
when using Testcontainers you can specify your container image in your pom.xml: https://github.com/keycloak/keycloak/blob/main/pom.xml#L170
federation configuration in RealmConfig is not implemented yet, but we plan to do so, since we are migrating old tests to the new framework -> work in progress
-- as a workaround you can import your realm using a json file: https://github.com/keycloak/keycloak/blob/main/tests/base/src/test/java/org/keycloak/tests/admin/partialexport/PartialExportTest.java#L49-L50

Does it answer your question?

@stianst Stian if I missed something, can you please add your comment? Thanks.

Nahuel92 May 21, 2025

Hi @lhanusov!

Thank you, that helps. I'll spend some time to see if I can work around the missing pieces (i.e., configuring the user federation DB on Keycloak).

The motivation behind this is that I want to write integration tests for my custom federation so that, in the future, I can have confidence that nothing gets broken when upgrading Keycloak to newer versions (this actually happened to me, and I want to avoid it in the future).

lhanusov Jun 6, 2025
Collaborator Author

Stian added this PR, to unblock development for people missing some features, but it is a workaround, which will be supported only for some time: #40312

Nahuel92 Jun 10, 2025

Hi lhanusov!

Thanks for the heads up. It may be useful for me because this week I'm finally working again on the integration tests. I'll share my feedback as soon as I can.

suyash1002 · 2025-04-30T07:14:14Z

suyash1002
Apr 30, 2025

Hey can anyone help me understand, How do I get started with keycloak test framework. I have gone through guides, I am new to keycloak test framework. How do I get started with the pom.xml, and also test framework for custom providers? Thanks in advance.

1 reply

lhanusov May 16, 2025
Collaborator Author

Hello @suyash1002, here is an example on how you can make it work:

example of the provider: https://github.com/keycloak/keycloak/tree/main/test-framework/examples/providers
example of the test module with the particular class: https://github.com/keycloak/keycloak/tree/main/test-framework/examples/providers
-- pom module dependency: https://github.com/keycloak/keycloak/blob/main/test-framework/examples/tests/pom.xml#L91-L96

Right now, if you want to test your custom provider, you need to have a separate test module. Server injection of custom provider for the same module is not ready yet -> it is planned as a future feature.

volck · 2025-06-11T06:01:43Z

volck
Jun 11, 2025

I'm working several custom providers right now. Being able to simply test them using this framework would be great. What is the ETA on this being available? Right now the quality of life for testing this is pretty poor and the cycle takes longer than it should.

4 replies

volck Jun 11, 2025

tagging @lhanusov just in case :)

< 10BC8 div class="discussions-timeline-scroll-target" id="discussioncomment-14211828" data-gid="DC_kwDOAKnDVc4A2Nr0" data-url="/keycloak/keycloak/discussions/34951/comments/14211828" data-error="" >

lhanusov Aug 25, 2025
Collaborator Author

Hello @volck, I apologize for not spotting this comment - this framework is already in place since last November, first few tests were migrated in January, now almost the whole admin package is migrated to the new framework - link

As an extension developer, you can leverage the framework as well, here is an example:

Your maven structure, must look like this (right now, there is no, on the fly dependency build):

parent-module
\____[provider-module](https://github.com/keycloak/keycloak/tree/main/test-framework/examples/providers)
        \____pom.xml
\____[test-module](https://github.com/keycloak/keycloak/blob/main/test-framework/examples/tests/src/test/java/org/keycloak/test/examples/MyCustomProviderTest.java)
        \____[pom.xml](https://github.com/keycloak/keycloak/blob/main/test-framework/examples/tests/pom.xml#L75-L80) (dependency to provider-module)
\____pom.xml

volck Aug 25, 2025

Great! Thanks for the reply! Already figured it out and currently have it implemented! 😊

lhanusov Sep 2, 2025
Collaborator Author

Good to hear that :-).

lhanusov · 2026-02-25T09:24:23Z

lhanusov
Feb 25, 2026
Collaborator Author

Closing, since the full support will be announced in the next few days. Thank you for your contributions.

0 replies

Preview - Keycloak Test Framework #34951

Uh oh!

lhanusov Nov 14, 2024 Collaborator

Replies: 11 comments · 40 replies

Uh oh!

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

Uh oh!

Uh oh!

stianst Feb 17, 2025 Maintainer

Uh oh!

michael-clay-mox Feb 17, 2025 8000

Uh oh!

Uh oh!

stianst Jan 22, 2025 Maintainer

Uh oh!

Uh oh!

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

Uh oh!

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

Uh oh!

lhanusov Jan 30, 2025 Collaborator Author

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

Uh oh!

lhanusov Jan 30, 2025 Collaborator Author

Uh oh!

stianst Jan 30, 2025 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lhanusov Feb 3, 2025 Collaborator Author

Uh oh!

Uh oh!

Uh oh!

lhanusov Feb 26, 2025 Collaborator Author

Uh oh!

Uh oh!

lhanusov Apr 16, 2025 Collaborator Author

Uh oh!

Uh oh!

lhanusov May 16, 2025 Collaborator Author

Uh oh!

Uh oh!

lhanusov
Nov 14, 2024
Collaborator

Replies: 11 comments 40 replies

stianst Jan 30, 2025
Maintainer

stianst Feb 17, 2025
Maintainer

stianst Jan 22, 2025
Maintainer

stianst Jan 30, 2025
Maintainer

stianst Jan 30, 2025
Maintainer

stianst Jan 30, 2025
Maintainer

stianst Jan 30, 2025
Maintainer

stianst Jan 30, 2025
Maintainer

lhanusov Jan 30, 2025
Collaborator Author

stianst Jan 30, 2025
Maintainer

lhanusov Jan 30, 2025
Collaborator Author

stianst Jan 30, 2025
Maintainer

lhanusov Feb 3, 2025
Collaborator Author

lhanusov Feb 26, 2025
Collaborator Author

lhanusov Apr 16, 2025
Collaborator Author

lhanusov May 16, 2025
Collaborator Author