8000 Refactor builders in testsuite part 1 by stianst · Pull Request #48315 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

Refactor builders in testsuite part 1 - #48315

Merged
stianst merged 2 commits into
keycloak:mainfrom
stianst:tf-builder-refactor
Apr 23, 2026
Merged

stianst merged 2 commits into
keycloak:mainfrom
stianst:tf-builder-refactor

Conversation

@stianst
@stianst stianst commented Apr 21, 2026 •
Copy link
Copy Markdown
Contributor

No description provided.

@stianst
stianst force-pushed the tf-builder-refactor branch 3 times, most recently from 2f3fb62 to 497a248 Compare April 21, 2026 12:26
@stianst stianst changed the title tf builder refactor Builder refactor WIP Apr 21, 2026
@stianst
stianst force-pushed the tf-builder-refactor branch from 497a248 to d5230c4 Compare April 21, 2026 12:46
@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.model.singleUseObject.SingleUseObjectModelTest#testCluster

Keycloak CI - Store Model Tests

java.lang.AssertionError: 
threads didn't terminate in time: [main (RUNNABLE):
	at java.management@25.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:505)
	at java.management@25.0.2/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:493)
...

Report flaky test

org.keycloak.testsuite.model.user.UserModelTest#testAddRemoveUserConcurrent

Keycloak CI - Store Model Tests

java.lang.NullPointerException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at java.base/java.util.concurrent.ForkJoinTask.getException(ForkJoinTask.java:561)
...

Report flaky test

@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

@stianst
stianst force-pushed the tf-builder-refactor branch from 5f4645e to 1120d0f Compare April 22, 2026 11:17
@stianst stianst changed the title Use UserConfigBuilder in old testsuite Refactor builders in testsuite part 1 Apr 22, 2026
@stianst
stianst force-pushed the tf-builder-refactor branch from 1120d0f to 27bca59 Compare April 22, 2026 11:23
@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@stianst
stianst requested a review from lhanusov April 22, 2026 13:55
@lhanusov lhanusov left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stianst thanks for this PR, please check my comments and let me know what you think.


import org.keycloak.representations.idm.AuthenticationExecutionExportRepresentation;

public class AuthenticationExecutionExportBuilder {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan to move it 8BD0 to the builders module and extend the Builder class?

@@ -4,61 +4,61 @@
import org.keycloak.representations.idm.AuthenticationFlowRepresentation;
import org.keycloak.testframework.util.Collections;

public class AuthenticationFlowConfigBuilder {
public class AuthenticationFlowBuilder {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan to move it to the builders module and extend the Builder class?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@@ -23,169 +23,169 @@
import org.keycloak.representations.idm.UserRepresentation;
import org.keycloak.testframework.util.Collections;

public class RealmConfigBuilder {
public class RealmBuilder {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan to move it to the builders module and extend the Builder class?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Comment on lines +98 to +99
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, not sure what happened here as I used IntelliJ to refactor, good catch though

Comment on lines +121 to +122
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus add import

Suggested change
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name(SCOPE_DEPARTMENT).protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("profile").protocol("openid-connect").build());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment on lines +57 to +61
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-1-sel").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-1").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-2-sel").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-2").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-3-sel").build());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus add import

Suggested change
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-1-sel").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-1").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-2-sel").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-2").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("phone-3-sel").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("phone-1-sel").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("phone-1").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("phone-2-sel").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("phone-2").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("phone-3-sel").build());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment on lines +115 to +117
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("customer").protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("client-a").protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("some-optional-scope").protocol("openid-connect").build());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus add import

Suggested change
10BC8
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("customer").protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("client-a").protocol("openid-connect").build());
testRealm.getClientScopes().add(org.keycloak.testframework.realm.ClientScopeBuilder.create().name("some-optional-scope").protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("customer").protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("client-a").protocol("openid-connect").build());
testRealm.getClientScopes().add(ClientScopeBuilder.create().name("some-optional-scope").protocol("openid-connect").build());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@@ -138,7 +139,7 @@ public RealmBuilder client(ClientRepresentation client) {
return this;
}

public RealmBuilder clientScope(ClientScopeBuilder clientScope) {
public RealmBuilder clientScope(org.keycloak.testframework.realm.ClientScopeBuilder clientScope) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plus add import

Suggested change
public RealmBuilder clientScope(org.keycloak.testframework.realm.ClientScopeBuilder clientScope) {
public RealmBuilder clientScope(ClientScopeBuilder clientScope) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you plan to remove this file in the next step?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@stianst
stianst force-pushed the tf-builder-refactor branch from 27bca59 to d5abe56 Compare April 23, 2026 06:07
Closes keycloak#48323

Signed-off-by: stianst <stianst@gmail.com>
@stianst
stianst force-pushed the tf-builder-refactor branch from e10cda2 to 8759445 Compare April 23, 2026 08:53
Signed-off-by: stianst <stianst@gmail.com>
@keycloak-github-bot keycloak-github-bot Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link
Copy Markdown

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.authz.PolicyEvaluationTest#

Keycloak CI - Base IT (3)

org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...
org.opentest4j.AssertionFailedError: expected: <PERMIT> but was: <null>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
...

Report flaky test

@stianst
stianst marked this pull request as ready for review April 23, 2026 10:31
@stianst
stianst enabled auto-merge (squash) April 23, 2026 10:31
@stianst
stianst merged commit 17a3738 into keycloak:main Apr 23, 2026
90 checks passed
@lhanusov lhanusov left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stianst stianst mentioned this pull request Jun 1, 2026
mabartos pushed a commit to mabartos/keycloak-adaptive-authn that referenced this pull request Jun 4, 2026
…k-test-framework-builders (#46)

* fix(tests): replace RealmConfigBuilder with RealmBuilder from keycloak-test-framework-builders

Following keycloak/keycloak#48315

Signed-off-by: Thomas DELORGE <thomas.delorge@orbeet.io>

* fix(ci): dual realm config sources for multi-version Keycloak matrix

Split AdaptiveRealmConfig into java-nightly (RealmBuilder) and
java-released (RealmConfigBuilder), selected via keycloak.test.realm.source.
CI uses separate build-args and test-args; stable builds use released
sources, 26.5.7 tests run against 26.6.0. Scope builders dependency to nightly only.

Signed-off-by: Thomas DELORGE <thomas.delorge@orbeet.io>

---------

Signed-off-by: Thomas DELORGE <thomas.delorge@orbeet.io>
sashyo pushed a commit to tide-foundation/keycloak-IGA that referenced this pull request Jul 20, 2026
Refactor builders in testsuite part 1

Closes keycloak#48323

Signed-off-by: stianst <stianst@gmail.com>
@stianst
stianst deleted the tf-builder-refactor branch October 1, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor builders in testsuite part 1

3 participants

0