No downtime upgrades - tests #14434

vramik · 2022-09-15T14:34:36Z

vramik
Sep 15, 2022
Collaborator

The times comes to the point when we might need to start thinking and implementing set of tests which will cover no-downtime upgrade between different versions of keycloak. This discussion should serve as a single point where we can collect requirements, limitations, ideas etc.

Few notes from discussions:

no-downtime upgrade workflow
New store is designed to be no-downtime upgradable by connecting a server with new version into a cluster of servers with older versions. The cluster has to work well together. It implies that it might be one of a tested scenarios.

ability to read entities from version n+1
For heterogeneous cluster to work well it has to be ensured that entity from older keycloak version could be read and used by new version of keycloak and vice-versa. It could also be one of the possible tests.

first iteration may be jpa map storage and users area as it's the one where the entity version already bumped
later we may add such a test for other areas as well

run in GHA
It might be beneficial to have such tests in GHA to be able to catch possible issues early in development process.

arquillian base testsuite
It may not be ideal to use current arquillian base testsuite for the $SUBJ as it's planned to be possibly replaced by something else in near-ish future. Still it might be long time till it'd be replaced and we should probably come up with something soon.

vramik · 2023-02-14T10:14:44Z

vramik
Feb 14, 2023
Collaborator Author

Few additional notes.

base testsuite

possibility to start legacy keycloak container using testcontainers
smoke test (rest calls to use CRUD operations to test reading objects from the other version) should be relatively easy
- it should be avoided using adminClient as there could be some incompatible changes between the versions
both keycloaks could run on quarkus -> all quarkus-specific factories would be tested
to test more fine grained tests (with knowledge of individual fields) access to session is required
- to be able to use runOnServer capability (to access the session) testsuite-providers with legacy kc version and deployed into the container is required
  - possibility to extract the runOnServer feature from testsuite-providers - the exctracted feature could be more stable so we might be able to use single version of it

model tests

If we want to leverage model tests we have an access to session there.
we could use two containers, each of them representing different version
extracted runOnServer feature could be used to execute pieces of of code in appropriate server
i.e. have two separate docker images listening and accepting serializables

Test environment
Do we need to start legacy keycloak before the current one or would it be sufficient to "just add" a legacy keycloak into a cluster? From JPA POV is should be ok to do the latter. There might be a need to go with first one for hotrod as there are some indexes created in multiple steps (as pointed out by @mhajas )

4 replies

vramik Feb 14, 2023
Collaborator Author

The good start seems to be a base testsuite with the smoke test. Then we could follow by extraction of runOnServer feature.

vramik Feb 16, 2023
Collaborator Author

Test environment
Do we need to start legacy keycloak before the current one or would it be sufficient to "just add" a legacy keycloak into a cluster? From JPA POV is should be ok to do the latter. There might be a need to go with first one for hotrod as there are some indexes created in multiple steps (as pointed out by @mhajas )

TL;DR: According to local tests it turned out we should follow the "standard" workflow -> start legacy server first, then start current one.

If we'd follow "just add" a legacy keycloak into a cluster it may happen (in jpa map storage specifically) that some of legacy keycloak liquibase chanset are not evaluated as EXECUTED or MARK_RUN [1] and therefore legacy keycloak tries to execute them again, which could lead to issues like relation "kc_realm" already exists.

[1] https://javadoc.io/doc/org.liquibase/liquibase-core/4.18.0/index.html

Example with such a failure could be seen with following workflow:

start current keycloak/main (kc21) with jpa map storage
start keycloak 20 with jpa map storage
see the log for error:

Caused by: liquibase.exception.MigrationFa
7438
iledException: Migration failed for change set META-INF/realms/jpa-realms-changelog-1.xml::realms-1::keycloak:
     
Reason: liquibase.exception.DatabaseException: ERROR: relation "kc_realm" already exists [Failed SQL: (0) CREATE TABLE public.kc_realm (id UUID NOT NULL, version INTEGER DEFAULT 0 NOT NULL, metadata JSONB, CONSTRAINT kc_realm_pkey PRIMARY KEY (id))]

The cause of it is that there has been this change:

keycloak/model/map-jpa/src/main/resources/META-INF/realms/jpa-realms-changelog-1.xml

Line 25 in dbe2257

->

keycloak/model/map-jpa/src/main/resources/META-INF/realms/jpa-realms-changelog-1.xml

Line 25 in f7ad002

For more information: #15724

mhajas Feb 16, 2023

Thank you for investigating this @vramik. I am not sure this is not a bug. I would say, we should support downgrade even if the previous version never ran in the cluster.

How would the operator know whether the previous version was running or not when someone triggers a downgrade?

The way I understood it was that under any circumstances we need to count on the fact that if we have node version X running we need to handle a situation when a node version X-1 spawns.

cc: @hmlnarik

vramik Feb 16, 2023
Collaborator Author

Thank you @mhajas. I am also not sure. It depends where we want to make a line what is still supported and what not.

E.g. in scenario where version X would be spawned, then upgrade to X+1 would be done and then downgraded to X, then X-1 won't be possible.

I was reading the design https://github.com/keycloak/keycloak-community/blob/main/design/keycloak.x/storage-persistence.md again, but it is not described there into such details.

Sadly the issue with changesets cannot be fixed as the window is closed already with kc20 released, but we can make it workable since kc20. It'd mean we should add a test for this scenario if it'd be supported.

To me it makes sense to support downgrades to the version X.

@hmlnarik do you know what scenarios we need to support and what won't be supported on the field of downgrades?

Navigation Menu

Repository navigation

No downtime upgrades - tests #14434

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

No downtime upgrades - tests #14434

Uh oh!

Uh oh!

vramik Sep 15, 2022 Collaborator

Replies: 1 comment · 4 replies

Uh oh!

Uh oh!

vramik Feb 14, 2023 Collaborator Author

Uh oh!

vramik Feb 14, 2023 Collaborator Author

Uh oh!

vramik Feb 16, 2023 Collaborator Author

TL;DR: According to local tests it turned out we should follow the "standard" workflow -> start legacy server first, then start current one.

Uh oh!

mhajas Feb 16, 2023

Uh oh!

Uh oh!

vramik Feb 16, 2023 Collaborator Author

vramik
Sep 15, 2022
Collaborator

Replies: 1 comment 4 replies

vramik
Feb 14, 2023
Collaborator Author

vramik Feb 14, 2023
Collaborator Author

vramik Feb 16, 2023
Collaborator Author

vramik Feb 16, 2023
Collaborator Author