8000 File store basis by hmlnarik · Pull Request #16001 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

File store basis - #16001

Merged
hmlnarik merged 18 commits into
keycloak:mainfrom
hmlnarik:13632-File-map-storage-Group-no-downtime-store-main
Feb 7, 2023
Merged

hmlnarik merged 18 commits into
keycloak:mainfrom
hmlnarik:13632-File-map-storage-Group-no-downtime-store-main

Conversation

@hmlnarik
@hmlnarik hmlnarik commented Dec 14, 2022 •
Copy link
Copy Markdown
Contributor

This PR introduces file store basis. It includes generic file store based on YAML in a directory structure, and Keycloak is able to run with file store for almost all areas, see Q&A. The limitations are:

Fixes: #13632
Fixes: #15885

@hmlnarik hmlnarik added area/storage Storage layer — JPA entities, database schemas, Liquibase migrations, caching team/storage-sig labels Dec 14, 2022
@hmlnarik
hmlnarik requested review from sguilhen and vramik December 14, 2022 16:45
@hmlnarik hmlnarik self-assigned this Dec 14, 2022
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch 4 times, most recently from 9c9dc8c to b2c1ce9 Compare December 19, 2022 14:31
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch from b2c1ce9 to 86008bd Compare January 9, 2023 17:10
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch 2 times, most recently from 59a236e to f1c613b Compare January 25, 2023 19:30
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch 2 times, most recently from 89604fb to e0ec287 Compare January 27, 2023 13:18
@hmlnarik hmlnarik changed the title Group file store File store basis Jan 27, 2023
@hmlnarik
hmlnarik marked this pull request as ready for review January 27, 2023 13:22

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression

This path depends on a [user-provided value](1). This path depends on a [user-provided value](2). This path depends on a [user-provided value](3). This path depends on a [user-provided value](4). This path depends on a [user-provided value](5). This path depends on a [user-provided value](6). This path depends on a [user-provided value](7). This path depends on a [user-provided value](8). This path depends on a [user-provided value](9). This path depends on a [user-provided value](10). This path depends on a [user-provided value](11). This path depends on a [user-provided value](12). This path depends on a [user-provided value](13). This path depends on a [user-provided value](14). This path depends on a [user-provided value](15). This path depends on a [user-provided value](16). This path depends on a [user-provided value](17). This path depends on a [user-provided value](18). This path depends on a [user-provided value](19). This path depends on a [user-provided value](20). This path depends on a [user-provided value](21). This path depends on a [user-provided value](22). This path depends on a [user-provided value](23). This path depends on a [user-provided value](24). This path depends on a [user-provided value](25). This path depends on a [user-provided value](26). This path depends on a [user-provided value](27). This path depends on a [user-provided value](28). This path depends on a [user-provided value](29). This path depends on a [user-provided value](30). This path depends on a [user-provided value](31). This path depends on a [user-provided value](32). This path depends on a [user-provided value](33). This path depends on a [user-provided value](34). This path depends on a [user-provided value](35). This path depends on a [user-provided value](36). This path depends on a [user-provided value](37). This path depends on a [user-provided value](38). This path depends on a [user-provided value](39). This path depends on a [user-provided value](40). This path depends on a [user-provided value](41). This path depends on a [user-provided value](42). This path depends on a [user-provided value](43). This path depends on a [user-provided value](44). This path depends on a [user-provided value](45). This path depends on a [user-provided value](46). This path depends on a [user-provided value](47). This path depends on a [user-provided value](48). This path depends on a [user-provided value](49). This path depends on a [user-provided value](50). This path depends on a [user-provided value](51). This path depends on a [user-provided value](52). This path depends on a [user-provided value](53). This path depends on a [user-provided value](54). This path depends on a [user-provided value](55). This path depends on a [user-provided value](56). This path depends on a [user-provided value](57).
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch 2 times, most recently from 46a9066 to ee843c9 Compare January 30, 2023 20:35
@hmlnarik
hmlnarik requested a review from a team January 30, 2023 20:35
@hmlnarik
hmlnarik requested review from a team as code owners January 30, 2023 20:35
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch from ee843c9 to 7bb41ba Compare February 1, 2023 14:47
Comment on lines 137 to 135

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression

This path depends on a [user-provided value](1). This path depends on a [user-provided value](2). This path depends on a [user-provided value](3). This path depends on a [user-provided value](4). This path depends on a [user-provided value](5). This path depends on a [user-provided value](6). This path depends on a [user-provided value](7). This path depends on a [user-provided value](8). This path depends on a [user-provided value](9). This path depends on a [user-provided value](10). This path depends on a [user-provided value](11). This path depends on a [user-provided value](12). This path depends on a [user-provided value](13). This path depends on a [user-provided value](14). This path depends on a [user-provided value](15). This path depends on a [user-provided value](16). This path depends on a [user-provided value](17). This path depends on a [user-provided value](18). This path depends on a [user-provided value](19). This path depends on a [user-provided value](20). This path depends on a [user-provided value](21). This path depends on a [user-provided value](22). This path depends on a [user-provided value](23). This path depends on a [user-provided value](24). This path depends on a [user-provided value](25). This path depends on a [user-provided value](26). This path depends on a [user-provided value](27). This path depends on a [user-provided value](28). This path depends on a [user-provided value](29). This path depends on a [user-provided value](30). This path depends on a [user-provided value](31). This path depends on a [user-provided value](32). This path depends on a [user-provided value](33). This path depends on a [user-provided value](34). This path depends on a [user-provided value](35). This path depends on a [user-provided value](36). This path depends on a [user-provided value](37). This path depends on a [user-provided value](38). This path depends on a [user-provided value](39). This path depends on a [user-provided value](40). This path depends on a [user-provided value](41). This path depends on a [user-provided value](42). This path depends on a [user-provided value](43). This path depends on a [user-provided value](44). This path depends on a [user-provided value](45). This path depends on a [user-provided value](46). This path depends on a [user-provided value](47). This path depends on a [user-provided value](48). This path depends on a [user-provided value](49). This path depends on a [user-provided value](50). This path depends on a [user-provided value](51). This path depends on a [user-provided value](52). This path depends on a [user-provided value](53). This path depends on a [user-provided value](54). This path depends on a [user-provided value](55). This path depends on a [user-provided value](56). This path depends on a [user-provided value](57).
@vramik vramik left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @hmlnarik. It's great to see it. I like it very much.

I'd like to take a closer look at it, in the meantime I went through the code and I have few preliminary questions, please see inline.

Comment thread model/map-file/pom.xml Resolved
@vramik
vramik self-requested a review February 1, 2023 15:57
@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch from 8c5c410 to 6055376 Compare February 7, 2023 07:47
@hmlnarik
hmlnarik commented Feb 7, 2023 •
Copy link
Copy Markdown
Contributor Author

Rebased due to conflict in pom.xml

@martin-kanis
Copy link
Copy Markdown
Member

When I start Keycloak (keycloak-server from testsuite) and log in as an admin, then log out, log in again and so on, I see that user session are not being removed. I see multiple user session yaml files and also multiple active sessions under "sessions" tab in admin console. Is this expected?
When trying with pure CHM (with additional server's restarts to persist data in files), I don't see multiple user session inside map-user-sessions.json

@hmlnarik
hmlnarik force-pushed the 13632-File-map-storage-Group-no-downtime-store-main branch 10BC8 from 6055376 to a28b739 Compare February 7, 2023 10:00
@hmlnarik
hmlnarik commented Feb 7, 2023
Copy link
Copy Markdown
Contributor Author

When I start Keycloak (keycloak-server from testsuite) and log in as an admin, then log out, log in again and so on, I see that user session are not being removed. I see multiple user session yaml files and also multiple active sessions under "sessions" tab in admin console. Is this expected? When trying with pure CHM (with additional server's restarts to persist data in files), I don't see multiple user session inside map-user-sessions.json

@martin-kanis Thank you, this has been fixed in the last commit (b47cbe8).

}

public void touch(Path path) throws FileAlreadyExistsException, IOException {
Files.createFile(path);

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression

This path depends on a [user-provided value](1). This path depends on a [user-provided value](2). This path depends on a [user-provided value](3). This path depends on a [user-provided value](4). This path depends on a [user-provided value](5). This path depends on a [user-provided value](6). This path depends on a [user-provided value](7). This path depends on a [user-provided value](8). This path depends on a [user-provided value](9). This path depends on a [user-provided value](10). This path depends on a [user-provided value](11). This path depends on a [user-provided value](12). This path depends on a [user-provided value](13). This path depends on a [user-provided value](14). This path depends on a [user-provided value](15). This path depends on a [user-provided value](16). This path depends on a [user-provided value](17). This path depends on a [user-provided value](18). This path depends on a [user-provided value](19). This path depends on a [user-provided value](20). This path depends on a [user-provided value](21). This path depends on a [user-provided value](22). This path depends on a [user-provided value](23). This path depends on a [user-provided value](24). This path depends on a [user-provided value](25). This path depends on a [user-provided value](26). This path depends on a [user-provided value](27). This path depends on a [user-provided value](28). This path depends on a [user-provided value](29). This path depends on a [user-provided value](30). This path depends on a [user-provided value](31). This path depends on a [user-provided value](32). This path depends on a [user-provided value](33). This path depends on a [user-provided value](34). This path depends on a [user-provided value](35). This path depends on a [user-provided value](36). This path depends on a [user-provided value](37). This path depends on a [user-provided value](38). This path depends on a [user-provided value](39). This path depends on a [user-provided value](40). This path depends on a [user-provided value](41). This path depends on a [user-provided value](42). This path depends on a [user-provided value](43). This path depends on a [user-provided value](44). This path depends on a [user-provided value](45). This path depends on a [user-provided value](46). This path depends on a [user-provided value](47). This path depends on a [user-provided value](48). This path depends on a [user-provided value](49). This path depends on a [user-provided value](50). This path depends on a [user-provided value](51). This path depends on a [user-provided value](52). This path depends on a [user-provided value](53). This path depends on a [user-provided value](54). This path depends on a [user-provided value](55). This path depends on a [user-provided value](56). This path depends on a [user-provided value](57).
"map": {
"storage-admin-events": {
"provider": "${keycloak.adminEventsStore.map.storage.provider:concurrenthashmap}"
"provider": "${keycloak.adminEventsStore.map.storage.provider,keycloak.mapStorage.provider.default:concurrenthashmap}"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this changes go into testsuite/integration-arquillian/tests/base/src/test/resources/META-INF/keycloak-server.json as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the arquillian testsuite, there are map-storage-* profiles which define all the necessary properties for map storage. Here it is only for convenience usage from CLI. So while I agree it would be nice, I consider this a follow-up task if the need arises.

@martin-kanis martin-kanis left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hmlnarik Thank you for the huge amount of work!

@hmlnarik
hmlnarik merged commit f71ab09 into keycloak:main Feb 7, 2023
@hmlnarik hmlnarik mentioned this pull request Feb 13, 2023
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/storage Storage layer — JPA entities, database schemas, Liquibase migrations, caching flaky-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add write ability to file store File map storage: Basic storage

5 participants

0