systemd graceful shutdown #10180

ririsoft · 2022-02-13T14:55:29Z

ririsoft
Feb 13, 2022

Hello,

I am using a basic systemd unit service file for keycloak 17.0.0 quarkus distribution on Debian 11 :

# /etc/systemd/system/keycloak.service
[Unit]
Description=The WildFly Application Server
After=syslog.target network.target postgresql.service
Before=httpd.service
PartOf=postgresql@13-main.service

[Service]
User=keycloak
UMask=0027
ExecStart=/opt/keycloak/keycloak-17.0.0/bin/kc.sh start 

[Install]
WantedBy=multi-user.target

But shutdown is always in systemd failed status with exit code 143.
Logs are not helping:

févr. 13 15:38:49 horous kc.sh[493274]: 2022-02-13 15:38:49,948 INFO  [org.infinispan.CLUSTER] (Shutdown thread) ISPN000080: Disconnecting JGroups channel `ISPN`
févr. 13 15:38:50 horous kc.sh[493274]: 2022-02-13 15:38:50,002 INFO  [io.quarkus] (Shutdown thread) Keycloak stopped in 0.113s

Do you have any best practice for shutting down keycloak.x properly with systemd, please ?
Thank you very much in advance for your help here.

tgurr · 2022-02-14T12:25:25Z

tgurr
Feb 14, 2022

Experiencing the same problem, I'm not sure if this is the correct solution or rather just a dirty workaround but it seems to be quite common in other systemd services files in java based software like e.g. openhab, adding:

SuccessExitStatus=0 143 to the keycloak.service works around the problem.

Feb 14 13:20:21 localhost kc.sh[174020]: 2022-02-14 13:20:21,725 INFO  [io.quarkus] (Shutdown thread) Keycloak stopped in 0.049s
Feb 14 13:20:21 localhost systemd[1]: keycloak.service: Deactivated successfully.
Feb 14 13:20:21 localhost systemd[1]: Stopped keycloak.service - Keycloak server.
Feb 14 13:20:21 localhost systemd[1]: keycloak.service: Consumed 11.066s CPU time.

4 replies

ririsoft Feb 17, 2022
Author

Indeed this is a workaround, but I was wondering if there is actually an issue on Quarkus/Keycloak side so that 143 is returned instead of 0.

pedroigor Feb 17, 2022
Collaborator

We should not return 143 but 0 or 1. The server seems to be shut down by looking at the logs and the status should be 0.

Are you able to do the same thing using the legacy dist without the change you did?

tgurr Feb 17, 2022

I didn't try 17.0.0 legacy specifically but adding any SuccessExitStatus to the systemd unit file at least wasn't necessary with 16.1.1 as far as I can tell.

pedroigor Feb 17, 2022
Collaborator

Thanks. Created an issue to investigate the behavior.

darius-m · 2022-02-21T09:29:30Z

darius-m
Feb 21, 2022

It may also be connected to this issue. For WildFly, the start script contains a loop that tries to kill Keycloak, but does not properly check if it is still running.

0 replies

pedroigor · 2022-02-21T11:01:07Z

pedroigor
Feb 21, 2022
Collaborator

Looks like the JVM default is to return an exit code of 128+signal-id. E.g for SIGTERM you'd see an exit code of 143 (128+15).

We can check if changing our script solves the issue (assuming Wildfly is working fine with systemd) but I'm wondering if we can document this behavior and provide a guide that shows how to make Keycloak a systemd service.

Would that be enough? I don't see any issue using the SystemExitStatus to set the expected exit code(s).

17 replies

soundsessential Feb 21, 2022

Arrgh, scratch that. It appears to work but on clicking on a Realm and then Roles (and some other buttons, but not all) you get a blank menu and the page doesn't go where you want it to. You end up on https://keycloak.mydomain.com:8443/admin/master/console/#/notfound

I think, until there's further guidance from folks way more smart than me, I'll go with launching with !.

@darius-m Thank you, sincerely, for taking the time. Much appreciated.

darius-m Feb 21, 2022

I think that the best approach would be using a dedicated proxy that can drop privileges. That way, even if the server is partially compromised, extracting the keys from memory or open file descriptors would be a bit harder compared to accessing the keys in the filesystem. Ideally, Keycloak would also be able to do this when certificates are concerned.

However, considering that Keycloak is most of the time a critical service, if someone manages to compromise the service and gain access on the server as the keycloak user, where they can access user information or change the server's configuration, a compromised Let's Encrypt key may not be the largest problem.

On a side note, I recommend that you use a similar approach to other files used by Keycloak, where other users cannot access any of the files. All commands that concern Keycloak should be run as the keycloak user, regardless of purpose (both administration or SSO web service). Granting the service root privileges is not usually recommended; allowing other users read / execute permissions on the files is not necessary.

I personally prefer using file ACLs to make sure new files / directories are not created (by default) with permissions for others, but this makes inspecting permissions quite a bit harder, since ACLs are not displayed by ls. If you want to do this, you can run the following commands (two commands are required, since the default permissions are not set immediately). Directories and files that have ACLs set have the last character of the permissions column (that is normally .) changed to +.

setfacl -R -d -m o::- /opt/keycloak
setfacl -R -m o::- /opt/keycloak # this is basically the same as "chmod -R o=- /opt/keycloak"

ACLs can also be used to grant only the keycloak user / group additional permissions on the certificate files, but I strongly believe that this would make permission checking a lot harder, so I cannot recommend it.

Your last error is probably an issue with some other Keycloak files. If the service starts and you can connect to it, the certificates can be accessed, so there should not be any issues there. Please make sure the keycloak user can properly access all files under the /opt/keycloak directory (from the commands you mentioned above, it should be correct, but there must be an error since you cannot access all components). Also check if you find anything in the logs - you should be able to use journalctl -xe -u keycloak or check the logs directory.

soundsessential Feb 21, 2022

Ha ha, that's all so over my head but I again appreciate the effort. So I checked the logs after clicking around a bit and making it fail. This is in case maybe it's a bug but I imagine it's my keycloak user, although I did give it ownership as above:

Feb 21 18:30:19 keycloak17-ubuntu kc.sh[940]: 2022-02-21 18:30:19,769 WARN [org.keycloak.encoding.ResourceEncodingProvider] (executor-thread-0) Failed to encode resource: java.io.FileNotFoundException: /opt/keycloak/keycloak-17.0.0/bin/../data/tmp/kc-gzip-cache/9oi5d/admin/keycloak/partials/notfound.html.gz (Permission denied)

I should probably build this from scratch again to rule out any conflicts I've introduced.

soundsessential Feb 21, 2022

Yeah so just to confirm, with a clean fresh install, applying permissions as above to the lets encrypt folders again results in that error. It seems to be consistently happening when on the master realm > realm settings, you can get to the first 5 tabs (general thro' themes) but from localizations onwards it results in the error and the blank screenshot.

I'll leave this with people with more knowledge than myself! I'll have to go with my elevated privileges workaround for now but if anyone finds a solution to allow the keycloak user to access the lets encrypt certs without these issues (or if it's nothing to do with the certs and has a solution to the error), please do let me know. I'd love to push Keycloak in our community but absolutely want to do it in the best possible way. I'll post a link here to the blog post when it's live in case anyone wants to follow it to see what might be up. Thanks all!

soundsessential Feb 22, 2022

Our tutorial is available. As the work on the systemd unit file moves forward, if anyone has any ideas on why the error above may be happening on a user with ownership of the keycloak folder and permissions to the Let's Encrypt folder (although this is not applied within the tutorial as yet), please do let us know. We're eager to get this updated to not using elevated privileges. Thanks all.
https://www.soundsessential.com/blog/214-keycloak-17-filemaker-installation-configuration-tutorial-part-3-preparing-keycloak-17
https://www.soundsessential.com/blog/215-keycloak-17-filemaker-installation-configuration-tutorial-part-4-starting-keycloak-17-next-steps

pedroigor · 2022-03-08T15:19:19Z

pedroigor
Mar 8, 2022
Collaborator

Sorry for the late reply. Looks like we have quite a few material and information to provide instructions on how to run using systemd.

That said, I would like to know from you what are the next steps to make solve this discussion/issue.

Looks like we can use this article https://www.soundsessential.com/blog/215-keycloak-17-filemaker-installation-configuration-tutorial-part-4-starting-keycloak-17-next-steps as the basis for a new guide in our website.

The remaining issue though is related to figuring out how to avoid running the service with elevated privileges.

Does that make sense?

8000

0 replies

tgurr · 2022-04-04T10:19:35Z

tgurr
Apr 4, 2022

If it's of interest this is the systemd file we currently use for keycloak >= 17:

[Unit]
Description=Keycloak server
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/bin/kc.sh start --auto-build
WorkingDirectory=/opt/keycloak
ReadWritePaths=/opt/keycloak/conf /opt/keycloak/data /opt/keycloak/lib/quarkus
SuccessExitStatus=0 143

# Hardening options
CapabilityBoundingSet=
AmbientCapabilities=
NoNewPrivileges=true
ProtectHome=true
ProtectSystem=strict
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
PrivateTmp=true
PrivateDevices=true
LockPersonality=true

[Install]
WantedBy=multi-user.target

5 replies

soundsessential Apr 4, 2022

Are you using SSL certs and if so, where do you save these to? I seem to have had trouble with having Lets Encrypt certs save to their default location and then have a non root user get access to them.

I think I may have had issues with some of the data paths too, which your ReadWritePaths entry may resolve. I will test this. Thanks for sharing.

tgurr Apr 4, 2022

Are you using SSL certs and if so, where do you save these to?

I'm using nginx as a reverse proxy for that.

/opt/keycloak/conf/keycloak.conf

# Listen IP, default: 0.0.0.0
http-host=127.0.0.1

# HTTP listener, default: false
http-enabled=true

# The proxy address forwarding mode if the server is behind a reverse proxy.
proxy=edge

# Hostname for the Keycloak server.
hostname=keycloak.domain.com

/etc/nginx/nginx.conf

    server {
        listen 80;
        listen [::]:80;
        server_name keycloak.domain.com;
        return 301 https://$server_name$request_uri;

        access_log off;
        error_log off;
    }

    server {
        listen 443 http2 ssl;
        listen [::]:443 http2 ssl;
        server_name keycloak.domain.com;

        # SSL security configuration
        include /etc/nginx/ssl_security.conf;

        access_log off;
        error_log off;

        # restrict access to master realm
        location /realms/master/ {
            proxy_pass          http://localhost:8080/realms/master/;
            proxy_set_header    Host               $host;
            proxy_set_header    X-Real-IP          $remote_addr;
            proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Host   $host;
            proxy_set_header    X-Forwarded-Server $host;
            proxy_set_header    X-Forwarded-Port   $server_port;
            proxy_set_header    X-Forwarded-Proto  $scheme;
            allow 10.xxx.xxx.xxx;
            allow 10.xxx.xxx.xxx/24;
            deny all;
        }

        location / {
            proxy_pass          http://localhost:8080/;
            proxy_set_header    Host               $host;
            proxy_set_header    X-Real-IP          $remote_addr;
            proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header    X-Forwarded-Host   $host;
            proxy_set_header    X-Forwarded-Server $host;
            proxy_set_header    X-Forwarded-Port   $server_port;
            proxy_set_header    X-Forwarded-Proto  $scheme;
        }

/etc/nginx/ssl_security.conf

ssl_protocols TLSv1.3 TLSv1.2;
ssl_session_cache shared:SSL:10m;
ssl_ciphers HIGH:!aNULL:!MD5:!3DES:!CAMELLIA:!AES128:!AES256-SHA:!AES256-SHA256:!AES256-GCM-SHA384:!AES256-CCM8:!AES256-CCM;
ssl_prefer_server_ciphers on;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
add_header Content-Security-Policy "upgrade-insecure-requests" always;
ssl_certificate /etc/ssl/domain.com/domain.com.crt;
ssl_certificate_key /etc/ssl/domain.com/domain.com.key;
ssl_stapling on;

soundsessential Apr 4, 2022

Ah gotcha, that's great info too. Thanks.

psytester Apr 7, 2022

Possible additional extension:
Because the console output is spamming (depending on log level) the /var/log/messages and also journalctl -xe, I redirect stdout & stderr to a file:
ExecStart=/opt/keycloak/bin/kc.sh start --auto-build 2>&1 >> /opt/logfiles/console.log

SpiderD555 Nov 25, 2022

Are you using SSL certs and if so, where do you save these to? I seem to have had trouble with having Lets Encrypt certs save to their default location and then have a non root user get access to them.

Thanks for diving into this topic, it seems I have encountered the same problem as you.
I have also created my own workaround (which is basically setting correct file permissions, so keycloak can read certificate and key files).
I hope it will help you (or someone else reading this topic).

systemd graceful shutdown #10180

Uh oh!

Replies: 5 comments · 26 replies

Uh oh!

Uh oh!

ririsoft Feb 17, 2022 Author

Uh oh!

pedroigor Feb 17, 2022 Collaborator

Uh oh!

Uh oh!

pedroigor Feb 17, 2022 Collaborator

Uh oh!

Uh oh!

pedroigor Feb 21, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pedroigor Mar 8, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 5 comments 26 replies

ririsoft Feb 17, 2022
Author

pedroigor Feb 17, 2022
Collaborator

pedroigor Feb 17, 2022
Collaborator

pedroigor
Feb 21, 2022
Collaborator

pedroigor
Mar 8, 2022
Collaborator