Repository navigation
Keycloak does not work on systems with FIPS mode enabled #9916
Description
Describe the bug
When system wide crypto policy is set to FIPS there are issues with running Keycloak. This has recently surfaced as RHEL, Fedora, and CentOS has changed the way OpenJDK behaves in this regard. Previously FIPS had to be explicitly enabled for OpenJDK, but is now enabled by default when system-wide policies are configured to conform to FIPS 140-2.
With OpenJDK configured with FIPS conformance there are a number of things not working properly in Keycloak, which is somewhat to be expected as Keycloak has never had official FIPS support.
This issue is limited to ability to run Keycloak on a FIPS enabled system, and not to support FIPS compliance in Keycloak. For FIPS 140-2 support within Keycloak see #9967.
Original JIRA: https://issues.redhat.com/browse/KEYCLOAK-19771
Version
All versions of Keycloak are affected and the issue surfaces rather based on the host system configuration/upgrades.
Container images published to Quay the issue happens in >=15.1.0, but does not happen in <=15.0.2. This is not caused by an issue in Keycloak, but rather the fact that the images <=15.0.2 are not updated with the latest upgrades from RHEL.
Expected behavior
Keycloak is able to run on a FIPS enabled system.
Actual behavior
add-user-keycloak script fails, and Keycloak fails to generate initial certificates. There are probably also further issues, but starting a Keycloak on a FIPS enabled system after the database has been initial at least succeeds.
How to Reproduce?
Enable FIPS mode on a RHEL or Fedora host (fips-mode-setup --enable).
Start Keycloak directly on the host, and you will get the following error:
Error during startup: org.keycloak.component.ComponentValidationException: Failed to generate keys
Run bin/add-user-keycloak.sh -u admin and you will get the following error:
java.lang.RuntimeException: PBKDF2 algorithm not found
Starting Keycloak as a container with -e KEYCLOAK_USER=user -e KEYCLOAK_PASSWORD=pass results in the PBKDF2 algorithm not found error, while interestingly enough not setting these environment variables the container starts successfully, which is different behaviour to running Keycloak from a ZIP where the failed to generate keys error occurs at startup.
Workaround
Until FIPS is supported by Keycloak, FIPS has to be disabled for Java to make it possible to run Keycloak on systems with FIPS mode enabled.
For bear-metal/ZIP installations use the JAVA_OPTS_APPEND to set -Dcom.redhat.fips=false. For example:
export JAVA_OPTS=-Dcom.redhat.fips=false
bin/kc.sh start-dev
For containers use the JAVA_OPTS_APPEND to set -Dcom.redhat.fips=false. For example:
podman run -p 8080:8080 -e JAVA_OPTS_APPEND="-Dcom.redhat.fips=false" -e KEYCLOAK_ADMIN=<user> -e KEYCLOAK_ADMIN_PASSWORD=<pass> quay.io/keycloak/keycloak
Activity
rush2sk8 commentedon Dec 13, 2021
I too am having this issue. I am running keycloak in a kubernetes environment and it immediately fails and restarts with "java.lang.RuntimeException: PBKDF2 algorithm not found"
wjorda commentedon Dec 23, 2021
This is preventing us from upgrading from 15.0.2 to 16.0.0 running in Kubernetes.
Given that there is now a CVE (https://access.redhat.com/security/cve/CVE-2021-4133) related to 15.1.1 and below, this issue is of critical importance.
rush2sk8 commentedon Dec 30, 2021
I had a temporary fix by just using 15.0.1 for my base keycloak image.
stianst commentedon Jan 3, 2022
Not sure what's going on here, but I'm not able to reproduce this issue. Can someone provide steps to reproduce this?
Etherchannel01 commentedon Jan 3, 2022
Would like to add in that I am experiencing the same issue on Openshift when using the operator for version 16.0.0. I will deploy a new keycloak instance with the operator, postgres will come up fine, keycloak pod will attempt to start but instantly crashes with the error "java.lang.RuntimeException: PBKDF2 algorithm not found"
wjorda commentedon Jan 3, 2022
Using the command from the Docker guide:
This appears to affect all container versions newer than 15.0.1. Looks like a problem with the generate users script based on the research done above and my own personal experience. Omitting the environment variables from the command above allowed my cluster to start up.
wjorda commentedon Jan 4, 2022
Update: I tested on a non-FIPS system and it worked as normal.
Per Red Hat Support (https://access.redhat.com/solutions/6484731) it looks like the issue is related to FIPS being enabled on the host's kernel. We are running Docker on a hardened CentOS 8 with FIPS enabled.
Unfortunately disabling FIPS (even temporarily) is not an option for us, so that will make implementing the documented workarounds difficult.
wjorda commentedon Jan 5, 2022
Running on the affected host with
com.redhat.fips=falsein JAVA_OPTS (passed in via the containers environment variables) to disable FIPS mode in the JVM also works.We have been running with FIPS enabled since v10 without major issues until the 15.0.2 release. We use LDAPS for user storage so the add-keycloak-user.sh script typically only gets called at startup and further user authentication is handled by our directory server. I am not an expert on this but my understanding is that enabling FIPS in the host's kernel causes the JVM in the container to run under FIPS mode by default as well (https://access.redhat.com/articles/5895481).
rx294 commentedon Jan 7, 2022
@stianst the RHEL UBI8 based images we use inherits host FIPS mode and enforces it within containers.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening (Section 5.4)
stianst commentedon Jan 7, 2022
Thanks, should be possible to easily reproduce this issue then by enabling FIPS in a container. Got no clue why PBKDF2 wouldn't be available though as FIPS doesn't cover password hashing algorithms.
rush2sk8 commentedon Jan 10, 2022
We are also running 15.0.2 in a container in a hardened FIPS enabled Centos 7 machine and can confirm this is the issue.
rwd5213 commentedon Jan 12, 2022
There is some related errors when running keytool when setting up TLS such as putting certs in
/etc/x509/https. Saw two different keytool errors:keytool error: java.security.KeyStoreException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_SESSION_READ_ONLYkeytool error: java.io.IOException: parseAlgParameters failed: PBE AlgorithmParameters not availableadding that same flag
-J-Dcom.redhat.fips=falseto the keytool commands in /opt/jboss/tools/x509.sh script mitigates the errors.Saw this in 16.1.0 on a Centos 7 FIPS enabled instance.
29 remaining items
Disable FIPS for adding keycloak user
Disable FIPS for adding keycloak user
Internal Keycloak setup for Ovirt Engine SSO (#10)
archoversight commentedon Jun 10, 2022
The work-around of setting
JAVA_OPTS_APPENDdoesn't work on the18.0.0-legacycontainer as the flag is never used when executing the/opt/jboss/keycloak/bin/add-user-keycloak.shscript.archoversight commentedon Jun 10, 2022
Adding
-storetype jksto the key tool command invocation also fixes this issue.trixpan commentedon Aug 13, 2022
hello there @JPercivall :-)
skyllarr commentedon Sep 29, 2022
I was able to fix this error
keytool error: java.security.KeyStoreException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_SESSION_READ_ONLYby running the keytool command withsudomposolda commentedon Oct 18, 2022
I am resolving this issue as Keycloak 20 distribution will contain initial support for FIPS . The details are in this README: https://github.com/keycloak/keycloak/blob/main/docs/fips.md .
There are still some limitations and it is possible that some functionalities won't work as expected (in the testsuite, we see errors related to SAML, X509 authentication and Kerberos/SPNEGO authentication). As the FIPS support is still work in progress.
For the details around FIPS work, you can take a look at the project: FIPS 140-2 (view)
The feedback is welcome. Feel free to create new GH issue if you find further issues in the Keycloak 20 release related to FIPS support (not yet released, but will be probably by the end of October). When you create such GH issue, feel free to comment here with the reference to new issue and feel free to tag me directly in this GH comment (or in the new GH issue).
isaacjensen commentedon Nov 3, 2022
Hey @mposolda, the Trellix team would like some credit for contributions made. Are these listed anywhere? Do we plan to have contributors called out for this FIPS effort?