Repository navigation
docs: Use BASH TCP redirect for HEALTHCHECK - #38131
Conversation
There was a problem hiding this comment.
This is an example of what I had in mind when I wrote "creative use of the Linux filesystem" in section 5.1.2 of the Server Configuration Guide. The syntax given matches my understanding of how this technique works
/dev/tcp is technically a built-in of Bash, not a real Linux filesystem object. So it requires bash to work, but there is no reason to not keep bash as our shell in our container images. Also, since this is not a proper HTTP client, it does depend on the tolerance of the Quarkus HTTP server to work. If this is added to the documentation without an integration test, then it could break without us knowing.
That said, it could be a useful example for those users running Keycloak with non-Kubernetes infrastructure
| { printf 'HEAD /health/ready HTTP/1.0\r\n\r\n' >&0; grep 'HTTP/1.0 200'; } 0<>/dev/tcp/localhost/9000 | ||
| ---- | ||
|
|
||
| The above code depends on the values of the {project_name} options `http-relative-path` (`http-management-relative-path`) and `http-management-port`. In case those are changed the code needs to be modified accordingly. |
There was a problem hiding this comment.
If we decide to put this in the docs, we should state that this would work only when HTTPS is disabled for the management port. AFAIK HTTPS is enabled on the management port once you add a TLS certificate for the HTTP traffic on the main port, which is recommended setup for production setups.
There was a problem hiding this comment.
Yeah, I was thinking that as well. There's a way around that by installing socat, but at that point you might as well just install curl, as both will require openssl, which is the key package we exclude in the images.
There was a problem hiding this comment.
The command { printf 'HEAD /health/ready HTTP/1.0\r\n\r\n' >&0; grep 'HTTP/1.0 200'; } 0<>/dev/tcp/localhost/9000 assumes HTTP-only access to the management port. However, as noted, HTTPS is enabled when a TLS certificate is added, making this check unreliable for production setups. A fallback or alternative approach should be mentioned.
|
So we seem to be stalled on this one. The solution presented is simple, it has merit, but apparently only works in a non-production situation, so it's not that useful to include in the documentation. The real issue is that Docker still doesn't support external health checks? It's a flawed architecture to run a command inside a container to see if the container is working or not. I see in Java 22 you can run a java file without precompilation. Which would be ideal, because you could write a health check that uses NSS instead of openssl. However, Keycloak uses Java 21 right now. |
|
Hi @ASzc and everyone, I'm glad this initiated this much interest! Not being a security expert per-se - what's the problem with Except for it is an addition to the vanilla container itself, where I see the point of recommending To my defense (looking like a n00b recommending something not fit for production), I'm actually using keycloak like this in production without TLS - its own, to be precise, my |
|
OpenSSL is excluded because is not needed to run Keycloak. OpenJDK uses NSS as its security library instead. If OpenSSL is included in the container image, it will get picked up by CVE scanners and flagged as a problem multiple times per year. The actual exploitability of any OpenSSL CVE in a Keycloak container image is very questionable, but the only way to guarantee there is no relevance is to not include OpenSSL. This saves a lot of unnecessary work to re-release Keycloak container images with patched OpenSSL. |
|
Thanks for the clarification @ASzc. Given all the valid arguments, any TLS-supporting solution without building a custom container seems to be currently impossible to achieve. Knowing Java 22 can now interpret inlined code is likely also too brutal for a healthcheck commad :) I suppose to make this simple and versatile, someone would have to write a healthcheck client in whatever Java Keycloak uses now and make it a standalone executable shipped with the original container, but that's hardly meaningful to develop/manage, given how niche this use-case is. I think the last thing we/you can do is to
8000
add the line that this only works for plain HTTP and still recommend |
|
IMHO health checks need to be as lightweight as possible. Running it with a Java HTTP client might have perf implications, I wouldn't do it by default in the container. It'd still need to be something the user explicitly opts-in. So not sure there'd be much value in that. As for documenting the alternative command proposed in this PR. I'm not a shell expert, but if it doesn't work with TLS I don't see much reason to document it. Keycloak pushes for secure-by-default principles, TLS required by default is one of them. I understand the non-TLS scenarios are beneficial mostly in dev scenarios, however health checks are needed more in prod env where TLS can be expected. |
|
Thanks for your input @vmuzikar, the cat's out of the box anyways in the issues I filed (and obviously got noticed by some users). Sometimes reaching for information doesn't have to be easy or official. Let's close this. Thanks for all your attention, though! You seem like a great team, which is not to be taken for granted in the open-source world 🙇 ☮️ |
b7aa602 to
1ff3a15
Compare
There was a problem hiding this comment.
Seems agreeable to me in theory. Someone with docker installed should try the documented command. I just have podman and I don't trust the compatibility to be 100% on virtual filesystem items like this
Works on docker and locally on my linux system. |
|
Hi folks, do I see correctly that the management url no longer depends on |
It's back in there. I had initially thought it may require additional caveats due to the new settings, but decided that it was fine to leave it the way it was. |
There was a problem hiding this comment.
@shawkins and @vit-zikmund - see below for some review comments and questions.
Added to the discussion points and incorporated the suggested edits. |
|
I still had a little question @shawkins in #38131 (comment), it's the last one, I promise. Also based on your previous answer, wouldn't it be nice to make the text "If you enable TLS" a link to https://www.keycloak.org/server/enabletls? |
Add a BASH script to perform an in-container healtcheck. For the curious, here's how this works: 1. For the code within braces, a TCP connection is made to the keycloak's management port and a successful connection is redirected in the read-write fashion to the descriptor 0 (stdin). - When bash fails to connect (TCP RST), it ends up with an error right away. - When the connection is hanging (no reply till TCP retry timeout, usually about 1 minute), it just hangs, virtually being a subject to the HEALTHCHECK's timeout (which should be definitely smaller than the usual TCP retry timeout). 2. Then a simple hand-crafted HTTP HEAD request is sent to the socket using printf. This is supposed to always succeed, unless the send buffer of the socket is set ridiculously small on the target OS. In the other case it will just hang again, not being able to push all the bytes through, until that eventually happens or times out. 3. Next, the eventual response is being checked with grep to be the successful one. Only at this time it's return code (and the final) is 0. - When no response comes, it's hanging forever and is subject to timeout. - When a 503 response comes, grep doesn't match anything and returns 1. Closes: keycloak#38126 Signed-off-by: Vit Zikmund <vit.zikmund@themama.ai>
Signed-off-by: Steve Hawkins <shawkins@redhat.com>
Co-authored-by: Alexander Schwartz <alexander.schwartz@gmx.net> Signed-off-by: Steven Hawkins <shawkins@redhat.com>
Co-authored-by: Alexander Schwartz <alexander.schwartz@gmx.net> Signed-off-by: Steven Hawkins <shawkins@redhat.com>
Signed-off-by: Steven Hawkins <shawkins@redhat.com>
3902df5 to
391a750
Compare
There was a problem hiding this comment.
Approving (again). If there are any additional reviews necessary and the changes are minimal, I'm ok to have those reviewed with the CND team.
| === HEALTHCHECK | ||
|
|
||
| The Containerfile `+HEALTHCHECK+` instruction defines a command that will be periodically executed inside the container as it runs. The {project_name} container does not have any CLI HTTP clients installed. Consider installing `+curl+` as an additional RPM, as detailed by the <@links.server id="containers" /> {section}. Note that your container may be less secure because of this. | ||
| The Containerfile https://docs.docker.com/reference/dockerfile/#healthcheck[`+HEALTHCHECK+`] instruction defines a command that will be periodically executed inside the container as it runs. While the {project_name} container does not have any CLI HTTP clients installed, it is possible to leverage BASH support for redirects to TCP sockets and craft a simple HTTP request to the healthcheck endpoint: |
There was a problem hiding this comment.
Just one minor comment. I'm not sure about a link to Docker docs from the downstream perspective, I don't think we can include it there. We can either make it conditional to include it upstream only, or remove the link completely (as it was before). The latter seems more viable to me. @ahus1 WDYT?
There was a problem hiding this comment.
@vmuzikar: +1 for removing the link and only keeping the text HEALTHCHECK.
Add a BASH script to perform an in-container healtcheck.
For the curious, here's how this works:
Closes: #38126