Repository navigation
Replies: 5 comments 9 replies
|
thanks 👍 Case 1: Disk full while server is runningStarted: UP (server is there in general) Case2: No Cache connectionStarted: UP (server is there in general) |
|
Before giving practical examples, I would like to discuss some key principles first. Sorry if it was not clear from the description. I don't quite understand your assumptions, I appreciate it if you can elaborate a bit more. The requirements should be driven by the type of probe and not really by individual health checks. These probes should indicate the state of the server and indicate:
These are the probes we currently support but without much details about what type of checks should go in each one. For instance, today we have a single database health check that is marked as a readiness check. Some people might say this check does not make sense there, but I think it might be because you might want to remove individual instances from serving requests to alleviate the database load based on the metrics from the individual connection pools for each instance. Of course, considering the shared dependency (the database) is alive but with limitations to serve all server instances. If it makes sense as a default health check, I don't know. And the proposal is to not make any assumptions but let users register health checks accordingly to any health group. Your Case 1 is a perfect example of what I'm trying to figure out first:
The same goes for Case 2:
If this one is a readiness probe, for me it is fine to depend on downstream services if the checks/probes are implemented to not cause cascading failures. That is why I'm trying to focus on the overall design of the health endpoints and try to avoid default health checks on each probe but provide some default implementations that users can set to any type of probe. Which checks make sense to provide OOTB, that is an open discussion. I have proposed some, but they are not yet set in stone:
I hope we can enter into the details for each health check as soon as we agree about the overall design of the health endpoint and the constraints for each type of probe. |
|
For the startup probe / dedicated endpoint: Afaik, startup probes are intended to use the same probe as the liveness probe, just using a different |
|
I think it does. As per the link you mentioned, that approach is more likely to be used for certain types of applications. Let's consider our case:
I would say that keeping it aligned with k8s probes is probably the best so that we expose it to separate endpoints. It is also better to avoid specific requirements for a specific type of probe to impact the other. Or do you have something else in mind? |
|
No, makes sense what you say. it's perhaps just a bit unusual from my experience :) |
|
@pedroigor Where would things like loading the offline_tokens into the memory which can take some time will be counted at startup or readiness ? |
|
IIRC, loading offline_tokens (what is now disabled by default) should block the startup. Considering this, I would say it is a startup check. |
|
@thomasdarimont W.r.t. to the SPI. I noticed you have this https://github.com/thomasdarimont/keycloak-health-checks/tree/master/src/main/java/com/github/thomasdarimont/keycloak/healthchecker/spi. I'm wondering if we could discuss it and see if we can just use it. Wdyt? |
|
@pedroigor sure! I just returned from vacation. I'll read through the discussion and get back to you asap. |
|
Some remarks:
Perhaps we don't need this for the first iteration, but we should consider this in the design from the beginning. In my healthchecker SPI some health checks do only make sense on realm level (e.g. LDAP, Email) where others (database, infinispan, filesystem) are more universal. However, in the case of LDAP with multiple configured federations, there might be some providers which are more critical than others. Here we'd need a way to express whether a provider / component should be considered for health checks. We could provide different health APIs for checking Server Health vs. Realm Health. Server / Instance Level Health Checks
Realm level Health ChecksThe If we had realm level checks and multiple realms would that require Kubernetes to perform multiple health-checks? There could also be a realm setting that toggles whether or not a realm health endpoint is exposed or not, custom realm level health checks could then be exposed via a dedicated endpoint like |
|
@pedroigor I played around a bit and ported a small part of the health-checker SPI to the current Keycloak codebase: In this branch I have an example for a custom server level health check For implementing realm level health checks I sketched an SPI that enables users to provide custom health indicators that can be conditionally enabled per realm. Some examples for using this SPI are In the PoC the realm level health checks can be invoked via the new Having realm level healthchecks implemented like this allows for all sorts of additional operations on health checks, like explicitly executing a particular healthcheck, supply parameters to a health checks, secure the health check with realm level authorization etc. The current implementation yields a response like this (follows the microprofile health wireformat): {
"state": "UP",
"checks": [
{
"state": "UP",
"data": {
"host": "localhost"
},
"name": "email"
},
{
"state": "UP",
"data": {
"server": "server1"
},
"name": "ldap"
},
{
"state": "UP",
"data": {
"server": "server2"
},
"name": "ldap"
}
]
} |
+1. I'm quite comfortable with readiness checks and what we can potentially provide to indicate if an instance should be removed from the backend pool due to being loaded or even to give it some time to recover (without necessarily a restart) from a broken state. Mainly the load scenario, TBH. However, liveness checks are not clear to me and I think we need more feedback from people running in Kubernetes.
+1. Your work is a good starting point and if we reach an agreement here we can move discussions around it to a design document.
It should be a capability provided by the SPI and its implementation and probably with some level of configuration. I would also focus on passive checks as mentioned in the proposal so that even without a cache mechanism, checks are still cheap. For active checks, I think a mix of caching or some sort of circuit break could also help.
I think the work @Alexander72 is doing will help us to find those checks. But for now, I would rather focus on the checks that makes sense to provide OOTB. You mentioned some and some of them are implemented in your PoC. That should helps us to move forward without an agreement around default checks as well as work on the configuration part of it that should allow people to enable/disable checks accordingly. I'm also wondering how far individual checks like database, infinispan, etc, are useful. For instance, a health system should consider the user experience and how they are able to use the system. With that in mind, I can see checks based on the metrics around login failures and errors that can be collected during the server lifetime. Without necessarily check for individual components but use cases and how the system is used.
I'm still not sure if we should deal with that in our side but rely on the ingress/proxy configuration. I see this useful if you are exposing Keycloak directly and our focus are Kubernetes and running behind an ingress. In fact, that is what you should be doing anyways to achieve HA too.
Makes sense. Looks like we should focus first on the system-level checks. Per-realm checks should be also aligned with our SaaS story, I think. Per-realm checks probably deserve a separate discussion if we want to move forward with it. |
|
It would be worth splitting this into two separate discussions:
From a priority perspective 1 is the most important. Having well behaved probes is one of the corner stones of having a well behaved system on K8s. Around probes in general I think what @pedroigor has laid out in the initial description is going in the correct direction, and we should focus on the general design of it, then we can boil down to individual checks later. I also agree with the ability to make it configurable, but that should not be a must from a users perspective. Keycloak should out of the box have the right checks for the right probes, and not require users to understand or tweak this themselves. I'm not 100% convinced that we even need to support custom checks here, so would like some clear use-cases for that upfront. One open question to me is if having the probes as http endpoints on the default http listener is the correct way to go. For example if an individual node is flooded with requests, that could lead to liveness probe timing out, and K8s restarting the node, while in fact it should just have backed-of traffic to the node. |
|
@thomasdarimont Is there anyway to implement LDAP HealthCheck Endpoint as your current HealthCheck endpoint is not supported from 20.0.0 and when I checked with 22.0.2, there are number of changes w.r.t files from keycloak. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Introduction
This discussion is a follow-up to the Health Endpoint original proposal. Please, consider reading it first in order to understand better some of the proposals herein presented.
Some of the information herein documented is basically exposing the current status of health checks in Keycloak so that we can evaluate what can be improved.
Some key points that are still unclear and undocumented are:
liveness,readiness, orstartupcheckliveness,readiness, andstartupchecksFor now, I would like to avoid discussing implementation details but try to focus on the design, if possible.
The Health Protocol
Health checks are obtained from the
/healthendpoint.By querying the
/healthendpoint, the health checks available from allhealth groupsare serialized into a JavaScript Object Notation (JSON) with the following structure:More details about the health protocol can be read from the MicroProfile Health Protocol and Wireformat document.
Default Health Groups and Probes
Within the health endpoint, there are some key health check groups available by default. Each one is targeted for a specific health probe:
/health/started/health/readiness/health/livenessThe
startedhealth group should be used to probe if the server is fully started.The
readinesshealth group should be used to probe if the server is ready to serve requests.The
livenesshealth group should be used to probe if the server is alive and indicate whether or not the instance should be restarted.The default health groups are based on the Kubernetes Probes and the checks available from each health group depend on the constraints and requirements for each type of probe.
Probing the server startup
Startup probes should use the
/health/startedendpoint to check if the server is fully started. In general, this endpoint should not return any specific status check and the overall status should always beUP. If any failure happens when starting the server the endpoint is not even available and the server is going to exit with a status code indicating the error.A positive status from this endpoint indicates that:
Although the server does not perform any long-running operation after startup that could bring a negative overall status, this endpoint is still relevant to make sure the server is started and possibly support long-running operations that could impact its
startedstate.Probing the server readiness
Readiness probes should use the
/health/readinessendpoint to check if the server is ready to serve requests.By default, the server does not provide any readiness check because the overall status relies on the server being started. Once started, the assumption is that the main components are bootstrapped, fully initialized, and ready to serve requests.
However, it should be possible to register some built-in readiness checks if needed such as:
These checks should not be used for collecting metrics but to indicate if the server should stop serving traffic even if temporarily.
The server should not be opinionated about the checks available from this endpoint but allow users to combine them accordingly to their needs.
Probing the server liveness
Liveness probes should use the
/health/livenessendpoint to check if the server needs a restart after transitioning to a broken state. In general, this endpoint should not return any specific status check and the overall status should always beUPindicating that the server instance is still alive and reachable through its HTTP(s) interface.As a general rule, any check added to this endpoint should not depend on downstream services because a restart is very unlikely to solve the problem and possibly introduce cascading failures. The same goes for any other check that can not be solved by a server restart. For instance, causing restarts due to database failures might impact requests that can be served with the information from the cache and a restart is very unlikely to solve the problem. The same goes for cluster failures where restarts can cause even more problems.
The checks that should go into this endpoint are not clear as they are more related to environment-specific issues such as deadlocks, memory leaks, or disk failures.
It should be possible to register any available health check if needed.
@andreaTP Do you have an idea about what liveness checks make sense for the operator so that we can recover instances transitioned to a broken state?
Active and Passive Health Checks
Health checks can perform active or passive checks:
Active - Checks that periodically perform an operation in order to calculate a status
Passive - Checks that calculate the status based on metrics, failures, errors, and information about the traffic
We should be favoring passive checks as much as possible not only because they reduce the impact on the server runtime but also because they enable a variety of checks based on the metrics collected by the server.
One open question here is if it makes sense to have metrics disabled when health check is enabled.
User-defined Health Checks
It should be possible to register custom health checks and expose them through the health endpoint.
We should be providing an SPI to avoid depending on the stack (e.g.: Quarkus or Wildfly) that the server is using.
All reactions