Keycloak new Management Interface port breaks in GCE Ingress + Operator #36323

mike-pt · 2025-01-07T00:21:50Z

mike-pt
Jan 7, 2025

Before reporting an issue

I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

operator

Describe the bug

I'm not sure if this is a bug as it seems intended but ended up breaking compatibility with GKE setups (probably the later).

While upgrading to v25 (tried 26 too) with the operator ingress is now failing the health checks, I noticed the was a change in the port.

I can see the service now exposing both port 8080 and 9000 and so the operator does create the service with new port.

I am using a custom ingress because with the "ingress" enabled in the operator I bump into other issues in GKE, it seems mostly tailored for openshift, also with GCE ingress we get free managed certificates.

The issue is GCE insists in creating a default healthcheck

checkIntervalSec: 15
creationTimestamp: '2025-01-06T15:37:21.226-08:00'
description: Default kubernetes L7 Loadbalancing health check for NEG.
healthyThreshold: 1
id: 'xxxxxx'
kind: compute#healthCheck
logConfig:
  enable: true
name: k8s1-xxxxx-keycloak-alpha-keycloak-service-8080-xxxx
selfLink: https://www.googleapis.com/compute/v1/projects/alpha/global/healthChecks/k8s1-xxxxx-keycloak-alpha-keycloak-service-8080-xxxxx
tcpHealthCheck:
  portSpecification: USE_SERVING_PORT
  proxyHeader: NONE
timeoutSec: 15
type: TCP
unhealthyThreshold: 2

I tried to add custom probes:

unsupported:
  podTemplate:
    spec:
      containers:
         -
          livenessProbe:
            httpGet:
              path: /health/live
              port: 9000
              scheme: HTTP
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 9000
              scheme: HTTP
          startupProbe:
            httpGet:
              path: /health/started
              port: 9000
              scheme: HTTP
            initialDelaySeconds: 60   # Increase to allow Keycloak startup
            periodSeconds: 10         # Reduce probe frequency
            timeoutSeconds: 5         # Allow slow responses
            failureThreshold: 30      # Wait up to 5 minutes before failing

Unfortunately this seems to be ignored by GCE ingress (though I see the probs listed in the pods)

From the docs google either uses BackendConfig or Readiness probe.

https://cloud.google.com/kubernetes-engine/docs/troubleshooting/ingress-health-checks

This was working fine before and though I understand the security issue with other endpoints I'm not to worried about health endpoints.... I don't see much wrong with those being public tbh and they are useful for external health-checks/ping from monitor tools. Is there a way to still have those endpoints in port 8080? Or is this all or nothing? Or is there some other way to set the readinessProbe? I seems google only cares for it if its on the same port.

Version

25 and 26

Regression

The issue is a regression

Expected behavior

Operator Deploy would work fine with GCE ingress after upgrade

Actual behavior

After upgrade ingress is not healthy and the service is not exposed.

How to Reproduce?

GCP and custom ingress resource (disable ingress in operator)

Sample ingress yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: -keycloak-ingress
  namespace: keycloak
  annotations:
    kubernetes.io/ingress.global-static-ip-name: kc-gw-lb
    networking.gke.io/managed-certificates: "${CERT_NAME}"
    networking.gke.io/v1beta1.FrontendConfig: keycloak-ingress
    kubernetes.io/ingress.className: "gce"
spec:
  rules:
  - host: ${HOSTNAME}
    http:
      paths:
      # - path: /auth
      - pathType: ImplementationSpecific
        backend:
          service:
            name: ${ENV_SUBSYS}-keycloak-service
            port:
              number: 8080
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: CERT_NAME
  namespace: keycloak
spec:
  domains:
    - $HOSTNAME

Anything else?

No response

shawkins · 2025-01-08T17:03:25Z

shawkins
Jan 8, 2025
Collaborator

Some things may need clarified. You didn't say what your starting version was.

Keep in mind that the result of the pod probes are used to determine the pod health and whether the pod should be reachable from relevant services. Kube will test those on the pod without going through the ingress. Are you seeing the pod being terminated or marked as not ready? If so, that would indicate a problem with the probes / health check endpoints. In any case there's probably no need to use custom probes.

If the pod is ready, but traffic is not reaching it from your gce ingress, then that seems related to the whatever additional check is being being performed by "Default kubernetes L7 Loadbalancing health check for NEG." From https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#remarks it looks like the default behavior is to send a request to / - which should make this effectively unrelated to the built-in health endpoints. So you need to point that to or expose something on the server that will respond with 200 - that could be the health endpoints, or it could just be something like /welcome-content/logo.svg

I'm not sure what version this would have worked in by default - all recent versions will redirect from /.

1 reply

mike-pt Jan 9, 2025
Author

You will notice /auth is commented, we do use /auth though as historically that's was how it worked so we set the base path via env but that's not the issue here anyway.

The issue is the management port is now separate, which form release notes was introduced in v26.

Because of this ingress will not work with the built in operator service.

I did find a work around by creating a custom service resource and setting the backend config manually too.

The downside is the regular service is still there.

Would be nice if either the service config was possible to costumize via keycloak operator resource or at least having a service.enable:false like ingress.

I will post here later the k8s resources for service and backend config etc.

sandeepsudheendran · 2026-08-05T06:01:21Z

sandeepsudheendran
Aug 5, 2026

Hi @mike-pt I'm running into what appears to be the same issue while upgrading Keycloak (using a GKE Ingress with an external HTTP(S) Load Balancer).

After upgrading, the GCE/GKE backend becomes UNHEALTHY, even though the Keycloak pods are running. The ingress annotations show the backend as unhealthy, and it seems related to the management interface moving the health endpoints to port 9000, while the GKE Ingress health check is still targeting the serving port (8080).

I'm curious if this issue was ever resolved.

Was there an official fix in a newer Keycloak Operator or Keycloak release?
If not, what workaround are you using?
Did you solve it with a BackendConfig custom health check, by exposing the management port, or by configuring Keycloak to serve health endpoints on port 8080 again?

Any guidance or update would be greatly appreciated. Thanks!

2 replies

mike-pt Aug 5, 2026
Author

I've since moved to GKE gateway instead of ingress; Ingress is deprecated and GKE Gateway is actually much more versatible, plus the Certificate Manager Integration in GCP is awesome.

Here is my sample httproute.yaml:

 cat httproute.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: ${ENV}-keycloak-route
  namespace: ${KEYCLOAK_NAMESPACE}
spec:
  parentRefs:
  - name: ${GATEWAY_NAME}
    namespace: ${GATEWAY_NAMESPACE}
  hostnames:
  - "${HOSTNAME}"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: ${ENV}-keycloak-service
      port: 8080
---
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: ${ENV}-keycloak-health
  namespace: ${KEYCLOAK_NAMESPACE}
spec:
  default:
    checkIntervalSec: 10
    timeoutSec: 5
    config:
      type: HTTP
      httpHealthCheck:
        # Management interface (port 9000) is at root via KC_HTTP_MANAGEMENT_RELATIVE_PATH=/
        port: 9000
        requestPath: /health
  targetRef:
    group: ""
    kind: Service
    name: ${ENV}-keycloak-service

In your KC resource make sure to set these to:

  proxy:
   headers: xforwarded
 ingress:
   enabled: false

I use envsubst and prefix things with env, you don't really need too, but as you can see I can target the operator service and still define a different path/port for healtheck endpoint. Defining a health check is in fact mandatory (I keep the HC on same yaml but you can split).

Of course this means you need to create a gateway but its actually not that complicated, see https://docs.cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways

shawkins Aug 5, 2026
Collaborator

I thought we had an issue already on using the gateway api directly from the operator, but couldn't find one. So there is now #51461

The workaround of creating your own gateway is preferable to using options like http-management-health-enabled or legacy-observability-interface to keep the health endpoints on the main interface.

Navigation Menu

Repository navigation

Keycloak new Management Interface port breaks in GCE Ingress + Operator #36323

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Keycloak new Management Interface port breaks in GCE Ingress + Operator #36323

Uh oh!

Uh oh!

mike-pt Jan 7, 2025

Before reporting an issue

Area

Describe the bug

Version

Regression

Expected behavior

Actual behavior

How to Reproduce?

Anything else?

Replies: 2 comments · 3 replies

Uh oh!

shawkins Jan 8, 2025 Collaborator

Uh oh!

mike-pt Jan 9, 2025 Author

Uh oh!

sandeepsudheendran Aug 5, 2026

Uh oh!

Uh oh!

mike-pt Aug 5, 2026 Author

Uh oh!

shawkins Aug 5, 2026 Collaborator

mike-pt
Jan 7, 2025

Replies: 2 comments 3 replies

shawkins
Jan 8, 2025
Collaborator

mike-pt Jan 9, 2025
Author

sandeepsudheendran
Aug 5, 2026

mike-pt Aug 5, 2026
Author

shawkins Aug 5, 2026
Collaborator