8000 [Keycloak-Operator]: Rolling Updates -- Strategy=Auto, operator error keycloak-update-job is invalid -- Strategy=Explicit, operator always replaces · Issue #40360 · keycloak/keycloak · GitHub
[go: up one dir, main page]

Skip to content

[Keycloak-Operator]: Rolling Updates -- Strategy=Auto, operator error keycloak-update-job is invalid -- Strategy=Explicit, operator always replaces #40360

Closed

Description

@kneufeld-pbp

Before reporting an issue

  • I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

operator

Describe the bug

On the latest v26.2.5 of keycloak-k8s-resources, operator and RedHat Build of Keycloak Server. Keycloak-Operator rolling updates are not working as expected. When spec.update.strategy=auto, the operator fails with error: keycloak-update-job is invalid, and when spec.update.strategy=Explicit, the operator will always replace the pod; rolling update is not observed.

Version

26.2.5

Regression

  • The issue is a regression

Expected behavior

  1. When Keycloak's CRD has spec.update.strategy=auto, the operator should automatically determine the best strategy to deploy a new update of the Keycloak server, and default to RecreateOnImageChange if the checks fail.
  2. When Keycloak's CRD has spec.update.strategy=Explicit, the operator should deploy a new update of the Keycloak server by rolling deployment, when spec.update.revision matches the previous update.

Actual behavior

When Keycloak's CRD has spec.update.strategy=auto:

Operator throw the following error:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.20.0.1:443/apis/batch/v1/namespaces/identity/jobs/keycloak-update-job?fieldManager=keycloakcontroller&force=true. Message: Job.batch "keycloak-update-job" is invalid: │
│  spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.template.spec.initContainers[0].lifecycle, mess │
│ age=Forbidden: may not be set for init containers without restartPolicy=Always, reason=FieldValueForbidden, additionalProperties={})], group=batch, kind=Job, name=keycloak-update-job, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Job.batch │
│  "keycloak-update-job" is invalid: spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalPropertie │
│ s={}), reason=Invalid, status=Failure, additionalProperties={}).

When the error is thrown, the operator will not deploy a new version of the Keycloak Server Pod until spec.update is removed or the strategy is changed.

When Keycloak's CRD has spec.update.strategy=Explicit

The operator will always deploy a new Keycloak Server Pod by replacing it - no rolling update is ever observed.

How to Reproduce?

Deploy keycloak-k8s-resources: v26.2.5 as usual and start updating the Keycloak Server image property with new tags.

Anything else?

System Details:
keyclaok-operator version: v26.2.5
RedHat Build of Keycloak: v26.2.5
keycloak-k8s-resources: v26.2.5
AWS EKS Automode - Kubernetes: v1.32

Activity

  1. shawkins commented on Jun 10, 2025

    @shawkins
    Contributor

    The operator will always deploy a new Keycloak Server Pod by replacing it - no rolling update is ever observed.

    Can you clarify this? The expected behavior is simply that the statefulset is updated, then Kubernetes handles the rollout - which will seqentially replace the pods. How many keycloak instances are set in the CR? What do you mean by replacing?

  2. shawkins commented on Jun 10, 2025

    @shawkins
    Contributor

    When the error is thrown, the operator will not deploy a new version of the Keycloak Server Pod until spec.update is removed or the strategy is changed.

    To clarify this situation - are you explcitly setting the lifecycle in the unsupported podTemplate or is that happening by some other mechanism?

  3. kneufeld-pbp commented on Jun 10, 2025

    @kneufeld-pbp
    Author

    To clarify this situation - are you explcitly setting the lifecycle in the unsupported podTemplate or is that happening by some other mechanism?

    No, not intentionally.

    Can you clarify this? The expected behavior is simply that the statefulset is updated, then Kubernetes handles the rollout - which will seqentially replace the pods. How many keycloak instances are set in the CR? What do you mean by replacing?

    We have been testing between 1 and 3 instances and are currently set to one.

    1. When the update strategy is set to Explicit, the pods terminated and replaced, with no rolling deployment, whether the instances are 1, 2, or 3.

    2. When the update strategy is set to Auto, the operator errors with

    Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://172.20.0.1:443/apis/batch/v1/namespaces/identity/jobs/keycloak-update-job?fieldManager=keycloakcontroller&force=true. Message: Job.batch "keycloak-update-job" is invalid: │
    │  spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.template.spec.initContainers[0].lifecycle, mess │
    │ age=Forbidden: may not be set for init containers without restartPolicy=Always, reason=FieldValueForbidden, additionalProperties={})], group=batch, kind=Job, name=keycloak-update-job, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Job.batch │
    │  "keycloak-update-job" is invalid: spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalPropertie │
    │ s={}), reason=Invalid, status=Failure, additionalProperties={}).

    And the StatefulSet does not appear to be updating., until strategy is changed or update is removed.

    This is my Keycloak file:

    apiVersion: k8s.keycloak.org/v2alpha1
    kind: Keycloak
    metadata:
      name: keycloak
      namespace: identity
    spec:
      update:
        strategy: Explicit
        revision: 26.2.5
      additionalOptions:
        - name: health-enabled
          value: "true"
        - name: metrics-enabled
          value: "true"
        - name: tracing-jdbc-enabled
          value: "false"
        - name: http-metrics-histograms-enabled
          value: "true"
        - name: cache-metrics-histograms-enabled
          value: "true"
        - name: event-metrics-user-enabled
          value: "true"
        - name: event-metrics-user-events
          value: login,logout
        - name: event-metrics-user-tags
          value: realm,idp,clientId
        - name: spi-theme-welcome-theme
          value: pbp-login
        - name: spi-theme-static-max-age
          value: "-1"
        - name: spi-theme-cache-themes
          value: "false"
        - name: spi-theme-cache-templates
          value: "false"
        - name: events-enabled
          value: "true"
        - name: events-listeners
          value: jboss-logging
        - name: events-expiration
          value: "3600"
        - name: log-level
          value: INFO,org.keycloak.events:DEBUG
        - name: db-driver
          value: software.amazon.jdbc.Driver
        - name: quarkus.http.limits.max-initial-line-length
          value: "8192"
        - name: spi-brute-force-protector-default-brute-force-detector-allow-concurrent-requests
          value: "true"
      db:
        passwordSecret:
          key: password
          name: aws-secret-magr-keycloak-db
        url: jdbc:aws-wrapper:postgresql://{{DATABASE_FQDN}}:5432//{{DATABASE_NAME}}
        usernameSecret:
          key: username
          name: aws-secret-magr-keycloak-db
        vendor: postgres
      features:
        enabled:
          - opentelemetry
      hostname:
        backchannelDynamic: false
        strict: false
      http:
        httpEnabled: true
        httpPort: 8080
      image: 01234567890.dkr.ecr.ca-central-1.amazonaws.com/rhbk-keycloak:38ffff36-1749497357
      ingress:
        enabled: false
      instances: 1
      proxy:
        headers: xforwarded
      scheduling:
        topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: keycloak
            maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
      startOptimized: false
      tracing:
        enabled: true
        endpoint: http://datadog-agent.datadog:4317
        resourceAttributes:
          additionalProperties: env=dev
        samplerRatio: 1
        samplerType: traceidratio
        serviceName: keycloak
      unsupported:
        podTemplate:
          metadata:
            annotations:
              ad.datadoghq.com/keycloak.check_names: |
                ["openmetrics"]
              ad.datadoghq.com/keycloak.init_configs: |
                [{}]
              ad.datadoghq.com/keycloak.instances: |
                [
                  {
                    "prometheus_url": "http://%%host%%:9000/metrics",
                    "namespace": "keycloak",
                    "send_distribution_buckets": true,
                    "tags":["keycloak-pod:%%host%%"],
                    "metrics": ["*"]
                  }
                ]
              ad.datadoghq.com/keycloak.logs: |
                '[{"source": "keycloak", "service": "keycloak", "tags": ["env:dev"]}]'
            labels:
              app.kubernetes.io/env: dev
              app.kubernetes.io/name: keycloak
              tags.datadoghq.com/env: dev
              tags.datadoghq.com/service: keycloak
              tags.datadoghq.com/source: keycloak
              tags.datadoghq.com/version: 38ffff36-1749497357
          spec:
            containers:
              - envFrom:
                  - configMapRef:
                      name: keycloak-additional-env
                name: keycloak
            terminationGracePeriodSeconds: 60

    This is the generated StatefulSet

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      annotations:
        javaoperatorsdk.io/previous: 4fdf768c-b7b5-4b76-8fd7-a936fd598412,5107452
        operator.keycloak.org/migrating: "false"
        operator.keycloak.org/missing-secrets: "false"
        operator.keycloak.org/recreate-update: "false"
        operator.keycloak.org/update-reason: Explicit strategy configured. Revision matches.
        operator.keycloak.org/update-revision: 26.2.5
        operator.keycloak.org/watching-secrets: keycloak-db-secrets-manager-secret;keycloak-initial-admin
      creationTimestamp: "2025-06-04T16:10:53Z"
      generation: 90
      labels:
        app: keycloak
        app.kubernetes.io/instance: keycloak
        app.kubernetes.io/managed-by: keycloak-operator
      name: keycloak
      namespace: identity
      ownerReferences:
        - apiVersion: k8s.keycloak.org/v2alpha1
          kind: Keycloak
          name: keycloak
          uid: 37ed79e7-ff65-4f80-ae89-e07655b8cf48
      resourceVersion: "5108259"
      uid: 56e29c39-7b0f-432a-9a13-316fc143f9dd
    spec:
      persistentVolumeClaimRetentionPolicy:
        whenDeleted: Retain
        whenScaled: Retain
      podManagementPolicy: OrderedReady
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          app: keycloak
          app.kubernetes.io/instance: keycloak
          app.kubernetes.io/managed-by: keycloak-operator
      serviceName: ""
      template:
        metadata:
          annotations:
            ad.datadoghq.com/keycloak.check_names: |
              ["openmetrics"]
            ad.datadoghq.com/keycloak.init_configs: |
              [{}]
            ad.datadoghq.com/keycloak.instances: |
              [
                {
                  "prometheus_url": "http://%%host%%:9000/metrics",
                  "namespace": "keycloak",
                  "send_distribution_buckets": true,
                  "tags":["keycloak-pod:%%host%%"],
                  "metrics": ["*"]
                }
              ]
            ad.datadoghq.com/keycloak.logs: |
              '[{"source": "keycloak", "service": "keycloak", "tags": ["env:int"]}]'
            operator.keycloak.org/watched-secret-hash: b217e53b36a62379f7250157f98db2a04ed404a4efb21a8379c075ebdc5de561
          creationTimestamp: null
          labels:
            app: keycloak
            app.kubernetes.io/component: server
            app.kubernetes.io/env: int
            app.kubernetes.io/instance: keycloak
            app.kubernetes.io/managed-by: keycloak-operator
            app.kubernetes.io/name: keycloak
            tags.datadoghq.com/env: int
            tags.datadoghq.com/service: keycloak
            tags.datadoghq.com/source: keycloak
            tags.datadoghq.com/version: 892777a-1749508351
        spec:
          affinity:
            podAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - podAffinityTerm:
                    labelSelector:
                      matchLabels:
                        app: keycloak
                        app.kubernetes.io/component: server
                        app.kubernetes.io/instance: keycloak
                        app.kubernetes.io/managed-by: keycloak-operator
                    topologyKey: topology.kubernetes.io/zone
                  weight: 10
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - podAffinityTerm:
                    labelSelector:
                      matchLabels:
                        app: keycloak
                        app.kubernetes.io/component: server
                        app.kubernetes.io/instance: keycloak
                        app.kubernetes.io/managed-by: keycloak-operator
                    topologyKey: kubernetes.io/hostname
                  weight: 50
          containers:
            - args:
                - -Djgroups.dns.query=keycloak-discovery.identity
                - -Djgroups.bind.address=$(POD_IP)
                - --verbose
                - start
              env:
                - name: KC_HOSTNAME_STRICT
                  value: "false"
                - name: KC_HOSTNAME_BACKCHANNEL_DYNAMIC
                  value: "false"
                - name: KC_FEATURES
                  value: opentelemetry
                - name: KC_TRACING_ENABLED
                  value: "true"
                - name: KC_TRACING_SERVICE_NAME
                  value: keycloak
                - name: KC_TRACING_ENDPOINT
                  value: http://datadog-agent.datadog:4317
                - name: KC_TRACING_SAMPLER_TYPE
                  value: parentbased_traceidratio
                - name: KC_TRACING_SAMPLER_RATIO
                  value: "0.2"
                - name: KC_TRACING_RESOURCE_ATTRIBUTES
                  value: k8s.namespace.name=identity,additionalProperties=env=int
                - name: KC_HTTP_ENABLED
                  value: "true"
                - name: KC_HTTP_PORT
                  value: "8080"
                - name: KC_HTTPS_PORT
    8BD0
    
                  value: "8443"
                - name: KC_DB
                  value: postgres
                - name: KC_DB_USERNAME
                  valueFrom:
                    secretKeyRef:
                      key: username
                      name: aws-secret-magr-keycloak-db
                - name: KC_DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: password
                      name: aws-secret-magr-keycloak-db
                - name: KC_DB_URL
                  value: jdbc:aws-wrapper:postgresql://{{DATABASE_FQDN}}:5432/{{DATABASE_NAME}}
                - name: KC_PROXY_HEADERS
                  value: xforwarded
                - name: KC_BOOTSTRAP_ADMIN_USERNAME
                  valueFrom:
                    secretKeyRef:
                      key: username
                      name: keycloak-initial-admin
                - name: KC_BOOTSTRAP_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: password
                      name: keycloak-initial-admin
                - name: KC_CACHE
                  value: ispn
                - name: KC_CACHE_STACK
                  value: kubernetes
                - name: KC_HEALTH_ENABLED
                  value: "true"
                - name: KC_METRICS_ENABLED
                  value: "true"
                - name: KC_TRACING_JDBC_ENABLED
                  value: "false"
                - name: KC_HTTP_METRICS_HISTOGRAMS_ENABLED
                  value: "true"
                - name: KC_CACHE_METRICS_HISTOGRAMS_ENABLED
                  value: "true"
                - name: KC_SPI_THEME_WELCOME_THEME
                  value: pbp-login
                - name: KC_SPI_THEME_STATIC_MAX_AGE
                  value: "-1"
                - name: KC_SPI_THEME_CACHE_THEMES
                  value: "false"
                - name: KC_SPI_THEME_CACHE_TEMPLATES
                  value: "false"
                - name: KC_LOG_LEVEL
                  value: INFO
                - name: KC_DB_DRIVER
                  value: software.amazon.jdbc.Driver
                - name: KC_QUARKUS_HTTP_LIMITS_MAX_INITIAL_LINE_LENGTH
                  value: "8192"
                - name: KC_SPI_BRUTE_FORCE_PROTECTOR_DEFAULT_BRUTE_FORCE_DETECTOR_ALLOW_CONCURRENT_REQUESTS
                  value: "true"
                - name: POD_IP
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: status.podIP
                - name: KC_TRUSTSTORE_PATHS
                  value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              envFrom:
                - configMapRef:
                    name: additional-env
              image: 01234567890.dkr.ecr.ca-central-1.amazonaws.com/rhbk-keycloak:892777a-1749508351
              imagePullPolicy: Always
              lifecycle:
                preStop:
                  exec:
                    command:
                      - /bin/sh
                      - -c
                      - sleep 40
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health/live
                  port: 9000
                  scheme: HTTP
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              name: keycloak
              ports:
                - containerPort: 8443
                  name: https
                  protocol: TCP
                - containerPort: 8080
                  name: http
                  protocol: TCP
                - containerPort: 9000
                  name: management
                  protocol: TCP
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /health/ready
                  port: 9000
                  scheme: HTTP
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 1
              resources:
                limits:
                  memory: 2Gi
                requests:
                  memory: 1700Mi
              startupProbe:
                failureThreshold: 600
                httpGet:
                  path: /health/started
                  port: 9000
                  scheme: HTTP
                periodSeconds: 1
                successThreshold: 1
                timeoutSeconds: 1
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 60
          topologySpreadConstraints:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: keycloak
              maxSkew: 1
              topologyKey: topology.kubernetes.io/zone
              whenUnsatisfiable: ScheduleAnyway
      updateStrategy:
        rollingUpdate:
          partition: 0
        type: RollingUpdate
    status:
      availableReplicas: 1
      collisionCount: 0
      currentReplicas: 1
      currentRevision: keycloak-75fd8b4d78
      observedGeneration: 90
      readyReplicas: 1
      replicas: 1
      updateRevision: keycloak-75fd8b4d78
      updatedReplicas: 1
    
  4. 4 remaining items

  5. keycloak-github-bot commented on Jun 10, 2025

    @keycloak-github-bot

    Due to the amount of issues reported by the community we are not able to prioritise resolving this issue at the moment.

    If you are affected by this issue, upvote it by adding a 👍 to the description. We would also welcome a contribution to fix the issue.

  6. added a commit that references this issue on Jun 16, 2025
    bd14961
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0