Custom ingress: why do AWS LB Controller delete the load balancer when a Keycloak pod is killed? #14567

winterrobert · 2022-09-23T16:34:25Z

winterrobert
Sep 23, 2022

We are trying to expose Keycloak using a custom ingress and AWS ALB:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /auth/health/live
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}, {"HTTP": 8080}, {"HTTPS": 8443}]'    
    alb.ingress.kubernetes.io/certificate-arn: xxxx
  labels:
    app: keycloak
  name: keycloak-ingress
spec:
  ingressClassName: alb
  tls:
  - hosts:
    - '{{ .Values.keycloak.hostname }}'
  
  rules:
  - http:
      paths:
      - backend:
          service:
            name: keycloak-service
            port:
              number: 8443
        path: /
        pathType: ImplementationSpecific

Our Keycloak config:

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: keycloak
spec:
  instances: 3
  disableDefaultIngress: true
  image: ourcustomimage
  serverConfiguration:
    - name: db
      value: postgres
    - name: db-url-host
      value: {{ .Values.keycloak.dbUrlHost}}
    - name: http-relative-path
      value: /auth
  unsupported:
    podTemplate:
      spec:
        serviceAccountName: "{{ .Values.keycloak.serviceAccountName }}"
        nodeSelector:
          karpenter.sh/capacity-type: {{ .Values.karpenterCapacityType }}
          karpenter.sh/provisioner-name:  {{ .Values.karpenterProvisionerName }}
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - keycloak
              topologyKey: "kubernetes.io/hostname"
        containers:          
          - name: "keycloak"
            args: ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]            
  hostname: {{ .Values.keycloak.hostname }}
  tlsSecret: {{ .Values.keycloak.tlsSecret }}

This is what our AWS LB Controller does when one of our three keycloak pods is removed (kubectl delete pod keycloak-0):

1: Deregisters the target (pod with ip 10.103.9.62) - this seems intuitive

{"level":"info","ts":1663938421.350132,"msg":"deRegistering targets","arn":"arn:aws:elasticloadbalancing:eu-west-1:xx:targetgroup/k8s-keycloak-keycloak-56ea148d7b/0f4b6813c1be126d","targets":[{"AvailabilityZone":"eu-west-1a","Id":"10.103.9.62","Port":8443}]}
{"level":"info","ts":1663938421.4012036,"msg":"deRegistered targets","arn":"arn:aws:elasticloadbalancing:eu-west-1:x:targetgroup/k8s-keycloak-keycloak-56ea148d7b/0f4b6813c1be126d"}

2: Deletes the load balancer - this seems harsh

{"level":"info","ts":1663938421.6561663,"logger":"controllers.ingress","msg":"deleting loadBalancer","arn":"arn:aws:elasticloadbalancing:eu-west-1:x:loadbalancer/app/k8s-keycloak-keycloak-a5bde81292/bb86575bed927905"}
{"level":"info","ts":1663938421.79802,"logger":"controllers.ingress","msg":"deleted loadBalancer","arn":"arn:aws:elasticloadbalancing:eu-west-1:x:loadbalancer/app/k8s-keycloak-keycloak-a5bde81292/bb86575bed927905"}

3: Deletes the target group and the two remaining targets:

{"level":"info","ts":1663938421.798107,"logger":"controllers.ingress","msg":"deleting targetGroupBinding","targetGroupBinding":{"namespace":"keycloak","name":"k8s-keycloak-keycloak-56ea148d7b"}}
{"level":"info","ts":1663938421.834552,"msg":"deRegistering targets","arn":"arn:aws:elasticloadbalancing:eu-west-1:x:targetgroup/k8s-keycloak-keycloak-56ea148d7b/0f4b6813c1be126d","targets":[{"AvailabilityZone":"eu-west-1a","Id":"10.103.29.184","Port":8443},{"AvailabilityZone":"eu-west-1a","Id":"10.103.53.103","Port":8443},{"AvailabilityZone":"eu-west-1a","Id":"10.103.9.62","Port":8443}]}

I would expect it would do (1) then call it a day like it usually does. What have I missed?

The default Keycloak ingress uses nginx but besides being tied to the Keycloak operator does nothing obviously different that would explain this behaviour?


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    route.openshift.io/termination: passthrough
  labels:
    app: keycloak
    app.kubernetes.io/managed-by: keycloak-operator
  name: keycloak-ingress
  namespace: keycloak
  ownerReferences:
    - apiVersion: k8s.keycloak.org/v2alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Keycloak
      name: keycloak
      uid: 8e0c56c7-ede2-4b29-ba86-5b4e0168b4c5
spec:
  defaultBackend:
    service:
      name: keycloak-service
      port:
        number: 8443
  rules:
    - host: domain.com
      http:
        paths:
          - backend:
              service:
                name: keycloak-service
                port:
                  number: 8443
            pathType: ImplementationSpecific

winterrobert · 2022-09-23T16:46:26Z

winterrobert
Sep 23, 2022
Author

Ok solved it; naming our custom ingress the same as the default ingress (keycloak-ingress) was a death sentence even when specifying disableDefaultIngress: true; changing it to anything else made it stay alive when a keycloak pod was deleted.

0 replies

Navigation Menu

Repository navigation

Custom ingress: why do AWS LB Controller delete the load balancer when a Keycloak pod is killed? #14567

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Custom ingress: why do AWS LB Controller delete the load balancer when a Keycloak pod is killed? #14567

Uh oh!

winterrobert Sep 23, 2022

Replies: 1 comment

Uh oh!

winterrobert Sep 23, 2022 Author

winterrobert
Sep 23, 2022

winterrobert
Sep 23, 2022
Author