Signed JWT - Federated (Kubernetes Service Accounts) feedback #48130

slaskawi · 2026-04-16T09:30:59Z

slaskawi
Apr 16, 2026

Summary

When integrating the new "Signed JWT - Federated" Client Authenticator for Kubernetes Service Accounts into UDS Core, we encountered a few corner cases that are worth discussing:

The Client ID validation seems too tight, which is a blocker for using the Keycloak Terraform Provider
The feature misses the entire use case for Managed Kubernetes Clusters and centralized Authorization Servers/IdPs
The FederatedJWTClientValidator and AbstractJWTClientValidator encapsulate methods too tightly, which makes them non-extensible
The KubernetesJwksEndpointLoader lacks configuration (or extension points) to enable/disable attaching tokens to the requests

I put all the necessary details along with proposed solutions below. We managed to get all the bits working in the UDS Core platform (see uds-identity-config/pull/838) but we ended up forking many aspects of the original implementation.

@ahus1 @stianst @ryanemerson @sschu I'd love to get your thoughts on these. I'm more than happy to contribute these features back to upstream, but before I start creating tickets and PRs, I wanted to ensure that's the proper path forward.

Client ID validation

As described in keycloak/terraform-provider-keycloak#1542 and #48024, tools like the Keycloak Terraform Provider attach Client ID to each authentication request by default. The JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants RFC also highlights that with Client Assertions, sending the client_id is optional:

Authentication of the client is optional, as described in Section 3.2.1 of OAuth 2.0 [RFC6749] and consequently, the "client_id" is only needed when a form of client authentication that relies on the parameter is used.

B. For client authentication, the subject MUST be the "client_id" of the OAuth client.

Kubernetes doesn't follow the above convention and sets the sub claim to the Service Account (e.g. system:serviceaccount:my-namespace:my-serviceaccount). So if we wanted to follow the spec in this case, we'd need to use the Service Account names as Client ID. This is not feasible for many environments as Clients such as system:serviceaccount:my-namespace:my-serviceaccount are no longer human-readable.

I propose a different approach — relax Client ID validation for the Kubernetes case. This approach has already been proposed in:

Managed Kubernetes Clusters and centralized IdPs

Managed Kubernetes clusters, such as EKS or AKS, use a centralized IdP for both user authentication and issuing Service Account Tokens. When the --oidc-* settings are used in the Kubernetes API Server, it no longer issues tokens but rather delegates this to a different IdP. At the same time, the JWKS URL remains the same and valid, so this integration doesn't affect token signature validation. See the commands below:

# The JWKS endpoint - returns valid content
kubectl get --raw /openid/v1/jwks | jq
{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "<MASKED>",
      "alg": "RS256",
      "n": "<MASKED>",
      "e": "<MASKED>"
    },
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "<MASKED>",
      "alg": "RS256",
      "n": "<MASKED>",
      "e": "<MASKED>"
    }
  ]
}

# The OIDC Discovery - points to the centralized IdP (not Kube itself!!!)
kubectl get --raw /.well-known/openid-configuration | jq
{
  "issuer": "https://oidc.eks.<MASKED>.amazonaws.com/id/<MASKED>",
  "jwks_uri": "https://oidc.eks.<MASKED>.amazonaws.com/id/<MASKED>/keys",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

The jwks_uri as well as Kube's JWKS endpoint return the same keys.

Tokens issued by Kubernetes (and mounted to the Pods) in this scenario use a different issuer:

kubectl create token default | jwt decode -

Token header
------------
{
  "typ": "JWT",
  "alg": "RS256",
  "kid": "<MASKED>"
}

Token claims
------------
{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1776332430,
  "iat": 1776328830,
  "iss": "https://oidc.eks.<MASKED>.amazonaws.com/id/<MASKED>",
  "jti": "<MASKED>",
  "kubernetes.io": {
    "namespace": "default",
    "serviceaccount": {
      "name": "default",
      "uid": "<MASKED>"
    }
  },
  "nbf": 1776328830,
  "sub": "system:serviceaccount:default:default"
}

This doesn't play well with the implementation, which strongly assumes that Kubernetes will issue all the tokens. In this case, it's not Kubernetes — it's the OIDC service from EKS.

Also related to this, in a multi-cloud environment, services like UDS can be deployed on many clouds at the same time. So the implementation needs to be flexible enough to do dynamic issuer lookup before validating the JWT.

We're currently working on a patch for UDS Core to accommodate this scenario here: defenseunicorns/uds-identity-config#838

In essence, we implemented an alternative ClientAssertionStrategy. At first, it tries the default strategy (that assumes Kubernetes issues tokens), but when it fails, it performs an alternative lookup (this can probably be optimized further, but here's what we have):

        // 1. Try standard issuer-based lookup (handles k3d and matching-issuer cases)
        LookupResult defaultResult = defaultStrategy.lookup(context);
        if (defaultResult != null
                && defaultResult.identityProviderModel() != null
                && defaultResult.identityProviderModel().isEnabled()
                && defaultResult.clientModel() != null) {
            LOGGER.debug("Standard issuer-based lookup succeeded");
            return defaultResult;
        }

        // 2. Fallback: client-first lookup for managed K8s clusters where issuer doesn't match
        LOGGER.debug("Standard issuer-based lookup failed, falling back to client-first lookup");
        ClientAssertionState state = context.getState(ClientAssertionState.class, ClientAssertionState.supplier());
        String subject = state != null && state.getToken() != null ? state.getToken().getSubject() : null;
        if (subject == null) {
            LOGGER.debug("Fallback lookup skipped because client assertion subject is missing");
            return null;
        }
        var idpStorage = context.getSession().identityProviders();

        return context.getRealm().getClientsStream()
            .filter(c -> subject.equals(c.getAttribute(FederatedJWTClientAuthenticator.JWT_CREDENTIAL_SUBJECT_KEY)))
            .map(matchingClient -> {
                String idpAlias = matchingClient.getAttribute(FederatedJWTClientAuthenticator.JWT_CREDENTIAL_ISSUER_KEY);
                if (idpAlias == null) {
                    return null;
                }

                IdentityProviderModel idp = idpStorage.getByAlias(idpAlias);
                if (idp != null
                        && UDSKubernetesIdentityProviderFactory.PROVIDER_ID.equals(idp.getProviderId())
                        && idp.isEnabled()) {
                    LOGGER.debugf("Fallback lookup matched client '%s' to IdP '%s'",
                        matchingClient.getClientId(), idpAlias);
                    return new LookupResult(matchingClient, idp);
                }
                return null;
            })
            .filter(Objects::nonNull)
            .findFirst()
            .orElse(null);
    }

The implementation also requires a custom IdP that is almost identical to the KubernetesIdentityProvider but uses an additional lookup for the issuer:

    public boolean verifyClientAssertion(ClientAuthenticationFlowContext<
10000
/span> context) throws Exception {
        String discoveredIssuer = discoverIssuer(config.getIssuer());
        if (discoveredIssuer == null) {
            Response challengeResponse = ClientAuthUtil.errorResponse(Response.Status.INTERNAL_SERVER_ERROR.getStatusCode(), "server_error", "Failed to discover OIDC issuer from Kubernetes API server");
            context.failure(AuthenticationFlowError.INTERNAL_ERROR, challengeResponse);
            return false;
        }

        UDSFederatedJWTClientValidator validator = new UDSFederatedJWTClientValidator(
            context, this::verifySignature, discoveredIssuer, config.getAllowedClockSkew(), true
        );
        validator.setMaximumExpirationTime(3600);
        return validator.validate();
    }
    
    String discoverIssuer(String kubeApiServerUrl) {
        // The final implementation should probably use an Infinispan Cache here. 
        String cached = ISSUER_CACHE.get(kubeApiServerUrl);
        if (cached != null) {
            LOGGER.debugf("Using cached OIDC issuer '%s' for K8s API server '%s'", cached, kubeApiServerUrl);
            return cached;
        }

        String discovered = fetchIssuerFromOidc(kubeApiServerUrl);
        if (discovered == null) {
            // Discovery failed -- don't cache, fail the request and let the client retry.
            // We don't want any fallback in this case as it may produce inconsistent results.
            return null;
        }
        // putIfAbsent is atomic -- if another thread raced us, use their result
        String existing = ISSUER_CACHE.putIfAbsent(kubeApiServerUrl, discovered);
        return existing != null ? existing : discovered;
    }
    
    private String fetchIssuerFromOidc(String kubeApiServerUrl) {
        String wellKnownEndpoint = kubeApiServerUrl + "/.well-known/openid-configuration";
        try {
            String saToken = KubernetesUtils.readServiceAccountToken();

            SimpleHttp simpleHttp = SimpleHttp.create(session);
            var request = simpleHttp.doGet(wellKnownEndpoint).acceptJson();
            if (saToken != null) {
                request.auth(saToken);
            }

            try (SimpleHttpResponse response = request.asResponse()) {
                int status = response.getStatus();
                if (status < 200 || status >= 300) {
                    LOGGER.errorf("OIDC discovery from '%s' returned HTTP %d",
                        wellKnownEndpoint, status);
                    return null;
                }

                OIDCConfigurationRepresentation oidcConfig = JsonSerialization.readValue(
                    response.asString(), OIDCConfigurationRepresentation.class);
                String discoveredIssuer = oidcConfig.getIssuer();

                if (discoveredIssuer == null || discoveredIssuer.isEmpty()) {
                    LOGGER.errorf("OIDC discovery from '%s' returned empty issuer",
                        wellKnownEndpoint);
                    return null;
                }

                if (!discoveredIssuer.startsWith("https://")) {
                    LOGGER.errorf("OIDC discovery from '%s' returned non-HTTPS issuer '%s'",
                        wellKnownEndpoint, discoveredIssuer);
                    return null;
                }

                LOGGER.infof("Discovered OIDC issuer '%s' for K8s API server '%s'", discoveredIssuer, kubeApiServerUrl);
                return discoveredIssuer;
            }
        } catch (Exception e) {
            LOGGER.errorf(e, "Failed to discover OIDC issuer from '%s'", wellKnownEndpoint);
            return null;
        }
    }

The above piece of code has been implemented in a very defensive way. We believe it can be unified and squashed into one discovery call that should satisfy both cases (managed Kubernetes clusters and local ones).

The most complicated aspect of this is testing. Doing it properly, requires deploying Keycloak on EKS/AKS (and potentially others) which leads to this discussion: #47669

The `FederatedJWTClientValidator` and `AbstractJWTClientValidator` encapsulate methods too tightly, which makes them non-extensible

When working on integrating Tofu with the "Signed JWT - Federated" Client Authenticator, we could not easily bypass the Client ID validation by providing our own validateClient method. Unfortunately, all the methods in AbstractJWTClientValidator are private, which requires copy-pasting them into our codebase and maintaining the changes.

A much better approach would be to change all the method signatures to protected, enabling custom implementations to override only certain aspects of the token validation.

The `KubernetesJwksEndpointLoader` lacks configuration (or extension points) to enable/disable attaching tokens to the requests

The KubernetesJwksEndpointLoader prevents sending the token on issuer mismatch:

            if (jwt.getIssuer().equals(issuer)) {
                logger.trace("Including service account token in request");
                return token;
            } else {
                logger.debug("Not including service account token due to issuer missmatch");
            }

This approach makes a lot of sense to avoid sending tokens to fake services prepared by an attacker. It also works great when the issuer is known, but in case of dynamic configurations (like managed Kubernetes clusters - multi cloud environmens), that is not always the case. If we decide to implement a more dynamic approach, we will need a configuration switch that allows always including the token when probing the OIDC Discovery endpoint (which requires a token for managed cloud offerings).

J0F3 · 2026-04-16T16:07:57Z

J0F3
Apr 16, 2026

Managed Kubernetes clusters, such as EKS or AKS, use a centralized IdP for both user authentication and issuing Service Account Tokens.

That is an interesting point. However, this seems to be specific to AWS/EKS. For Azure/AKS this is definitely not the case. For AKS, the tokens are issued by Kubernetes: https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer#get-the-discovery-document. I also successfully tested the current implementation with service account tokens from AKS.

As described in keycloak/terraform-provider-keycloak#1542 and #48024, tools like the Keycloak Terraform Provider attach Client ID to each authentication request by default.

In my opinion, this is more an issue of the client, which sets the client ID when using a bearer token for authentication. So, in this specific case, I think this should be addressed in the Terraform provider.
The Quarkus OIDC client, for example, seems to handle this correctly and does not send the client ID when a client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer token is used.

But I see your point. At the beginning, I also had a hard time understanding why it was not working because of the strict client-id validation.

1 reply

slaskawi Apr 17, 2026
Author

Yes, I do believe we have two use cases - when Kubernetes issues all the tokens (Minikube, K3d, K3s etc) and when there's an external Authorization Server that does it (EKS, probably others).

I also agree with you on the Client ID - I think it's primary an early adoption problem. But still - Keycloak seems to enforce this containt without good reason, so the fix can actually be applied both sides.

Navigation Menu

Repository navigation

Signed JWT - Federated (Kubernetes Service Accounts) feedback #48130

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Signed JWT - Federated (Kubernetes Service Accounts) feedback #48130

Uh oh!

slaskawi Apr 16, 2026

Summary

Client ID validation

Managed Kubernetes Clusters and centralized IdPs

The FederatedJWTClientValidator and AbstractJWTClientValidator encapsulate methods too tightly, which makes them non-extensible

The KubernetesJwksEndpointLoader lacks configuration (or extension points) to enable/disable attaching tokens to the requests

Replies: 1 comment · 1 reply

Uh oh!

J0F3 Apr 16, 2026

Uh oh!

slaskawi Apr 17, 2026 Author

slaskawi
Apr 16, 2026

The `FederatedJWTClientValidator` and `AbstractJWTClientValidator` encapsulate methods too tightly, which makes them non-extensible

The `KubernetesJwksEndpointLoader` lacks configuration (or extension points) to enable/disable attaching tokens to the requests

Replies: 1 comment 1 reply

J0F3
Apr 16, 2026

slaskawi Apr 17, 2026
Author