Keycloak 26 : cookies error with Idp SAML #35397

antoinechamot · 2024-11-28T09:27:42Z

antoinechamot
Nov 28, 2024

Hi,

We just upgraded keycloak to 26. After the upgrade it is impossible to login with keycloak as broker and azure IDP. We get a 400 on the post request comming from microsoft and a cookie not found. "Please make sure cookies are enabled in your browser"

As i undertand it Keycloak cookies KC_ are set with sameSite Lax which block the request.

Why those cookies are set to lax ?
We have a Traefik reverse proxy that simply forward the https request to the container on 443 port then the container translate to the 8080 port.

Is there a way to force samesite to none ?

Answered by stianst

Nov 28, 2024

For those running into this issue:

Make sure your proxy is including the relevant headers
If your proxy is using X-Forwarded-* headers start Keycloak with --proxy-headers xforwarded
If your proxy is using Forwarded header start Keycloak with --proxy-headers forwarded

If you are still having issues follow https://www.keycloak.org/server/hostname#_troubleshooting to verify that Keycloak is aware of the correct URLs, and most importantly the scheme.

See https://www.keycloak.org/server/reverseproxy and https://www.keycloak.org/server/hostname for more details.

The reason why people are having issues with 26 is there are two changes there:

You need to explicitly tell Keycloak to use X-Forwa…

View full answer

jonkoops · 2024-11-28T10:25:47Z

jonkoops
Nov 28, 2024

This has been reported as an issue under #34295, I suggest you subscribe there. This is likely an improper proxy configuration causing the Keycloak server to be served from an insecure context, causing the cookie to not be available. Please make sure to follow the upgrade guide and check out the hostname (v2) and reverse proxy guides.

0 replies

antoinechamot · 2024-11-28T10:28:32Z

antoinechamot
Nov 28, 2024
Author

Ok it seems KC_PROXY_HEADERS xforwarded was missing. Now login is working but logout fails :-(. I will investigate

0 replies

stianst · 2024-11-28T10:52:54Z

stianst
Nov 28, 2024
Maintainer

For those running into this issue:

Make sure your proxy is including the relevant headers
If your proxy is using X-Forwarded-* headers start Keycloak with --proxy-headers xforwarded
If your proxy is using Forwarded header start Keycloak with --proxy-headers forwarded

If you are still having issues follow https://www.keycloak.org/server/hostname#_troubleshooting to verify that Keycloak is aware of the correct URLs, and most importantly the scheme.

See https://www.keycloak.org/server/reverseproxy and https://www.keycloak.org/server/hostname for more details.

The reason why people are having issues with 26 is there are two changes there:

You need to explicitly tell Keycloak to use X-Forwarded or Forwarded headers; rather than in the past you could just tell Keycloak to use reverse proxy headers. This was done as there was a security issue if the reverse proxy was using the Forwarded header, while Keycloak also looked at the X-Forwarded headers that could be forwarded by a malicious party
Keycloak used to send two cookies; one with samesite and one without. Now it only sends one. This was a temporary work-around as when samesite became a thing some browsers didn't handle this properly. However, the two cookies both result in HTTP request headers being larger as well as being less secure.

Additionally, what makes this a bit more complicated is that browsers consider http://localhost/ a secure context ;)

If anyone still is having issues after this, try enabling trace logging for org.keycloak.cookie.DefaultCookieProvider. That will print a statement for what cookies Keycloak receives, and if you drop us that list as well as when you have the problem. Is it on the first request to Keycloak after SAML redirect? Also, are you using SAML GET or POST?

0 replies

mike-pt · 2024-11-28T11:56:38Z

mike-pt
Nov 28, 2024

I came here form the GHI, in my case I'm still back in 24.0.5 and I see issues only in our PROD instance that is setup the exact same way.

Just to clarify on your comment about the headers @stianst if I'm using the operator have:

This should take care of it correct?

Left = our prod, right = our testing env, the prod one is setting "Strict" in the cookie no even lax, the test one sets NONE, also not lax.

We can't upgrade to any version =>25 yet, we are waiting for a PR I submitted here to be merged: #30669 but the PR as been reviewed and approved yet not merged.. no need to get into the detail of that but its only a problem in Gcloud SQL if on 5.7 or WHEN replication is enabled regardless of its 5 or 8.

We actually tried to move to Postgress even but migrations are not supported natively and we did run into multiple issue.

So we not only did all efforts to upgrade but I even tested the problem in multiple scenarios and created a PR that should ensure this always works but not using TEMPRORAY, anyway I'm already deviating to much, the point is we are actually stuck with 24.0.5 for a while.

With that said I still want to figure out why this can act different other than the headers because as you can see in the image the flag is set. Is there anything at DB level maybe that can override it? Or some realm settings (though I did check all and other then clients and users we share all the same configs).

Also what level should I enable for org.keycloak.cookie.DefaultCookieProviderDebug or Trace?

12 replies

stianst Nov 29, 2024
Maintainer

Best thing to debug any issues around hostname/URLs is to follow the steps in https://www.keycloak.org/server/hostname#_troubleshooting

jonkoops Nov 29, 2024

A patch release likely will not be far out, considering we have a couple of critical bugs.

mike-pt Nov 29, 2024

thanks @stianst I missed the debug flag actually that should help, gonna give it a try

mike-pt Dec 2, 2024

Ok with the debug flag, I test both test/prod and this was very helpfull, I shoudl have seen the extra header in whireshark tbh, but this makes things way more visible:

On prod I was seeing an extra header, as opposed to testing env, and one more IP in forwarded-for:

X-Forwarded-Host ${fqdn}

This one is not present in the testing env... all other values are showing the same (other then the actual URL):



    
URL | Value
-- | --
Request | https://fqdn/auth/realms/p8p/hostname-debug
Frontend | https://fqdnauth/ [OK]
Backend | https://fqdn/auth/ [OK]
Admin | https://fqdn/auth/ [OK]

Server mode | production [start]
Realm | realmName
Realm URL |  

hostname-strict | false
hostname-strict-backchannel | false
hostname-strict-https | true
hostname-port | -1
proxy | none
proxy-headers | xforwarded
http-enabled | true
http-relative-path | /auth
http-port | 8080
https-port | 8443

Host | fqdn
X-Forwarded-For | myIP, LB_IP, OTHER_GOOGLE_IP
X-Forwarded-Host | fqdn
X-Forwarded-Proto | https

The 3rd IP is random at every request.

This was very confusing initially because as stated I'm sure both are deployed with the same exact config... both use a global LB and GCE ingress and dns points to the ingress IP...
So the 3rd IP as to be internal after ingress... and checking network logs I could indeed see this random IP before reaching the KC pods.

So it got me think well whats diff in test and prod? Well that would be security... There is a WAF... I should say a "WAAS" actually, which actually runs at the application layer... or I should say in front of he k8s services if applied. I disabled it to test and now I get no cookie issue.

I assume the issue is actually X-Forwarded-Host, not the extra IP in forwarded-for, this is missing in the test env, and the docs of the security software do state:

sets the original Host HTTP header value in the X-Forwarded-Host HTTP header of the forwarded request...

So with v26 (or 25) because some of this will be set with "None" this should not be a problem even with the WAAS

jonkoops Dec 2, 2024

@vmuzikar FYI

mike-pt · 2025-01-11T17:33:29Z

mike-pt
Jan 11, 2025

We've finally upgraded our envs to mysql 8 and KC 26, but this issue still exists in our prod env, unless I disable WAAS.

Here is the info we see in the debug page:

URL 	Value
Request 	https://idm.example.org/auth/realms/myrealm/hostname-debug
Frontend 	https://idm.example.org/auth/ [OK]
Backend 	https://idm.example.org/auth/ [OK]
Admin 	https://idm.example.org/auth/ [OK]
Runtime 	Value
Server mode 	production [start]
Realm 	myrealm
Realm URL 	
Hostname SPI implementation 	V2
Configuration property 	Value
hostname-backchannel-dynamic 	false
hostname-strict 	false
proxy-headers 	xforwarded
http-enabled 	true
http-relative-path 	/auth
http-port 	8080
https-port 	8443
Header 	Value
Host 	idm.example.org
X-Forwarded-For 	MY_IP, GCP_LB_IP, RANDOM_IP_PRISMA_CLOUD
X-Forwarded-Host 	idm.example.org
X-Forwarded-Proto 	https

This cookie is set as "Strict":

• AUTH_SESSION_ID:"xxxxx-keycloak-1-61186"
Created:"Sat, 11 Jan 2025 17:14:55 GMT"
Domain:"idm.example.org"
Expires / Max-Age:"Session"
HostOnly:true
HttpOnly:true
Last Accessed:"Sat, 11 Jan 2025 17:14:58 GMT"
Path:"/auth/realms/myrealm/"
SameSite:"Strict"
Secure:true
Size:77

Without the WAAS (prisma cloud) in place this doesn't happen and SameStie=None and I also don't have "X-Forwarded-For" header

X-Forwarded-For 	MY_IP, LB_IP
X-Forwarded-Proto 	https

My guess is that this might be due to "X-Forwarded-Host" but I'm not sure.

Is there something that can be done in KC side or is this a issue with the WAAS software / ingress?

1 reply

mike-pt Jan 14, 2025

Update: This is not KC, logs actually look good and show all as None, which got me to debug further, I was able to track it to Prisma Cloud WAAS it self, it injects SameSite:Strict due to their " Cross Site Request Forgery Protection " rule, this can be disabled however! Tested and verified it works as expected

laopaoer-wallet · 2025-05-23T05:18:42Z

laopaoer-wallet
May 23, 2025

I'm experiencing the same issue when using Keycloak 26.2.4 with Azure SAML IDP.

During login, the following error occurs:

For previously logged-in users, it shows:
"Cookie not found. Please make sure cookies are enabled in your browser."

However, for users who have never logged in before, everything works fine.

Then when I added the following directive to my Nginx reverse proxy configuration:

proxy_cookie_flags ~.* secure samesite=None;

everything started working normally.

version: '3.9'
services:
keycloak-clustered-1:
image: keycloak/keycloak:26.2.4
container_name: keycloak-clustered-1
hostname: keycloak-clustered-1
restart: always
command: ['start']
volumes:
- ./keycloak/data:/opt/keycloak/data
- ./keycloak/themes:/opt/keycloak/themes
- ./keycloak/providers:/opt/keycloak/providers
environment:
- KEYCLOAK_ADMIN=admin
- KEYCLOAK_ADMIN_PASSWORD=admin
- KC_DB=mysql
- KC_DB_URL_HOST=mysql
- KC_DB_URL_DATABASE=keycloak
- KC_DB_USERNAME=keycloak_user
- KC_DB_PASSWORD=12345
- KC_DB_URL_PROPERTIES=?useSSL=false
- KC_HEALTH_ENABLED=true
- KC_LOG_LEVEL=INFO,org.infinispan:DEBUG,org.jgroups:DEBUG
- KC_HTTP_ENABLED=true
- KC_METRICS_ENABLED=true
- KC_HOSTNAME=https://xxx.com
- KC_HOSTNAME_ADMIN=https://xxx.com
- KC_FEATURES=cache-embedded-remote-store,declarative-ui,quick-theme
- KC_PROXY_HEADERS=xforwarded
- JAVA_OPTS=-Xms1g -Xmx3g -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=512m -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.err.encoding=UTF-8 -Dstdout.encoding=UTF-8 -Dstderr.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:FlightRecorderOptions=stackdepth=512
ports:
- "8888:8080"
- "7600:7600"
- "7800:7800"

0 replies

Keycloak 26 : cookies error with Idp SAML #35397

Uh oh!

Replies: 6 comments · 13 replies

Uh oh!

Uh oh!

antoinechamot Nov 28, 2024 Author

Uh oh!

stianst Nov 28, 2024 Maintainer

Uh oh!

Uh oh!

Uh oh!

stianst Nov 29, 2024 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 6 comments 13 replies

antoinechamot
Nov 28, 2024
Author

stianst
Nov 28, 2024
Maintainer

stianst Nov 29, 2024
Maintainer