Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked #11024

nicolasleborgne · 2022-03-31T08:26:25Z

nicolasleborgne
Mar 31, 2022

Hi there,

Following the Quarkus migration, I have some blocked event loop issues that i cant figure out why. Maybe someone has an idea? 🙏

Additional infos :

keycloak 17.0.0
running behind an Nginx reverse proxy
debian 11
service managed via supervisorctl

The full stack trace :

2022-03-31 09:16:21,093 WARN [io.vertx.core.impl.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 26116669 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
at io.vertx.core.http.impl.Http1xServerRequest.onEnd(Http1xServerRequest.java:553)
at io.vertx.core.http.impl.Http1xServerRequest.lambda$pendingQueue$1(Http1xServerRequest.java:126)
at io.vertx.core.http.impl.Http1xServerRequest$$Lambda$1178/0x0000000840b5a440.handle(Unknown Source)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240)
at io.vertx.core.streams.impl.InboundBuffer.drain(InboundBuffer.java:227)
at io.vertx.core.streams.impl.InboundBuffer.lambda$fetch$0(InboundBuffer.java:280)
at io.vertx.core.streams.impl.InboundBuffer$$Lambda$1185/0x0000000840ba8440.handle(Unknown Source)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63)
at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38)
at io.vertx.core.impl.EventLoopContext$$Lambda$1135/0x0000000840b13840.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@11.0.14/java.lang.Thread.run(Thread.java:829)

Schwaller · 2022-04-14T14:27:59Z

Schwaller
Apr 14, 2022

I've got the same issue - multiple times per startup of a l 8000 arger composition. I think this just tells you that the startup performance is way below the expected performance. Don't think we can do much about this though - seems like the keycloak team has to act ... unless you want to buy a new super computer.

It's maybe worth mentioning that a large spring app in container starts in 10 seconds while keycloak somehow burns down 2min ... can't imagine what keycloak could be doing in that time. Already fixed it to postgres to speed up but even with 17.0.1 performance is way below my personal expectation. Would be nice if the keycloak team could give some insight why it would make sense to take over 1min to start. (Basically it boils down to the fact that the pervious, non-quarkus version was actually faster when also importing a realm because it didnt have to start twice which is pretty bad imho)

2 replies

nicolasleborgne Apr 14, 2022
Author

Thanks for your feedback, I forgot to close this thread,

To be honest we do not really understand what happened but we found that default configuration with the Quarkus distribution is now in cluster mode. We have several Keycloak servers for different environments (test, staging ...) in an infrastructure provided internally, with no hands on it (😢). It appears that the node discovery system has linked some of these, and leads to this event loop issue. Since we have turn the config property cache to local, it works well !

On the other hand, we experience good performance with pre built configuration mode, it starts in few seconds, and we are far to have super computers (2cpus/4go for the smallest 😂)

Important to note that our usages are pretty small.

Schwaller Apr 19, 2022

That's a great hint - thanks man!
Why wouldn't i want my Keycloak to cluster randomly with other instances -.-

Navigation Menu

Repository navigation

Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked #11024

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked #11024

Uh oh!

nicolasleborgne Mar 31, 2022

Replies: 1 comment · 2 replies

Uh oh!

Uh oh!

Schwaller Apr 14, 2022

Uh oh!

nicolasleborgne Apr 14, 2022 Author

Uh oh!

Uh oh!

Schwaller Apr 19, 2022

nicolasleborgne
Mar 31, 2022

Replies: 1 comment 2 replies

Schwaller
Apr 14, 2022

nicolasleborgne Apr 14, 2022
Author