[KC17] [Logging] Running Keycloak as a container: how to send logs to logstash ? #10884

semangard · 2022-03-23T09:04:31Z

semangard
Mar 23, 2022

Hello,

We are running KeyCloak as a containers' cluster for a while on Docker Swarm.
For some reasons we need to send our logs to ELK through logstash.

With previous version this was possible with the following CLI :


# ##############################################################################
# Adjust default KeyCloak logging settings to send then to Logstash > ELK
# WARNING : commands CANNOT be ran by a batch (due to the if test and try-catch)
# http://docs.wildfly.org/23/Admin_Guide.html#Logging
# https://www.wildfly.org/news/2020/01/04/Custom-Logging-Filter/
# https://docs.wildfly.org/20/wildscribe/core-service/module-loading/index.html
# ##############################################################################

embed-server --server-config=standalone-ha.xml --std-out=echo

echo "--------------------------------------------------------------------------"

# ##############################################################################
# LOGSTASH handler
# ##############################################################################
try
    if (outcome == success) of /core-service=module-loading:module-info(name=ch.hcuge.logging)
        echo "Logging module already added => removing it..."
        module remove --name=ch.hcuge.logging
    end-if
catch
end-try
try
    echo "Adding LOGGING module..."
    module add --name=ch.hcuge.logging --resources=/opt/jboss/keycloak/providers/hug-keycloak.jar --dependencies=org.jboss.modules,java.logging,org.jboss.logging
catch
    echo "ERROR while adding logging module"
end-try
try
    echo "Adding custom filter..."
    /subsystem=logging/filter=customFilter:add(module=ch.hcuge.logging, class=ch.hcuge.logging.KeycloakMDCFilter)
catch
end-try
try
    echo "Adding JSON formatter..."
    /subsystem=logging/json-formatter=JSON-FORMATTER/:add(zone-id=GMT+1)
catch
end-try

# JUST for DEBUGGING JSON formatter
# try
#    echo "Updating console formatter..."
#    /subsystem=logging/console-handler=CONSOLE/:write-attribute(name=named-formatter,value=JSON-FORMATTER)
# catch
# end-try

try
    echo "Adding a socket binding..."
    /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=log-server:add(host="${env.LOGSTASH_DESTINATION:logstash}", port="${env.LOGSTASH_PORT:9689}")
catch
end-try
try
    echo "Adding custon filer to socket handler..."
    /subsystem=logging/socket-handler=log-server-handler:add(filter-spec=customFilter, outbound-socket-binding-ref=log-server, block-on-reconnect=true, named-formatter=JSON-FORMATTER,protocol=TCP, level="${env.LOGSTASH_LOG_LEVEL:DEBUG}")
    # /subsystem=logging/async-handler=async-log-server-handler:add(level=ALL, queue-length=${env.LOGSTASH_QUEUE:1000}, overflow-action=BLOCK,subhandlers=[log-server-handler])
catch
end-try
try
    echo "Adding the log handler..."
    /subsystem=logging/root-logger=ROOT:add-handler(name=log-server-handler)
catch
end-try

# ##############################################################################
# CONSOLE handler
# ##############################################################################
# add yyyy-MM-dd GMT+1 to each CONSOLE log
try
    echo "Changing CONSOLE pattern..."
    /subsystem=logging/pattern-formatter=COLOR-PATTERN:write-attribute(name="pattern", value="%K{level}%z{GMT+1}%d{yyyy-MM-dd HH:mm:ss,SSS XXX} %-5p [%c] (%t) %s%e%n")
catch
end-try

echo "--------------------------------------------------------------------------"

stop-embedded-server

And thus we just had to set the following env vars on our container spec:

 LOGSTASH_DESTINATION: ...
 LOGSTASH_PORT: ...
 LOGSTASH_QUEUE: ...

==> How to do something similar with KeyCloak 17 (on Quarkus) runned as a container (also JSON logs...) ?

Thanks and regards

tse00 · 2022-03-24T16:14:31Z

tse00
Mar 24, 2022

Hello,

we are currently also working on the setup for Logstash on Keycloak 17. Unfortunately after I made all steps needed, I still wasn't able to see any logs from Keycloak 17 on Logstash/Kibana. It seems like quarkus/keycloak don't recognize the gelf plugin since it doens't appear on the logs under "Installed features".

Semangard I hope this help you. With the help of the documentation and some information found on the internet, we got this setup:

copied quarkus-logging-gelf-2.7.0.jar to /providers and to /opt/keycloak/lib/lib/main/ (not sure if the last path is necessary)

created conf/quarkus.properties file and add the quarkus-gelf parameters:

quarkus.log.handler.gelf.enabled=true
quarkus.log.handler.gelf.host=...
quarkus.log.handler.gelf.port=...
quarkus.log.handler.gelf.facility=...

More info: https://quarkus.io/guides/centralized-log-management

Our Dockerfile looks something like this:

FROM quay.io/keycloak/keycloak:17.0.0 as builder

ENV KC_DB=postgres
ENV KC_FEATURES=quarkus-logging-gelf

# QUARKUS GELF
COPY ./docker/conf/quarkus.properties /opt/keycloak/conf/quarkus.properties
COPY ./docker/lib/lib/main/quarkus-logging-gelf-2.7.0.Final.jar /opt/keycloak/lib/lib/main/quarkus-logging-gelf-2.7.0.Final.jar
COPY ./docker/providers/quarkus-logging-gelf-2.7.0.Final.jar /opt/keycloak/providers/quarkus-logging-gelf-2.7.0.Final.jar

RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:17.0.0

COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
COPY --from=builder /opt/keycloak/lib/lib/main/quarkus-logging-gelf-2.7.0.Final.jar /opt/keycloak/lib/lib/main/quarkus-logging-gelf-2.7.0.Final.jar
COPY --from=builder /opt/keycloak/providers/ /opt/keycloak/providers/

WORKDIR /opt/keycloak
ENV KEYCLOAK_ADMIN=admin
ENV KEYCLOAK_ADMIN_PASSWORD=admin
ENV KC_PROXY=edge

ENTRYPOINT ["./bin/kc.sh", "start"]

We are sure the problem is not on the configuration of gelf/logstash since we use that service with the same parameters for other projects too without any issues.

Do I have to activate or use some parameter in order to use the gelf jar ?

Thanks you and best regards

7 replies

errier Mar 30, 2022

8000

@tse00
Thanks, this helped. Maybe I can help you too.

About the jar files:

You also need to copy the quarkus-logging-gelf-deployment-2.7.0.Final.jar file to the providers directory for it to be picked up on runtime. Edit: You also may need logstash-gelf-1.15.0.jar.
There is indeed no need to also copy the same jar file to /opt/keycloak/lib/lib/main.

Two remarks about KC_FEATURES=quarkus-logging-gelf.

The extension is called logging-gelf, not quarkus-logging-gelf.
Besides the first point this environment variable is used for Keycloak features, if I understand the documentation correctly. So this won't work for non Keycloak features, I think. The documentation provides you with a list of extensions it supports.

My logging now shows the extension:

[io.quarkus] (main) Installed features: [..., logging-gelf, ...]

tse00 Mar 30, 2022

@errier thanks a lot, is working now!!! Together with the quarkus-logging-gelf-deployment, I also needed to add the logstash-gelf. So to make this work 3 dependencies need to be copied to /providers

quarkus-logging-gelf-2.7.5.Final.jar
quarkus-logging-gelf-deployment-2.7.5.Final.jar
logstash-gelf-1.15.0.jar

semangard Mar 30, 2022
Author

It is possible to use ENV VARS into quarkus.properties ?

gunterze May 20, 2022

It's possible to use ENV VARS in quarkus.properties, e.g.

quarkus.log.handler.gelf.enabled=${GELF_ENABLED:false}

PJaroslaw Jul 22, 2022

Hi just wondering what I might have done wrong. I followed what's written in this post and added all the jars to /providers directory, but when I start up my keycloak-quarkus application, I don't see any logs being sent. Any ideas?

gdelbos · 2022-05-23T07:44:51Z

gdelbos
May 23, 2022

Hi here,

Using the information on this discussion, I was able to send Keycloak log to logstash using GELF.
However, in the previous distrubution (using wildfly) we were able to configure a JUL filter to add some MDC informations.
This is really helpful as it will ensure that any Keycloak logs sent to logstash will be properly labelled as "keycloak" logs.

However, I cannot find a proper solution allowing me to do this.

Does anyone have a solution or a workaround ?

Thanks for your help.

1 reply

gdelbos May 23, 2022

Nevermind, I could use the "additional-field" configuration of quarkus gelf.
Is not exactly the same thing but it is an acceptable workaround.

eg.

quarkus.log.handler.gelf.additional-field."mdc.service".value=${GELF_SERVICE_NAME:keycloak}

[KC17] [Logging] Running Keycloak as a container: how to send logs to logstash ? #10884

Uh oh!

Uh oh!

Replies: 2 comments · 8 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

semangard Mar 30, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 2 comments 8 replies

semangard Mar 30, 2022
Author