Encoders

What is an encoder

Encoders are responsible for transforming an event into a byte array as well as writing out that byte array into an OutputStream. Encoders were introduced in logback version 0.9.19. In previous versions, most appenders relied on a layout to transform an event into a string and write it out using a java.io.Writer. In previous versions of logback, users would nest a PatternLayout within FileAppender. Since logback 0.9.19, FileAppender and subclasses expect an encoder and no longer take a layout.

Why the breaking change?

Layouts, as discussed in detail in the next chapter, are only able to transform an event into a String which restricts their scope to non-binary output.

Encoder interface

Encoders are responsible for transforming an incoming event into a byte array. Here is the Encoder interface:

package ch.qos.logback.core.encoder;
/**
 * Encoders are responsible for transform an incoming event into a byte array
 */
public interface Encoder<E> extends ContextAware, LifeCycle {

    /**
     * Get header bytes. This method is typically called upon opening of an output
     * stream.
     *
     * @return header bytes. Null values are allowed.
     */
    byte[] headerBytes();

    /**
     * Encode an event as bytes.
     *
     * @param event
     */
    byte[] encode(E event);

    /**
     * Get footer bytes. This method is typically called prior to the closing of the
     * stream where events are written.
     *
     * @return footer bytes. Null values are allowed.
     */
    byte[] footerBytes();
}

As you can see, the Encoder interface consists of few methods, but surprisingly many useful things can be accomplished with these methods.

LayoutWrappingEncoder

Until logback version 0.9.19, many appenders relied on the Layout instances to control the format of log output. As there exists substantial amount of code based on the layout interface, we needed a way for encoders to interoperate with layouts. LayoutWrappingEncoder bridges the gap between encoders and layouts. It implements the encoder interface and wraps a layout to which it delegates the work of transforming an event into string.

Below is an excerpt from the LayoutWrappingEncoder class illustrating how delegation to the wrapped layout instance is done.

package ch.qos.logback.core.encoder;

public class LayoutWrappingEncoder<E> extends EncoderBase<E> {

  protected Layout<E> layout;
  private Charset charset;
 
   // encode a given event as a byte[]
   public byte[] encode(E event) {
     String txt = layout.doLayout(event);
     return convertToBytes(txt);
  }

  private byte[] convertToBytes(String s) {
    if (charset == null) {
      return s.getBytes();
    } else {
      return s.getBytes(charset);
    }
  } 
}

The doEncode() method starts by having the wrapped layout convert the incoming event into string. The resulting text string is converted to bytes according to the charset encoding chosen by the user.

PatternLayoutEncoder

Given that PatternLayout is the most commonly used layout, logback caters for this common use-case with PatternLayoutEncoder, an extension of LayoutWrappingEncoder restricted to wrapping instances of PatternLayout.

As of logback version 0.9.19, whenever a FileAppender or one of its subclasses was configured with a PatternLayout, a PatternLayoutEncoder must be used instead. This is explained in the relevant entry in the logback error codes.

immediateFlush property

As of logback 1.2.0, the immediateFlush property is part of the enclosing Appender.

Output pattern string as header

In order to facilitate parsing of log files, logback can insert the pattern used for the log output at the top of log files. This feature is disabled by default. It can be enabled by setting the outputPatternAsHeader property to 'true' for relevant PatternLayoutEncoder. Here is an example:

<configuration>
  <!-- omitted lines ... -->
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>foo.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
      <pattern>%d %-5level [%thread] %logger{0}: %msg%n</pattern>
      <outputPatternAsHeader>true</outputPatternAsHeader>
    </encoder>
  </appender>
  <!-- omitted lines ... -->
</configuration>

This will result output akin to the following in the log file:

#logback.classic pattern: %d [%thread] %-5level %logger{36} - %msg%n
2012-04-26 14:54:38,461 [main] DEBUG com.foo.App - Hello world
2012-04-26 14:54:38,461 [main] DEBUG com.foo.App - Hi again
...

The line starting with "#logback.classic pattern" is newly inserted pattern line.

JsonEncoder

since versions 1.3.8/1.4.8

JsonEncoder follows the JSON Lines standard. It transforms a logging event into a valid json text in conformance with [RFC-8259]. Each logging event in JSON text format is followed by a new line. More specifically, certain characters such as quotations marks, backslash, forward slash, form feed, line feed, carriage return characters are escaped according to the rules described in section 7 of RFC-8259.

Here is sample output post-formatted for readability:

{
   "sequenceNumber":0,
   "timestamp":1686403686358,
   "nanoseconds":358357100,
   "level":"INFO",
   "threadName":"main",
   "loggerName":"org.foo.Bar",
   "context":{
      "name":"default",
      "birthdate":1686403685679,
      "properties":{
         "moo":"299857071"
      }
   },
   "mdc":{
      "a1":"v1299857071"
   },
   "kvpList":[
      {
         "ik299857071":"iv299857071"
      },
      {
         "a":"b"
      }
   ],
   "message":"Hello \"Alice\"",
   "throwable":{
      "className":"java.lang.RuntimeException",
      "message":"an error",
      "stepArray":[
         {
            "className":"org.foo.Bar",
            "methodName":"httpCall",
            "fileName":"Bar.java",
            "lineNumber":293
         },
         {
            "className":"jdk.internal.reflect.DirectMethodHandleAccessor",
            "methodName":"invoke",
            "fileName":"DirectMethodHandleAccessor.java",
            "lineNumber":104
         },
         .... omitted lines
      ]
   }            
}

Note that actual output will be denser, with only one logging event per line.

Here is a sample configuration file using JsonEncoder.

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>foo.json</file>
    <encoder class="ch.qos.logback.classic.encoder.JsonEncoder"/>
  </appender>


  <root>
    <level value="DEBUG"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

since versions 1.5.0 Output of all top-level (json) members can be enabled or disabled. By default all top-level members are enabled except for the "formattedMessage". Below is a configuration file which enables "formattedMessage" and disables output of the raw "message" and the "arguments" members.

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>foo.json</file>
    <encoder class="ch.qos.logback.classic.encoder.JsonEncoder">
       <withFormattedMessage>true</withFormattedMessage>
       <withMessage>false</withMessage>
       <withArguments>false</withArguments>       
    </encoder>
  </appender>


  <root>
    <level value="DEBUG"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

As the above example suggests, output of a top-level member with the name "pellam" can be enabled/disabled using the XML element <withPellam>.

KryoEncoder

since version 1.6.7

KryoEncoder transforms logging events into a compact binary stream with the help of the Kryo 5 serialization library. KryoDecoder can read the resulting stream back and reconstitute the logging events, so that they can be inspected or replayed through the appenders of a LoggerContext. Compared to text-based encoders, KryoEncoder produces smaller output at a lower CPU cost. The decoded events retain the fields of the original event, including the MDC, markers, key-value pairs, the throwable (if any) and caller data (if available).

KryoEncoder ships in the logback-classic-kryo module, which in turn depends on the logback-core-kryo module. Both modules require Java 11 or later. Their JPMS module names are ch.qos.logback.classic.kryo and ch.qos.logback.core.kryo. If you are using Maven, add the following dependency to your project.

<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic-kryo</artifactId>
  <version>1.6.7</version>
</dependency>

Here is a sample configuration file using KryoEncoder.

<configuration>
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>app.kryo</file>
    <encoder class="ch.qos.logback.classic.kryo.KryoEncoder"/>
  </appender>

  <root>
    <level value="DEBUG"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

Stream formats

A Kryo stream starts with a 6-byte header consisting of the magic bytes "STWD", followed by the stream version and a flags byte indicating the serializer type. The appender writes the header when it starts, so that KryoDecoder can determine the format of the stream on its own. Two stream formats are supported, selected by the serializerType property of KryoEncoder.

The Custom serializer type is recommended in most cases. Setting serializerType to Vanilla automatically selects stream version 0.

Properties of KryoEncoder

Note that KryoEncoder is stateful, as the encoding of an event may refer to values written by previous events in the same segment.

Arguments of the logging message and values of key-value pairs are restricted to strings and the numeric wrapper types Integer, Long, Short, Float and Double. Other objects are converted to strings by invoking their toString() method before encoding. Thus, no application classes are ever written into or instantiated from a Kryo stream.

Reading Kryo streams with KryoDecoder

KryoDecoder reads a file produced by KryoEncoder. It takes the stream version and the serializer type from the header of the file. The decoded events, of type LoggingEventKryoVO, implement the ILoggingEvent interface. They can be collected into a list, passed to a Consumer, or replayed through the appenders of a LoggerContext.

import ch.qos.logback.classic.kryo.KryoDecoder;
import ch.qos.logback.classic.kryo.LoggingEventKryoVO;

KryoDecoder decoder = new KryoDecoder(new File("app.kryo"));
List<LoggingEventKryoVO> events = decoder.decodeAsList();

// alternatively, process each event as it is decoded
// KryoDecoder.DecodeStats stats = decoder.decodeWithConsumer(event -> ...);

// or replay the events through the appenders of a LoggerContext
// KryoDecoder.DecodeStats stats = decoder.decodeAndAppend(loggerContext);

A KryoDecoder instance reads its file only once. The DecodeStats object returned by decodeWithConsumer and decodeAndAppend reports the number of decoded events. Its truncated() method returns true if some events may be missing because the stream was damaged or cut short. A stream of an unsupported version is rejected with an IOException.

The KryoDecoderApp application replays a Kryo file through the appenders defined in a logback configuration file, for example to convert a Kryo file into a human-readable log file. The first argument is the Kryo file and the second is the configuration file.

java ch.qos.logback.classic.kryo.KryoDecoderApp app.kryo logback-replay.xml