Addressing the complexity of the Java logging ecosystem with capabilities

Calendar
Louis Jacomet
New Features

Table of Contents

Introduction

Gradle 6.0 comes with a number of improvements around dependency management that we present in a series of blog posts. In this post we explore the detection of incompatible dependencies on the classpath, through the concept of capabilities.

In order to illustrate this concept, we will look at the state of logging for Java applications and libraries. Aside from the Java core libraries providing java.util.logging (JUL), there are a number of logging libraries available to developers, for example:

The logging problem #

With such a vast offering, it is no surprise that different libraries use different logging APIs. Combined with the sometimes poor-quality metadata, it is quite common to end up with multiple logging implementations on the runtime classpath of an application.

This leads, amongst other things, to this well-known Slf4J warning:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:.../slf4j-log4j12-1.7.29.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:.../logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

While one could say “This is only a warning”, it shows us a place where problems can and do happen. One of our users, the engineering team at Netflix, told us the following story: By selecting the wrong Slf4J logging binding, the log files in one of their systems ended up in the wrong location – causing a disk to fill up that eventually crashed the system. They are surely not the only ones having experienced something like this.

Another example: the Slf4J documentation clearly states that if you mix the bridging and delegating JARs for a given integration, you will encounter a StackOverflowError.

Log4J 2 adds its own complexity to the landscape since it also offers bridging options, that include Slf4J bridging.

If we take a look at the different libraries, the way they interact, and the options they offer, we end up with the following graph:

Java logging landscape

Figuring out the right combination of libraries based on your choice of logging framework requires studying the compatibility notes of the frameworks you use and the ones you do not use, as they can be included without you noticing via transitive dependencies. In addition, none of these requirements are currently made available to build tools, meaning the tools cannot help developers by letting them know about invalid configurations or even pick the right combination based on a chosen logging stack.

Automatic detection of invalid logging setups #

With the concept of capabilities in Gradle, it becomes possible to provide information to the build tool so that it can, at the minimum, detect invalid setups.

A capability is essentially an identifier for a feature provided by a software module. Multiple modules can provide implementations of the same capability, which allows Gradle to detect if different modules are in conflict. This can be applied directly to two different implementations of the same logging API when they are on the dependency graph.

There are different ways to provide capability information: directly through metadata of a component, by adding it manually to a build or by providing it through a plugin.

We have created such a plugin for the logging domain, which captures all the required information for the logging libraries mentioned above and allows you to resolve conflicts. The dev.jacomet.logging-capabilities Gradle plugin will make sure you are never surprised by an invalid logging configuration at runtime since your project will report problems at build time!

Let’s review some of the situations it handles.

Slf4J and its multiple bindings #

Since Slf4J warns that multiple bindings are problematic, it effectively creates an exclusive implementation relationship. Any module that implements the slf4j-api to provide a binding cannot live on the classpath with another such implementation.

In order to detect that we have multiple Slf4J bindings in a given dependency graph, the plugin adds the capability dev.jacomet.logging:slf4j-impl:1.0 to all of the following modules: logback-classic, slf4j-simple, slf4j-log4j12, slf4j-jdk14, log4j-slf4j-impl and slf4j-jcl.

With that information in place, it becomes illegal to have two Slf4J bindings in any resolved dependency graph, enforcing at build time what was only reported at runtime by Slf4J before.

Here is an example build output with such a failure:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':doIt'.
> Could not resolve all files for configuration ':runtimeClasspath'.
   > Could not resolve org.slf4j:slf4j-simple:1.7.27.
     Required by:
         project :
      > Module 'org.slf4j:slf4j-simple' has been rejected:
           Cannot select module with conflict on capability 'dev.jacomet.logging:slf4j-impl:1.0' also provided by [ch.qos.logback:logback-classic:1.2.3(runtime)]
   > Could not resolve ch.qos.logback:logback-classic:1.2.3.
     Required by:
         project :
      > Module 'ch.qos.logback:logback-classic' has been rejected:
           Cannot select module with conflict on capability 'dev.jacomet.logging:slf4j-impl:1.0' also provided by [org.slf4j:slf4j-simple:1.7.27(runtime)]

Log4J or a different logging solution? #

Log4J had its first release in 2001, before JUL even existed, and has not had a release since 1.2.17 in 2012.

Given this, as developers, you will probably prefer a more recent solution, like an Slf4J binding or Log4J 2, to act as your logging framework but you may be using transitive dependencies that were developed against the Log4J API.

In order to apply your choice, you need to be aware of a few potential conflicts:

  • log4j:log4j needs to be replaced by either log4j-over-slf4j from Slf4J or log4j-1.2-api from Log4J 2
  • These two replacements are themselves exclusive
  • If you use log4j-over-slf4j you cannot use slfj-log4j12

Once again the dev.jacomet.logging-capabilities plugin takes care of declaring the necessary capabilities for you:

  • It will add the dev.jacomet.logging:slf4j-vs-log4j capability to log4j-over-slf4j and slfj-log4j12
  • It will add the dev.jacomet.logging:slf4j-vs-log4j2-log4j capability to log4j:log4j, log4j-over-slf4j and log4j-1.2-api

This gives you the guarantee that you will not mix incompatible bridging and implementations of Log4J in any resolved dependency graph.

A comprehensive solution #

Java logging landscape and conflicts

As you can see on the enhanced graph, there are many other problematic module combinations:

  • Similarly to the Log4J replacement, JUL can be replaced by either Slf4J or Log4J 2
  • For Apache Commons Logging, the Log4J2 integration requires commons-logging while the Slf4J one replaces it
  • …

For all these potential conflicts, the plugin dev.jacomet.logging-capabilities registers the necessary capabilities to detect all invalid combinations. Head to the plugin documentation for a comprehensive list of the capabilities and their role.

Behind the scenes #

The plugin leverages Gradle component metadata rules to add the capability information.

Head over to the plugin code to see how it adds a capability dev.jacomet.logging:slf4j-impl:1.0 to all the modules configured with the rule.

Similarly, rules are added for all the possible conflicts that the graph above identifies.

Enhancing the logging ecosystem at publication time #

With Gradle Module Metadata, the concepts presented in the previous section could be applied to the published metadata of logging libraries. It would make the use of custom ComponentMetadataRules or a plugin like the one above obsolete, because the information would be encoded by the library author into the library’s published metadata.

As a library author, capabilities can be added for publication as shown in the Gradle documentation.

Identifying which capability to declare #

An important part of the work is to determine the coordinates of these shared capabilities. Ideally that choice would be made by the original library which offers an extensible system. Then, third-party implementers would be able to conform to the capability declaration in their implementation.

I don’t want my build to break, I want Gradle to fix it! #

We have seen how capabilities are used to detect conflicts and fail a build in case of such a conflict. But that alone does not help us if we can not fix the detected conflicts. For this, Gradle offers capability resolution strategies.

The dev.jacomet.logging-capabilities plugin already sets up such resolution strategies and offers simple constructs to select and activate them. You can declaratively express your logging choices and the plugin makes sure to enhance your build with the relevant capabilities resolution and substitution rules so that only necessary logging libraries appear on the classpath.

The following will make sure Log4J 2 is used as the logger implementation:

plugins {
  `java-library`
  id("dev.jacomet.logging-capabilities")
}

loggingCapabilities {
  enforceLog4J2()
}

It will:

  • Configure Log4J 2 to bridge Slf4J, if there are Slf4J bridges in the graph
  • Configure the bridging of JUL with Log4J 2
  • Configure the bridging of commons-logging only if required
  • Replace log4j with log4j-1.2-api only if required

However it will not add Log4J 2 dependencies, which are to be added as dependencies – directly or transitively.

For a complete overview of the choice that can be expressed, head over to the plugin documentation.

Behind the scenes #

Gradle offers an API to indicate how to resolve capabilities conflicts.

The plugins uses it to tell Gradle that in case of a conflict on dev.jacomet.logging:slf4j-impl, the engine must select the module org.slf4j:slf4j-simple:1.7.25 for the test runtime classpath. The conflict resolution logic inspects the available candidates and performs a conditional selection.

Note that in the example above, another Slf4J implementation could be used if there was no capability conflict. It is however quite likely that you have the slf4j-simple dependency declared in your build if you intend to use it. And the plugin requires it in order to work properly.

Conclusion #

We have seen that capabilities are a modelling concept provided by Gradle to express mutual exclusivity between different libraries. As shown on the logging use case, they enable Gradle’s dependency resolution to fail when conflicting implementations of a feature are found in the dependency graph. Similarly to alignment, using capabilities with Gradle Module Metadata gives library authors the power to share more knowledge about when and in which combinations their library is intended to be used. With this information available, Gradle offers APIs for build authors to resolve conflicts in their own build declaratively without hackery.

The use cases that can be addressed with the capabilities concept go beyond the logging use case demonstrated here. Libraries that have changed coordinates, that exist in multiple formats (like cglib and cglib-nodep), or simply have different feature sets, can all leverage this concept to express that the presence of more than one module on a classpath should be considered an error.

Discuss

Related Posts