Why libraries like Guava need more than POMs

January 22, 2020

Jendrik Johannes

New Features

Introduction

More than 10 years ago, a new Java collections library was released by Google. This library, now known as Google Guava, would gain a lot of traction over the months and years and is possibly the most used Java library in production code today.

Due to the widespread adoption of Guava, many other libraries depend on it today. Chances are high that you will find it on the classpath of any reasonably large Java project through transitive dependencies, even if it is not used directly. With more and more code depending on such a widely used library, the potential for conflicts increases, adding to a project’s dependency hell.

Consider the following harmless-looking dependency declaration block:

dependencies {
   implementation("com.google.guava:guava:28.0-jre")
   implementation("org.codehaus.plexus:plexus-container-default:2.1.0")
   implementation("com.google.api-client:google-api-client:1.30.7")
}

We would expect to end up with a JRE (and not an Android) variant of Guava and we would expect the build tool to inform us if there are other suspicious conflicts on our classpath. Let’s take a look at a build scan showing the dependencies graph:

If we look closely, we can observe some unexpected things: Why was guava:28.0-jre, replaced with guava:28.1-android without warning? Why is there a google-collections dependency – wasn’t that the same as Guava? Why do I need j2objc-annotations on my runtime classpath? And what is this weird 9999.0-empty-to-avoid-conflict-with-guava dependency?

To understand this, we will discuss the dependency management challenges that appeared during the evolution of Guava and how they were handled. In the end, we will show how troubles can be avoided using Gradle Module Metadata.

Naming things is hard #

The dependency management troubles started early for what was previously known as the Google Collections Library. com.google.collections:google-collections:1.0 are the coordinates under which the final release of the Google Collections Library was published to the Maven central repository back in 2009. In 2010, the first stable version of Guava, com.google.guava:guava:10.0, was released including all of Google Collections along with other utilities – replacing the Google Collections Library.

As a result of this “rename” from google-collections to guava, the dependency management engines of Gradle and Maven were no longer able to detect conflicts between versions of Google Collections and Guava. Such an unhandled conflict led to two jars on the classpath containing different versions of the Google Collections classes. In the case where build authors discovered the conflict by chance, they had to manually exclude google-collections as transitive dependency or, in Gradle, register a replacement rule.

Alphabetic versioning trouble #

When version 22.0 of Guava was released in May 2017, Guava had moved from Java 6 to Java 8. Android, however, was still stuck on Java 6. Without a change, Android users would have been stuck on old versions forever. Therefore, Guava started shipping a separate Android variant stripped of all Java 8 specific functionality.

The two variants were released using the same com.google.guava:guava coordinates with two different version strings: 22.0 and 22.0-android. After a longer discussion on GitHub and a public GoogleDoc, the versioning pattern changed with 23.1 to 23.1-jre and 23.1-android. Instead of using a different classifier or coordinates for the different variants, using different versions allowed the dependency conflict resolution in Gradle and Maven to detect a conflict and select only one of the two variants. (The -jre suffix was introduced to make sure that the -jre version is always considered higher than the -android version by Maven because j beats a in alphabetical order.)

J (not always) beats A #

While the introduction of the -jre suffix solved some issues for Maven users, problems still remained for Gradle and Maven users if multiple versions of Guava were involved.

Looking again at our initial example, the actual version and the variant are both encoded in the version strings: 28.0-jre and 28.1-android. The build tools cannot know how to use this information. Gradle, looking at the full version string, picks the higher version: 28.1-android. This is a version without the Java 8 specific classes, which might very well break code that relies on those classes. The best solution would probably be to pick 28.1-jre as it satisfies both requests: 28.1 (which is assumed to be compatible with 28.0) and jre (which is compatible with android). However, requesting a version and a variant independently cannot be modelled with POM metadata.

Annoying annotation libraries #

Being aware of its wide-spread use, Guava’s code is mostly self contained, avoiding additional dependencies. Still, over time, Guava added some dependencies to annotation libraries, such as com.google.code.findbugs:jsr305 or com.google.errorprone:error_prone_annotations, required at compile time.

Many users are annoyed by these dependencies as they are also present on the runtime classpath. The annotation library dependencies are only there to avoid compile time warnings if the annotations on Guava’s classes are inspected by the Java compiler. At runtime, the annotations are not touched and thus the annotation library jars are not needed. However, each compile scope dependency defined in a POM is automatically present on the runtime classpath and there is no concept to declare a dependency only for compile time.

Duplication troubles #

In September 2018, one interface – ListenableFuture – was copied from Guava into a separate module – com.google.guava:listenablefuture:1.0 – to allow Android developers to use it in their APIs, without depending on all of Guava. To keep Guava self contained, the development team decided to copy that interface instead of completely moving it out of Guava. Instead, they published an empty version of the ListenableFuture module – com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava – on which Guava depends now. This is tricking Gradle’s dependency management engine to always use the empty 9999.0-empty-to-avoid-conflict-with-guava version of the ListenableFuture module when Guava is used. The real version 1.0, which contains the duplication of the ListenableFuture interface, is not picked then.

While this seems to be an ingenious approach, and it is for many cases of Android development where Gradle is used, there are also problems with it. JVM build authors, who only use Guava, complained that they always have an additional empty jar on their classpath even if there is no conflict. In Maven, the approach only works in certain setups, since Maven does not necessarily pick the highest version, but the closest – i.e. if com.google.guava:listenablefuture:1.0 is discovered first in the dependency graph, it will be picked over the empty version.

When you know what you want but you cannot express it #

If you now think that the Guava team should have done a better job with all those troubles, you are mistaken. In fact, as you can see from the linked discussions, the team was very concerned with each decision taken.

The root cause of the troubles is that the POM metadata model is not expressive enough to convey the required information. As Guava and other libraries have shown over the past decade, there is a need to express more information in metadata to solve many common use cases. As an answer to this need, we developed the Gradle Module Metadata format.

With Gradle Module Metadata, the problems described in this post can be addressed:

• Renaming to Guava Gradle Module Metadata offers the modelling concept of capabilities. With it, a module can express that it provides an alternative implementation of something implemented by another module.
  Each version of Guava could declare that it provides the com.google.collections:google-collections capability and Gradle then would detect the conflict if both Google Collections and Guava are part of the dependency graph.
• Publishing more variants With Gradle Module Metadata, each module has arbitrarily many variants. Each variant can point to different artifacts (jars) and can have different dependencies. A variant is identified by a number of attributes including, in the case of Java libraries, the org.gradle.jvm.version attribute.
  Guava could publish variants for different Java versions, JRE (Java 8) and Android (Java 6/7), in one module. Gradle will then select the right variant based on the Java version used.
• Compile-time only dependencies A module published with Gradle Module Metadata explicitly defines runtime and compile time (api) variants where each of them defines dependencies independently.
  Using this flexible API and implementation separation, Guava could add annotation library dependencies to the compile time variant only preventing them from leaking onto the runtime classpath.
• Copying ListenableFuture to a second module The capabilities concept offered by Gradle Module Metadata, that can be used to solve the renaming issue, can also be used here.
  Guava could declare that it provides the com.google.guava:listenablefuture capability which is enough for Gradle to detect a conflict with the real listenablefuture module if it appears in the dependency graph. In the most common case, where there is no conflict, the capability will have no effect. The dependency on the empty listenablefuture module can be removed.

Conclusion #

In this blog post we’ve taken you on a journey through the history of Guava, as an example of an evolving library that is widely used. As you might have experienced yourself, for such libraries, the chance for version and variant conflicts are high. But with the right amount of metadata published, these conflicts can be detected, and resolved, by build tools.

As we have shown, Guava’s developers are very aware of this and did not take decisions lightly. However, they were limited by the expressiveness of the POM format multiple times. Despite their best efforts, and the application of multiple tricks, many build authors still face issues with undetected and unresolved conflicts involving Guava.

To improve the situation in the future, we created a pull request that proposes publishing of Gradle Module Metadata for Guava. For already published versions of Guava, or other libraries, Gradle allows you to write a component metadata rule to add missing metadata. We have written such a rule for published versions for Guava and provide it as a Gradle plugin. If you apply this plugin to your build, you can explore what we have described in the blog post yourself:

plugins {
   id("de.jjohannes.missing-metadata-guava") version "0.1"
}

If we add the plugin to the example from the beginning, you can observe, for instance, the reduced runtime classpath and that the Java 8 variant of Guava was chosen, which provides the guava-28.1-jre.jar despite the selection of the 28.1-android version.

If you are developing libraries yourself, or know about libraries facing similar issues, feel free to reach out to us. We are happy to explore improvements by publishing Gradle Module Metadata!

Discuss