Avoiding dependency hell with Gradle 6

Calendar
Cédric Champeau
New Features

Dependency hell is a big problem for many teams. The larger the project and its dependency graph, the harder it is to maintain it. The solutions provided by existing dependency management tools are insufficient to effectively deal with this issue.

Gradle 6 aims at offering actionable tools that will help deal with these kind of problems, making dependency management more maintainable and reliable.

Take, for example, this anonymized dependency graph from a real world project:

A large dependency graph

There are hundreds of different libraries in this graph. Some are internal libraries, some are OSS libraries. A proportion of those modules see several releases a week. In practice, with a graph of this size, there’s no way you can avoid typical problems like:

Method not found at runtime

Multiple SLF4J bindings

Dependency issues can cause many problems when building and testing your products and it can be extremely challenging to figure out, on a daily basis, what caused a regression, why the project suddenly doesn’t build anymore or what dependency is responsible for an upgrade of another dependency.

If you are lucky, you would get a compile time error, but it’s common to only see problems occurring when executing tests or even at production runtime. In all these cases the error is often hard to trace back to the source, as it appears after the dependency resolution in the build tool has been successful. So from a dependency management perspective everything is correct, while in fact it is not.

The reason for this mismatch is that the engine resolving dependencies does not have enough information to detect - and if possibly automatically fix - a problem. To provide more information to the engine, modules need to carry more metadata. The good news is that this is the focus of Gradle 6!

Introducing Gradle 6 dependency management

Gradle 6 takes a step forward and is the enabler of a new era in dependency management. With the help of Gradle Module Metadata, Gradle now supports a richer, smarter dependency declaration model which enables the build tool to take better decisions, make builds more reliable, and reduce the cost of maintaining dependency graphs.

A lot of the problems seen in dependency management are usually the consequence of a disagreement between a consumer (e.g, the application you build) and a producer (e.g the library/dependency you use), because there isn’t enough information for the dependency management engine to make good decisions.

It is critical that library (e.g., Guava) or framework authors (e.g, Spring Boot, or internal framework) can express requirements in a richer way so that their users face less dependency management issues. They should be able to express things like “use this version if you don’t know which one to use”, or “if you use this feature, then you also need those additional dependencies”. Those are some of the many options Gradle 6 offers.

A typical dependency declaration is expressed in terms of group, artifact and version (also known as GAV coordinates, e.g. com.google.guava:guava:25.1). Let’s focus for a second on the version part. What does it mean, if you see 25.1:

  • was it the latest release at the moment you wrote the code?
  • was it one version you copied and pasted from StackOverflow and it worked?
  • would it work with 25.0?
  • is it ok to upgrade to 26.0?

A direct consequence of the lack of semantics associated with a single version declaration is that we’re likely to perform optimistic upgrading. We assume that because it works with 25.1, it should be fine to upgrade to 26.0. In practice, this works pretty well, and this has been the strategy used by Gradle for years.

However, there are cases where optimistic upgrading breaks:

  • major version upgrades (breaks binary compatibility)
  • vulnerabilities (you should really not include 1.6 because it has a CVE)
  • regressions (there’s a bug in 1.6)
  • library belongs to a larger set of modules which need to share the same version (e.g Jackson Core, Databind, Annotations, …)
  • …

As an example, Gradle 6 offers you the ability to express things in a richer model:

  • you need this dependency strictly within the [1.0, 2.0[ range (because it follows semantic versioning)
  • and within the range, you prefer 1.5 (because that’s what you have tested)
  • and you reject 1.6, because you know it has a bug that directly affects you
dependencies {
    implementation("org.sample:sample") {
        version {
            strictly("[1.0, 2.0[")
            prefer("1.5")
            reject("1.6")
        }
    }
}

This means that if nobody else cares, the engine would choose 1.5. If another dependency requires 1.7, we know we can safely upgrade to 1.7. However, if another dependency requires 2.1, we can now fail the build because two modules disagree.

In addition, there is information that a producer would not know about a dependency, because it changes after a library is published: discovered bugs, vulnerabilities, incorrect transitive dependencies, etc… This is information which can be pushed at any time to the dependency management engine as additional input!

It’s worth noting that the improvements Gradle provides are not only for consumers. As a library author, you have more flexibility than ever in the way you express what you produce: different modules which should have their versions aligned, libraries with optional features, a platform of suggestions for dependency versions, different binaries for different versions of a runtime and much more!

Gradle has been offering these features for several versions now. However their usage was mostly limited to multi-project setups. With Gradle 6, all these tools are now available to library authors and consumers alike by supporting them in published modules with Gradle Module Metadata. It enables a clearer expression of requirements and allows the engine to compute the best solution.

Gradle Module Metadata

Because the Gradle dependency model is richer than what other build tools offer (Ant+Ivy, Maven, Bazel …), we needed a metadata format to enable all those features for libraries published on binary repositories like Maven Central, Artifactory or Nexus. This metadata format is basically a serialization of the Gradle model. You can learn more about this in our dedicated blog post.

In Gradle 6.0, publication of Gradle Module Metadata is enabled by default.

As a library author, you shouldn’t be worried about using Gradle specific features: in all cases, publication of Maven or Ivy metadata is still possible and we did our best to map Gradle specific concepts to those formats when possible. In case it wasn’t possible, it just means some features will only be available for Gradle users, but typically Maven users wouldn’t lose anything compared to what they have today.

In practice

Last but not least, for Gradle 6 we have significantly rewritten the dependency management documentation section of our userguide to make it more use case centric.

In the upcoming weeks, we are going to publish a number of blog posts covering different use cases in more details. In particular we’ll explain what you can do with Gradle 6, including:

Gradle 6 is a major step towards better dependency management, but development doesn’t stop there: we know we still have a lot of work to do, and we’ll address your feedback, don’t hesitate!

Related Posts