Optional dependencies are not optional

Calendar
Cédric Champeau
New Features

In a previous blog post, we demonstrated how capabilities could be used to elegantly solve the problem of having multiple logging frameworks on the classpath. In this post, we will again use this concept in a different context: optional dependencies.

At Gradle, we often say that there are no optional dependencies: there are dependencies which are required if you use a specific feature. Let’s explain why.

Optional dependencies

Until recently, Gradle didn’t offer any way to publish optional dependencies, which is something which puzzled a number of Apache Maven™ users. To understand in what context optional dependencies are used, let’s look at a real world project. The Apache PDFBox library declares the following optional dependencies in its POM file:

<dependencies>
    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcmail-jdk15on</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcprov-jdk15on</artifactId>
        <optional>true</optional>
    </dependency>
</dependencies>

Those are 2 dependencies on a specific component, the BouncyCastle cryptography library.

Now let’s imagine your project depends on PDFBox, either using Gradle:

dependencies {
    implementation("org.apache.pdfbox:pdfbox:2.0.17")
}

or using Apache Maven™:

<dependencies>
   <dependency>
       <groupId>org.apache.pdfbox</groupId>
       <artifactId>pdfbox</artifactId>
       <version>2.0.17</version>
   </dependency>
</dependencies>

Now if you look at the dependencies that both Maven and Gradle resolve, you will see that the Bouncycastle library is absent. This is because it’s defined as an optional dependency. There are multiple problems with optional dependencies as they are defined today:

  • Library authors know why a dependency is optional, but consumers don’t: how do you know when you should add bcprov-jdk15on?
  • Optional dependencies are mixed up: how do you know when to add bcprov-jdk15on and if bcmail-jdk15on should be added as well?
  • Optional dependencies are ignored by both Maven and Gradle when resolving transitive dependencies: the information is purely documentation that may help a user to manually add additional dependencies to a build.

In our example, let’s imagine you’d like to generate signed PDFs. By declaring the dependencies like above, you’ll soon realize that you are missing dependencies. To figure out which dependencies are missing, you could look at the POM file of PDFBox and guess which version of Bouncycastle you need to use just by reading this.

Said differently, fixing is an educated guess: because you kind of know that Bouncycastle is related to security, you figure out that maybe if you add the dependencies, it will work. Problem solved?

From optional dependencies to features

The reality is that the dependency on Bouncycastle is not optional: it’s required if you want to sign PDFs. There’s an implicit feature of PDFBox which is “signing”, and if, and only if, you use that feature, then you need a couple more dependencies. Wouldn’t it be nice if the build tools allowed library authors to express that?

That’s exactly what Gradle’s feature variants are for!

For the sake of this demonstration, let’s imagine that PDFBox would use Gradle to build their project instead of Maven. Then they could declare a feature using this:

java {
    registerFeature('signing') {
        usingSourceSet(sourceSets.main)
    }
}

This declares that PDFbox has a feature named signing, and that this feature is “using the main source set”. In Gradle terms, it means that the feature is using the same source directory as the main library (src/main/java), combining the main sources of the library and the ones that perform signing using Bouncycastle. Gradle also offers the ability to write a feature in separate source set (src/myFeature/java) to isolate the code of the feature from the main code and publish it in a separate jar.

Now that the signing feature is defined, the dependencies specific to this feature can be declared:

dependencies {
    signingImplementation("org.bouncycastle:bcmail-jdk15on:1.64")
    signingImplementation("org.bouncycastle:bcprov-jdk15on:1.64")
}

That’s all! But what’s the benefit compared to declaring optional dependencies in Maven?

Well, if only we look at the POM file that Gradle would generate…

<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcmail-jdk15on</artifactId>
  <version>1.64</version>
  <optional>true</optional>
</dependency>
<dependency>
  <groupId>org.bouncycastle</groupId>
  <artifactId>bcprov-jdk15on</artifactId>
  <version>1.64</version>
  <optional>true</optional>
</dependency>

…It’s exactly the same as the one published by Maven! Gradle offers the ability to define and publish optional dependencies by defining features and feature specific dependencies and a Maven user can consume this POM file generated by Gradle similar to the corresponding Maven POM file: look at the POM file directly and figure out what dependencies to add manually.

But, as a Gradle user, the benefit is much higher because things can be expressed in terms of features instead of optional dependencies. Imagine that you need PDFBox and its signing feature. Then you need to declare two dependencies:

dependencies {
    implementation("org.apache.pdfbox:pdfbox:2.0.17")
    implementation("org.apache.pdfbox:pdfbox:2.0.17") {
        capabilities {
           requireCapability("org.apache.pdfbox:pdfbox-signing")
        }
    }
}

We have two distinct dependency declarations here:

  • The first one tells Gradle that we need PDFBox. It’s the “main dependency”.
  • The second one tells Gradle that we also want PDFBox’s signing feature (pdfbox-signing).

The second dependency “points to” a different variant because Gradle, by convention, created a capability corresponding to the feature name declared by PDFBox.

The big advantage is that users don’t have to figure out what dependencies they need to get signing working now: they will get them transitively!

It’s also interesting to notice that because we defined the dependency to Bouncycastle as an implementation dependency, the consumer doesn’t need it at compile time, only at runtime. That’s why the dependency doesn’t appear on the compile classpath!

What makes this possible?

The enabler for this is again Gradle Module Metadata. This file, just like the pom.xml file, contains metadata which the dependency resolution uses to find transitive dependencies.

In this case, the generated Gradle Module Metadata file for our “fake” PDFBox library contains the following:

{
  "name": "signingRuntimeElements",
  "attributes": {
    "..."
  },
  "dependencies": [
    {
      "group": "org.bouncycastle",
      "module": "bcmail-jdk15on",
      "version": {
        "requires": "1.64"
      }
    },
    {
      "group": "org.bouncycastle",
      "module": "bcprov-jdk15on",
      "version": {
        "requires": "1.64"
      }
    }
  ],
  "files": [
    {
      "name": "pdfbox-2.0.17-gradle.jar",
      "..."
    }
  ],
  "capabilities": [
    {
      "group": "org.apache.pdfbox",
      "name": "pdfbox-signing",
      "version": "2.0.17-gradle"
    }
  ]
}

It actually defines an additional variant, signingRuntimeElements, representing the signing feature we defined in the Gradle build above. This variant includes the feature specific dependencies to Bouncycastle and declares the pdfbox-signing capability, which we used to select the feature in our dependency declarations. This way, a consumer requesting this capability will properly resolve the required transitive dependencies to Bouncycastle!

Conclusion

In this blog post, we’ve highlighted that:

  1. Optional dependencies are not optional: they are always required if you use a specific feature
  2. We can properly model those features in order to group the dependencies together
  3. A consumer can express dependencies on a library, including specific features
  4. We can do this while maintaining compatibility with Maven

It’s also worth noting that there’s no limit to the number of features a library declares. In fact, if you use the Java Test Fixtures Plugin, Gradle automatically declares a feature for each of the test fixtures, features that consumers can decide whether or not to depend on!

To get more information about how to use feature variants in your own builds, please head over to our userguide.

Related Posts