Introducing Gradle Module Metadata

Gradle Module Metadata reaches 1.0 in Gradle 5.3 and here we explain why you should be as excited as we are!

Gradle Module Metadata was created to solve many of the problems that have plagued dependency management for years, in particular, but not exclusively, in the Java ecosystem. It is especially important because POM files (or Ivy files) are simply not rich enough to describe the reality of software nowadays where you might need to distinguish between binaries for different platforms or choose one particular implementation of an API when more than one is available.

We will describe more examples later in this post. Some issues may have workarounds, but where those workarounds are hacky ones or even error prone. For example, did you realize that these are problematic: using classifiers for different Java versions, exclusions to avoid a particular logger binding, or adding first level dependencies just because you need to override a particular version?

Gradle Module Metadata 1.0 is an answer to those problems and the first step towards better dependency management throughout our industry.

What does it allow in practice?

Have you ever cursed when you had both guava-jdk5 and guava-jdk8 on the classpath, and your application only worked because of lucky ordering of entries? Have you ever faced the problem of having different SLF4J bindings and only noticed at runtime? That’s because these libraries have different variants that can’t be described properly by existing metadata formats. How is a build tool even supposed to understand the difference between a jdk8 JAR, a sources one, or even an all one? Gradle Module Metadata is designed to explain the difference in such a way that consumers can express more precise requirements. For example, a consumer can specifically ask for something they can use with JDK 8. And in the case of SLF4J, the build tool will recognize that the Log4J binding is mutually exclusive with the java.util.logging one.

The whole idea is to support variant-aware dependency management, which is based on a description of the variants of a component — such as the main binaries, source packages, platform-specific binaries, and so on — and their corresponding dependencies.

Some of our partners have been using Gradle metadata for months now. Kotlin native, for example, is using Gradle Module Metadata to represent the different binaries you can get when compiling a Kotlin project to different architectures. Google is using variant-aware dependency management, but lacked an “external model” for it. Gradle Module Metadata is that external model and will allow the proper publication of Android Archives (AARs).

Those are just examples, but these issues and many more can be solved by leveraging Gradle metadata. As more library authors adopt Gradle Module Metadata, our industry will solve more problems as a whole.

How Gradle Module Metadata affects you

Gradle Module Metadata 1.0 enables fine-grained dependency resolution for all Gradle users. Starting with Gradle 5.3, if you are a consumer and the library you use has Gradle metadata published, Gradle will automatically consume any Gradle metadata that is published to Maven or Ivy repositories.

However, Gradle 5.3 will not automatically publish it by default — that will come in 6.0. You can publish Gradle Module Metadata today, but you have to opt into its publication by using either the Maven Publish or Ivy Publish plugins and enabling the experimental publishing feature by adding the following line to your settings script:

settings.gradle(.kts)

enableFeaturePreview("GRADLE_METADATA")

How does Gradle Module Metadata affect Maven or Ant+Ivy builds?

Nothing changes for Maven and Ivy consumers: if you have opted into publishing Gradle Module Metadata, the corresponding file is published alongside the POM file (if you’re publishing to a Maven repository) or Ivy file (if you’re publishing to an Ivy repository). Be aware that the mapping from a build component to Maven and Ivy metadata is lossy: for example you don’t know what Java version was used to build something, so it makes it impossible for consumers to know if they are compatible or not beforehand. Another example is when you use Gradle specific features like rich versions. We do our best to map them to concepts in Maven or Ivy, but information will still be lost in the process due to the limitations of their metadata formats.

Note that Gradle 5.2 introduced warnings emitted by the publishing plugins when it knows the mapping to be lossy or problematic for other build tools.

Gradle Module Metadata is a JSON file whose extension is .module. Each file describes a single software component with zero or more variants. Here’s the content of an example metadata file for a “com.acme:client:1.0-SNAPSHOT” component with several variants:

{
  "formatVersion": "1.0",
  "component": {
    "group": "com.acme",
    "module": "client",
    "version": "1.0-SNAPSHOT",
    "attributes": {
      "org.gradle.status": "integration"
    }
  },
  "createdBy": {
    "gradle": {
      "version": "5.3",
      "buildId": "4wqjtkcv2fbmjjsewyu66wbvfq"
    }
  },
  "variants": [
    {
      "name": "apiElements",
      "attributes": {
        "org.gradle.dependency.bundling": "external",
        "org.gradle.jvm.version": 11,
        "org.gradle.usage": "java-api-jars"
      },
      "dependencies": [
        {
          "group": "com.mycompany",
          "module": "core",
          "version": {
            "requires": "2.5"
          }
        }
      ],
      "files": [
        {
          "name": "client-1.0-SNAPSHOT.jar",
          "url": "client-1.0-SNAPSHOT.jar",
          "size": 539,
          "sha1": "1f94fe53d33babdc9de537bb3a0108dbc0e25e4b",
          "md5": "6364cdd9923e1eda9b328bc80f93969c"
        }
      ]
    },
    {
      "name": "runtimeElements",
      "attributes": {
        "org.gradle.dependency.bundling": "external",
        "org.gradle.jvm.version": 11,
        "org.gradle.usage": "java-runtime-jars"
      },
      "dependencies": [
        {
          "group": "org.apache.commons",
          "module": "commons-lang3",
          "version": {
            "requires": "3.8"
          }
        },
        {
          "group": "com.mycompany",
          "module": "core",
          "version": {
            "requires": "2.5"
          }
        }
      ],
      "files": [
        {
          "name": "client-1.0-SNAPSHOT.jar",
          "url": "client-1.0-SNAPSHOT.jar",
          "size": 539,
          "sha1": "1f94fe53d33babdc9de537bb3a0108dbc0e25e4b",
          "md5": "6364cdd9923e1eda9b328bc80f93969c"
        }
      ]
    },
    {
      "name": "shadowApiElements",
      "attributes": {
        "org.gradle.dependency.bundling": "shadowed",
        "org.gradle.usage": "java-api"
      },
      "files": [
        {
          "name": "client-1.0-SNAPSHOT-all.jar",
          "url": "client-1.0-SNAPSHOT-all.jar",
          "size": 601730,
          "sha1": "9b70e54ffdce0541701d8f855bf75e059857eb0c",
          "md5": "3499bb6d9ccf86283854a5550135ea4a"
        }
      ]
    },
    {
      "name": "shadowRuntimeElements",
      "attributes": {
        "org.gradle.dependency.bundling": "shadowed",
        "org.gradle.usage": "java-runtime-jars"
      },
      "files": [
        {
          "name": "client-1.0-SNAPSHOT-all.jar",
          "url": "client-1.0-SNAPSHOT-all.jar",
          "size": 601730,
          "sha1": "9b70e54ffdce0541701d8f855bf75e059857eb0c",
          "md5": "3499bb6d9ccf86283854a5550135ea4a"
        }
      ]
    }
  ]
}

This file declares 4 variants, and attributes let the build tool know what they are used for. In particular you will see here that there are 2 variants for “API” and 2 variants for “runtime”, when usually you only see one for each. The reason is that this particular component declares an additional variant where the dependencies are shadowed (fat jar). This gives the opportunity for a consumer to decide whether it wants dependencies as individual jars, or just the fatjar variant of the library.

If you are interested in more technical details, please refer to the Gradle module metadata specification 1.0

Early adopters are welcome, feel free to give your feedback!