Introducing source dependencies

Calendar
Adam Murdoch
New Features

Table of Contents

Introduction

This post introduces a new Gradle dependency management feature called “source dependencies”.

Normally, when you declare a dependency on a library, Gradle looks for the library’s binaries in a binary repository, such as JCenter or Maven Central, and downloads the binaries for use in the build.

Source dependencies allow you to instead have Gradle automatically check out the source for the library from Git and build the binaries locally on your machine, rather than downloading them.

We’d love to get your feedback on this feature. Please try it out and let us know what works for you and any problems you run into. You can leave feedback on the Gradle forums or raise issues on the Gradle native GitHub repository.

You should be aware that this feature is currently incubating and may change in breaking ways in future releases of Gradle. Please do not use this feature in production until the feature is promoted to stable in a future release.

How can you try it? #

You can find many samples that use source dependencies in the native-samples repository. C++ builds are a natural place to use source dependencies, but this isn’t a C++ specific feature. Source dependencies work for any type of build that use Gradle’s dependency management features, such as Java, Kotlin, or Android builds. You can try this feature out with the latest Gradle 4.10 or a nightly.

Let’s look at an example application. This sample shows an application that uses a library that lives in a separate Git repository.

A source dependency looks the same as a binary dependency in the build file. You declare a dependency on a particular version of a module, such as a library. Here, our sample needs version 1.0 of the “utilities” library:

dependencies {
    implementation 'org.gradle.cpp-samples:utilities:1.0'
}

You also need to tell Gradle where to find the source for the library. This is similar to how you need to tell Gradle where to find binaries by defining some binary repositories in your build, however, the syntax is different. In the settings.gradle file we define a source mapping:

sourceControl {
    gitRepository("https://github.com/gradle/native-samples-cpp-library.git") {
        producesModule("org.gradle.cpp-samples:utilities")
    }
}

Now, when Gradle needs to find a version of the “utilities” library, it will look for a matching tag in the Git repository. Gradle will check out the matching Git tag, build the binaries and make the result available. This works the same way as an included build.

Here’s the result of running gradle assemble on this sample:

> Task :native-samples-cpp-library:list:compileDebugCpp
> Task :native-samples-cpp-library:utilities:compileDebugCpp
> Task :native-samples-cpp-library:list:linkDebug
> Task :compileDebugCpp
> Task :native-samples-cpp-library:utilities:linkDebug
> Task :linkDebug
> Task :installDebug
> Task :assemble

BUILD SUCCESSFUL in 4s
3 actionable tasks: 3 executed

Gradle has cloned the source of the libraries, built the library binaries and then linked the application against these.

You can also declare a dependency on a branch, using a slightly different syntax. This example application demonstrates how to do this.

dependencies {
    implementation('org.gradle.cpp-samples:utilities') {
        version {
            branch = 'release'
        }
    }
}

Source dependencies work for plugins too, but currently need some additional configuration in the settings.gradle in order to work. This example library shows how to do this.

What are source dependencies useful for? #

Source dependencies are useful when the binaries for a module that you use are not available in a binary repository. For example:

  • When you’re working off a branch of a library and binaries have not yet been published for this branch. For example, you may want to use some bug fix from a branch that has not yet been merged or released.
  • When you’re using a library that publishes source and no binaries. This is common for C++ libraries - and applications - where the libraries are released in source form only and you have to build your own binaries.
  • When you’re using a native library that does not publish binaries for your target platform. It is common for a C++ library or application to publish binaries only for certain common operating systems or tool chains. To use the library on other platforms, you have to build your own binaries.
  • When you don’t want to have to wait for another system, such as a CI server, to publish binaries for a dependency. For example, you might want to experiment with changes to several libraries.

Source dependencies make these use cases simpler to implement. Gradle takes care of automatically checking out the correct versions of dependencies, making sure the binaries are built when required. It does this everywhere that the build is run. The checked out project doesn’t even need to have an existing Gradle build. This example shows a Gradle build consuming two source dependencies that have no build system by injecting a Gradle build via plugins. The injected configuration could do anything a regular Gradle plugin can do, such as wrapping an existing CMake or Maven build.

Source dependencies can be combined with the Gradle build cache, to make these use cases efficient when everyone on your team is building the same dependencies from source. The Gradle build cache ensures that once the binaries have been built, they are reused everywhere else.

Source dependencies can be used alongside binary dependencies, and you can mix and match so that some dependencies are built from source and others are downloaded as binaries. Source dependencies also work nicely with composite builds.

How is this different to a composite build? #

Mostly, it’s not. Source dependencies are based on Gradle’s composite build feature and add automatic provisioning of the source code, to help with some specific use cases.

Source dependencies are different to included builds in intent. An included build represents some module that you are currently working on. Dependency resolution will always use binaries from the included build regardless of which version of the module is requested in the build, so that you can see the effect of your changes.

A source dependency, on the other hand, represents some module that you use, but that just happens to be available in source form rather than binary form. Dependency resolution will always select a version of the source dependency that matches whatever is requested in the build. With an included build, you can modify sources and directly see the changes, but with source dependencies, you need to commit and push changes before you can see their effect.

Missing features #

Source dependency support is very much experimental and is missing some important features:

  • Build scans do not yet support builds that use source dependencies.
  • IDE support is mostly not yet available. Most IDE integrations will fail with an error when source dependencies are used by the build. Source dependencies are currently only supported by Gradle’s Xcode integration for C++.
  • Dependency locking does not support source dependencies.
  • Source dependencies also have the same restrictions as included builds.
  • You cannot have a source dependency on a build that contains included builds. However, you can have a source dependency on a build that uses source dependencies.
  • You cannot yet mix source and binary dependencies for a particular module. Each module must come either from a Git repository or from a binary repository, but not both.
  • Only Git repositories are supported.

Roadmap #

We’d like to get your feedback and based on this implement the missing features listed above. Once we’re happy that the DSL and APIs are useful and behave well, we will promote the feature to stable. Let us know what you think on the Gradle forums or the Gradle native GitHub repository.

Discuss

Related Posts