State of Gradle Java 9 Support

This post provides an overview of Gradle’s Java 9 support, touching on runtime, cross-compilation, MRJARs, and Jigsaw modules support. We’ve fielded lots of questions since Java 9 was released last month, and decided it best to answer here.

What Gradle supports as of version 4.2.1

As of Gradle 4.2.1, building and running Java applications using major distributions of JDK 9 such as Oracle JDK9, OpenJDK9 and Azul JDK9 is fully supported. Further, cross-compilation (built by JDK9 but runs on JDK8) is supported.

Some builds will break when upgrading to Java 9, regardless of build tool used. The Java team have made good and necessary changes to the JDK to facilitate better software architecture and security, but this has meant removing access to some APIs. Even if your project is ready, some tools and Gradle plugins have not yet been updated to work with Java 9.

There is no convenience methods for consuming and assembling Multi-Release JARs, but you can take a look at this MRJAR-gradle example if you desire to use them.

Java Modules aka Jigsaw Support

If you’re not yet familiar with the Java 9 Platform Module System, also known as Project Jigsaw, you should read Project Jigsaw: Module System Quick-Start Guide. The motivation and terminology are well explained in The State of the Module System.

A module is defined as “a named, self-describing collection of code and data” whereby packages are treated as code boundaries, and are explicitly exported and required. Non-exported packages are not visible to module consumers, and further 2 modules cannot export the same packages, nor can they have the same internal packages. This means that packages cannot be “split” or duplicated between multiple modules, or compilation will fail.

Here is a guide that shows how to use Java modules with Gradle today. It walks you through the steps necessary to tell Gradle to use the modulepath and not classpath when compiling Java sources and patch modules for testing purposes.

A bottom-up approach (convert libraries with no dependencies first) is recommended if you wish to incrementally convert to Java 9 modules. After all, modules are consumable as regular JARs. Be mindful of automatic modules when “legacy” JARs are added to the modulepath.

Achieving encapsulation with the Java Library Plugin

One of the 2 major goals of the Java 9 module system is to provide better software architecture through strong encapsulation. Gradle 3.4 introduced the Java Library Plugin that enforces strong encapsulation for libraries by separating api dependencies (those meant to be exposed to consumers) from implementation dependencies whose internals are not leaked to consumers.

This, of course, does not eliminate use of Java classpaths as is stated as another goal of Java modules. You can learn about the motivation and usage of the Java Library Plugin in this post. It’s worth noting that the Java Library Plugin is useful for projects using Java 7 and above — you do not need to migrate to Java 9 to have some stronger encapsulation.

Here’s what this means in practice, given this example library:

apply plugin: 'java-library'

name = 'mylibrary'
group = 'com.mycompany'

dependencies {
   api project(':model')
   implementation 'com.google.guava:guava:18.0'
}

Let’s presume we have an application that uses mylibrary.

public class MyApplication {
    
    public static void main(String... args) {
        // This does not compile using 'java-library' plugin
        Set<String> strings = com.google.common.collect.ImmutableSet.of("Hello", "Goodbye");
        
        // This compiles and runs
        Foo foo = com.mycompany.model.internal.Foo();
        
        // This also compiles and runs
        Class clazz = MyApplication.class.getClassLoader().loadClass("com.mycompany.model.internal.Foo");
        Foo foo = (Foo) clazz.getConstructor().newInstance();
    }
}

You can see that you get some of the benefits by adopting the Gradle’s Java Library plugin. If you are migrate to Java modules, you can use this rough mapping:

  • implementation dependency => requires module declaration
  • api dependency => requires transitive module declaration
  • runtimeOnly dependency => requires static module declaration

Next Steps

Stay tuned for updates on first-class Java modules support in Gradle.

You can use the Building Java 9 Modules guide to learn how to use Java modules with Gradle today.

Join Gradle at KotlinConf 2017

Kotlin is the #1 rising Gradle “related topic” according to Google Trends. We want to improve user experience developing Kotlin applications with Gradle, and that requires great guides — but we need your help! Tell us which unwritten Kotlin Gradle guide you think would be most useful, and you’ll have a chance to get a free ticket to KotlinConf! See instructions below.

KotlinConf 2-3 Nov 2017, San Francisco, Pier 27

We are proud to be a silver partner for KotlinConf, which is just 3 weeks away! Hans Dockter will be presenting Building Kotlin Applications at Scale and some of the Gradle Kotlin DSL team will be on-site to talk Kotlin with you.

About the KotlinConf 2017 ticket raffle

We are giving away 2 standard tickets to the conference. You’ll be responsible for getting to the conference, so please don’t accept a ticket if you cannot attend.

All you need to do is file an issue on the gradle/guides GitHub repo with a unique and useful suggestion for a Gradle guide around building Kotlin with Gradle. For inspiration, you can check out the proposal for the Building Kotlin JVM Libraries guide, which was published recently.

We’d be happy to get lots of good ideas from you, but we’ll enter your name once. Winners will be chosen at random on Monday, October 16th. We’ll contact you if you’ve been selected, and you’ll have 3 days to confirm you can attend.

UPDATE: Thank you everyone! The contest is now closed.

If you’re interested in contributing to the Kotlin/Gradle community, just adding 👍 s on ideas you think would be good is helpful, but even more so submitting a PR with a sample or typo fix is most welcome. Remember the goal: make Kotlin development awesome for everyone!

Good luck! We hope to see you at KotlinConf!

State and future of the Gradle Software Model

We’ve received many inquiries about the status and direction of Gradle’s Software Model, especially from users building native libraries and applications.

In this blog post, we will explain the current state and future of the Software Model, and in particular how it relates to native development with Gradle. A lot of exciting improvements are planned for the remainder of 2017; see the roadmap below.

Situation with the Software Model

In a nutshell, the Software Model is a very declarative way to describe how a piece of software is built and the other components it needs as dependencies in the process. It also provides a new, rule-based engine for configuring a Gradle build. When we started to implement the Software Model we set ourselves the following goals:

  • Improve configuration and execution time performance.
  • Make customizations of builds with complex tool chains easier.
  • Provide a richer, more standardized way to model different software ecosystems

As we were developing the Software Model, the Gradle engineering team constantly tried to dogfood the concepts into existing software ecosystems. The Gradle plugins for building native applications is currently fully-based on the Software Model. Similarly, experimental Software Model-based plugins were developed for ecosystems like Android and Java.

Gradle adoption in the native ecosystem is picking up, and so is our investment. Since it’s inception, Gradle’s native support has proved itself as a welcome alternative for builds using Make. With its declarative and expressive model, support for different tool chains and platforms as well as features like parallel compilations, it offers a revolutionary way for building native libraries and applications.

It took us longer than expected to evolve the new configuration and Software Model and make it as powerful as the current Gradle model for Java and Android. Meanwhile, Gradle adoption skyrocketed, there are many complex builds out there using the current model and a vibrant ecosystem of 1500+ community plugins as well. We underestimated the complexity for those builds and plugins to migrate to the new model and saw understandable resistance with many of our partners to undergo this migration.

In hindsight, the scope of the new software and configuration model was too big. That is why at the Gradle Summit 2016, Hans Dockter announced that we were backporting many of its features to the current model. One year later, most of the features for the Java and Android ecosystem have been backported. This includes variant-aware dependency resolution and separation of API and implementation for Java components. Those features were game changers in terms of work avoidance and performance. Furthermore, we found other ways to drastically improve Gradle configuration performance, with more to come. There is no longer any need for a drastic, incompatible change in how Gradle builds are configured.

A way forward

You may therefore be wondering what is happening to the Software Model. We’re in the process of porting the configuration DSL of the native support to the current model. So the declarative nature and strong modelling language will be the same. The rule engine that was part of the Software Model will be deprecated. Everything under the model block will be ported as extensions to the current model. Native users will no longer have a separate extension model compared to the rest of the Gradle community, and they will be able to make use of the new variant aware dependency management.

What does the roadmap look like? Here are the areas of focus until the end of the year:

  • Current model support for native. New sets of plugins based on the current model are in development and are improved with every nightly release. They still need more work to achieve feature parity and stability, but already provide a lot of functionality. Try them out and give us feedback.
  • Parallel-by-default for compile and link tasks. Performance improvements are planned for the native ecosystem by enabling parallelism-by-default to compile and link tasks. This will have a positive impact on everyone building native with Gradle.
  • Transitive dependency resolution. We are porting this powerful feature from our JVM ecosystem to help native developers declare rich dependencies between native projects.
  • New native plugins on current model. Our plan is to have plugins that have most of the functionality of the Software Model plugins and will also have substantial new features like build caching and external source dependencies for native.
  • Improved tool chain support. We are looking at ironing out some of the wrinkles with tool chain declaration which is particularly important for embedded development.

For the most complete and up-to-date progress, we recommend having a look at the gradle-native project, the home for the native ecosystem feature planning.

User migration from Software Model plugins to the new ones will be pretty seamless. All core native tasks will be reused and the tool chain concept will be ported to the current model. We expect that a lot of your build logic can be simply reused. We will support the Software Model-based plugins for an extended period of time to ensure everyone has a successful migration.

If you are currently using, or are planning to use, Gradle to build native projects, by all means keep doing so. Gradle’s native support has proven time and time again to be more performant, flexible, and easier to use than currently available tools.

Exciting things afoot

Today, we’re working on IDE integration and XCTest support, with out-of-the-box HTML report generation and full build scan support. Tool chain definition will also be improved to allow easier integration with alternative tool chain families; this is especially exciting for users invested in the embedded world.

For multi-repository developers, you will be happy to learn that composite builds will work for all native projects.

C++ Build Scan

The new plugins will integrate with the Kotlin DSL which gives Gradle users proper IDE support including auto-completion and refactoring.

We will first implement the complete workflow for native development in the current model without any customization - i.e. no platforms, build types or tool chains configuration. By workflow we mean everything related to building binaries, as well as testing, packaging, deploying and integration with your favorite IDE. At first, the workflow will work for the most common cases. In the subsequent releases, we will proceed by adding customization incrementally to the whole workflow.

Community involvement

Our native community is one of the most active and engaged on the forum, and we want to encourage and grow that engagement even more. Please keep helping each other find the answers you are seeking in the forum, but also engage with us by, trying the various native sample projects, subscribing to the gradle-native project and filing issues, voting on issues that are most important to you, and even consider submitting pull requests if you’re excited to roll up your sleeves and pitch in.

The best ways to stay passively up-to-date on native platform support are to subscribe to the monthly newsletter or more frequent announcements on Twitter.

We look forward to working with you to develop the best native build tool!

Blazing Fast Android Builds

At Google I/O today, the Android Studio team released the first preview version of the Android Gradle plugin 3.0, based on Gradle 4.0 M2. It brings major performance improvements, especially for builds with plenty of subprojects. In this blog post, we will explain what you can expect from this preview version and how the Android Studio and Gradle team achieved these improvements. Before diving into this let’s look back at what goals led to the creation of the current Android build system.

The Complexity of Mobile Development

Developing mobile applications is inherently more complex than building traditional web or server applications of similar size. An app needs to support a wide array of devices with different peripherals, different screen sizes, and comparatively slow hardware. The popular freemium model adds another layer of variety, requiring different code paths for free and paid versions of the app. In order to provide a fast, slim app for every device and target audience, the build system needs to do a lot of the heavy lifting up front.

To improve developer productivity and to reduce runtime overhead, the Android build tools provide several languages and source generators, e.g. Java, RenderScript, AIDL and Native code. Packaging an app together with its libraries involves highly customizable merging and shrinking steps. The Android Studio team was faced with the challenge of automating all of these without exposing the underlying complexity to developers. Developers can focus on writing their production code.

Last but not least, developers expect a build tool to manage their dependencies, be extensible and provide deep IDE integration.

Gradle is ideally suited for those challenges and the Android Studio team created a fantastic Android build tool on top of the Gradle platform.

The performance challenge

No matter how elegant and extensible the plugin and no matter how seamless the IDE integration, when things take too long, developers become unproductive and frustrated. The Android Studio team has made steady progress on performance over the last years. The emulators became much faster, the time to deploy an app decreased by orders of magnitude with Instant Run and other improvements. These steps have now exposed the build itself as the final bottleneck. The Android Studio team and the Gradle team have continuously improved the performance of the plugin and the platform, but so far this has not been enough. Fundamental design issues preventing great performance.

So Gradle Inc. and Google teamed up in late 2016 to get this situation under control. The work was split up into three areas:

  • General improvements to Gradle and its Java support: Faster up-to-date checking, compile avoidance, stable incremental compilation and parallel dependency downloads.
  • General improvements to the Android tools, like dex and code shrinking, including incremental dexing.
  • New APIs for variant aware dependency management in Gradle and an Android plugin that uses these new APIs.

The latter allowed the Android Studio team to finally get rid of a lot of inefficient workarounds that they had to build because of these missing APIs.

To understand why variant aware dependency management is so important, imagine you have an app which depends on a single library. Both of them support ARM and x86 architectures, both have a free and a paid version and both of them can be built for debug and production. This creates a total of 8 variants. But at any given point in time, a developer is only working on exactly one variant, e.g. the “free x86 debug” variant.

Up until now, the Android plugin had to inspect the app’s dependencies very early in the build lifecycle to select the right variant of the library to build. This early phase is called configuration time, during which Gradle determines what tasks it needs to run in what order. More work at configuration time means slower builds no matter which tasks the user selected. It also affects how long it takes to synchronize the build with the IDE. The Android plugin’s eager dependency inspection lead to a combinatorial explosion of configuration time as more subprojects were added to a build.

This completely changes with Gradle’s new variant aware dependency management. The Android plugin can now provide matching strategies for the different variant dimensions (like product flavor and build type), which Gradle uses during dependency resolution to select the correct variant of the upstream library. This completely removes the need to resolve dependencies at configuration time and also allows the Android plugin to only build the parts of the library that the app needs.

In a particularly large app with 130 subprojects, the time it took to configure the project dropped from 3 minutes to 10 seconds with Android 2.3 tools to under 2 seconds with Android 3.0. The clean build time dropped from over 5 minutes to about 1 minute. The effect on incremental builds is dramatic when combined with new compile avoidance functionality. Making a single-line change and assembling the project is down to about 9 seconds. For monolithic projects these numbers won’t be as impressive, but they show that the build system now works very efficiently with modularized apps.

Android performance comparison

Last but not least, the Android Studio team is going to make the Android plugin 3.0 compatible with the Gradle build cache. The build cache allows build outputs to be reused across clean builds and across machine boundaries. This means that developers can reuse build outputs generated by CI and build pipelines can reuse results from earlier stages. It also speeds up switching between feature branches on developer machines. Preliminary tests are promising, the clean build for the large Android app mentioned above dropped from 60s to about 20s when using the cache.

Give it a try

The Android Studio team has written up a comprehensive migration guide. There may be compatibility issues with community plugins, as many of them depended on internals that work differently now.

If you are developing Android projects, give the preview a try and tell us how much your build times improved out of the box. Try modularizing your app a bit more and splitting api and implementation dependencies for even bigger performance gains. You can use Build Scans and its timeline view to get deep insight into the performance of your build, which tasks were executed and how long they took.

If you are an Android plugin author, the new version might require some changes for your plugin to stay compatible. Please file an issue if you encounter any problems while migrating.

What’s next?

You can expect more improvements on the Gradle side. For instance, we are currently working on parallel task execution by default.

It is also safe to expect more performance smartness from the Android Studio team including Android Studio optimizations to do as little work as possible when syncing the project. The Gradle and Android Studio teams are collaborating on this as well.

Support for community plugins will improve as the alpha versions mature and plugin authors adjust to it. The more people provide feedback, the faster these great improvements can be released as stable.

Introducing Gradle Build Cache Beta

Introduced in Gradle 3.5 to reduce build time.

What does it do?

The build cache reuses the outputs of Gradle tasks locally and shares task outputs between machines. In many cases, this will accelerate the average build time.

The build cache is complementary to Gradle’s incremental build features, which optimizes build performance for local changes that have not been built already. Many Gradle tasks are designed to be incremental, so that if the inputs and outputs of the task do not change, Gradle can skip the task. Even when the task’s inputs have changed, some tasks can rebuild only the parts that have changed. Of course, these techniques only work if there are already outputs from previous local builds. In the past, building on fresh checkouts or executing “clean” builds required building everything from scratch again, even if the result of those builds had already been created locally or on another machine (such as the continuous integration server).

Now, Gradle uses the inputs of a task as a key to uniquely identify the outputs for a task. With the build cache feature enabled, if Gradle can find that key in a build cache, Gradle will skip task execution and directly copy the outputs from the cache into the build directory. This can be much faster than executing the task again.

In particular, if you’re using a continuous integration server, you can configure Gradle to push task outputs to a shared build cache. When a developer builds, task outputs already built on CI are copied to the developer’s machine. This can greatly improve the developer’s local build experience.

When using the local build cache, instead of rebuilding large parts of the project whenever you switch branches, Gradle can skip task execution and pull the previous outputs from the local cache.

How does it work?

A cacheable Gradle task is designed to declare everything that can affect the output of the task as an input. Gradle calculates a build cache key by hashing over all of the inputs to a task. That build cache key uniquely identifies the outputs of the task. This is an opt-in feature for each task implementation, so not every task is cacheable or needs to be. Several built-in Gradle tasks (JavaCompile, Test, Checkstyle) have caching enabled to speed up the typical Java project.

The build cache key for a task takes into account:

  • The values of all inputs defined by the task via annotations (e.g. @InputFiles) or the runtime TaskInputs API.
  • The contents (and relative paths) of any file inputs.
  • The classpath of the task, which includes any plugins and Gradle version used.
  • The classpath of any task actions, which can include the build script.

When the build cache feature is enabled, Gradle will check if any of the configured build caches contain a match for the task’s build cache key when a task is not up-to-date. If Gradle does not find a match, the task will be executed as normal. After execution, Gradle will gather all of the task’s outputs and push them to the build caches, if configured to do so. If Gradle does find a match, the task’s outputs are deleted and the previous outputs are copied into the output directories.

Does it help?

We have been using the build cache for the Gradle CI builds since November 2016. We also have some partners who have been trying the build cache in their builds. We can’t share their data directly, but they’ve seen similar improvements in CI and developer builds as we have. On average, we see a 25% reduction in total time spent building each commit, but some commits are even better (80% reduction) and the median build saw a 15% reduction.

Stage 3 %-Improved

Here’s another look at the number of minutes spent between the cached and non-cached builds for Gradle. You can see how the reductions translates into about 90 minutes saved in a 360 minute build for us.

Stage 3 comparison

The build cache is a generic feature that avoids re-executing a task when it can, so builds large and small can benefit in some way. The structure of your project will influence how much you can gain overall. If your project consists of a single monolithic module, Gradle has other features that may also help, such as incremental compilation or composite builds. We’ll provide more information about how to get the most out of the build cache in a future blog post and at the Gradle Summit.

Make your build faster today

The Gradle 3.5 release is the first release to include the build cache feature.

We expect that the build cache feature will have general availability in the next release, but we would like for every project to give the build cache beta a try. To do that, we’d like you to try 3 things for us.

1) Try it on a simple project

After upgrading to 3.5, pick a simple Java project and run:

gradle --build-cache clean assemble
gradle --build-cache clean assemble

The second build should be faster because some task outputs are reused from the first build. These outputs will be pulled from your local build cache, which is located in a directory in your GRADLE_USER_HOME.

2) Try to share outputs between machines

To use a shared, remote cache, we provide a recommended configuration that uses your continuous integration builds to populate a shared build cache and allows all developers to pull from that build cache.

You’ll need a remote build cache backend to share between developers. We provide a build cache node docker image which operates as a remote Gradle build cache, and can connect with Gradle Enterprise for centralized management. The cache node can also be used without a Gradle Enterprise installation with restricted functionality.

3) Give us feedback

If you have feedback, we’d love to hear it. If you have a build scan you can share, that’s even better.

We’re excited to get the Gradle Build Cache feature out for feedback in Gradle 3.5, but we know there’s more we need to do to make the build cache stable and performant. We have some known issues that you should check before raising new issues on GitHub.

At this time, we don’t recommend that you leave the build cache enabled for production builds without understanding the risks. There are known issues that can cause your builds to fail or produce incorrect output, but your feedback on the types of problems or successes are very valuable to maturing the build cache feature. You can configure the build cache in your build and enable it on a trial basis by setting org.gradle.caching=true or running with --build-cache without impacting all builds.

For dogfooding the build cache for Gradle, we used a separate CI job to run a build with the build cache enabled. This allowed us to compare the build times with and without the build cache for the same set of changes.

Thanks and roadmap

After trying the build cache, you’ll probably have some questions about why more parts of your build are not cacheable. Regardless of the build cache backend you are using, Gradle Enterprise 2017.2 comes with features to understand build cache usage and behavior by collecting data, whether the build cache is enabled or not. Build scans keep track of the reason that a task was not cached. A task might not be cached if it has particular problems, like if it has no outputs or cacheability is not enabled for it. You can search the build scan timeline for each of these reasons.

In future versions of Gradle and Gradle Enterprise, we’ll collect more information related to the build cache and task cacheability, to make it easier to diagnose build failures or poor build cache performance.

For the next release, Gradle 4.0, we intend to focus on making the build cache safe to enable for all well behaved builds and providing feedback for situations where Gradle cannot safely cache outputs from a task. This also means we’ll be providing a well-behaved local build cache and several validation checks.

For the releases following that, we intend to spend time on expanding our documentation and Gradle guides to make it easier for you to cache more tasks and develop cache-friendly tasks.

Thanks for your continued help and support. Please consider making your build faster with the build cache with the three steps we outline above.