Evolving the Gradle API to reduce configuration time

July 9, 2018

Sterling Greene

New Features

Introduction

This post introduces a new API for declaring and configuring Gradle Tasks in build scripts and plugins. We intend for this new API to eventually replace the existing API because it allows Gradle to avoid configuring unnecessary build logic. Use of the new API will become the default recommendation soon, but the existing API will go through our usual deprecation process over several major releases.

We are asking early adopters to try out the new Gradle Tasks API to iron out any issues and gather feedback. We’ve created a new user manual chapter to give a quick introduction to the feature and explains some guidelines for migrating your build to use the new API.

We welcome any feedback you may have about this API. You can leave feedback on this Gradle issue.

Preview of new Gradle Tasks API in Gradle 4.9 #

One of the major differences between the existing and new Gradle Tasks API is whether or not Gradle spends the time to create Task instances and run configuration code. The new API allows Gradle to delay or completely avoid configuring tasks that will never be executed in a build. For example, when compiling code, Gradle does not need to configure tasks that run tests.

The time spent creating and configuring tasks that are never used is a contributor to overall Gradle configuration time. Configuration time affects every build, so making it faster is good for everyone.

The existing API also eagerly creates Task instances as soon as possible. This can make the ordering of plugins more fragile. The new API is intended to make the relationship between a Task and something that needs it more explicit by passing around a Provider for a Task.

We have also seen in some builds that configuring particular tasks are very expensive because they reach out to webservices, they run git status or they need to parse something. If those tasks are rarely used, every build is paying the price to configure that task. The new API allows plugin or build authors to declare the expensive tasks so that they’re only configured when necessary.

Later Gradle releases will provide more details and samples for the new API once it becomes the default recommendation.

Measured impact to Gradle builds #

For the last few months, we have been dogfooding the new Gradle Tasks API in the gradle/gradle build, Gradle Enterprise and a typical large project.

Over that time, we’ve made several changes to reduce configuration time in the Gradle build from 1.7s to 1.4s on our Linux performance agents. Part of these improvements come from the new API, the Gradle build still configures several hundred tasks unnecessarily, so we believe that we can reduce this number further.

Below, we’ve plotted the improvements directly related to the new API. We’ve measured this on a mid-2014 MacBook Pro (2.6 GHz Intel Core i5, 16GB RAM) with several different projects.

The perf-enterprise-large project is a generated project we use in our performance tests that has nearly 350 Java modules. When configuring this build with the existing API, Gradle creates and configures over 10000 tasks. With the new API, Gradle only configures 349 (97% of all tasks are avoided). Changes in a future version of Gradle will make this 1 task. From the graph, we can see that the average configuration time goes from 936ms to 703ms (-233ms).

The second group of plots is for the gradle/gradle build. Without the new API, Gradle would create and configure over 6300 tasks. We have more work to do, but we avoid creating and configuring 90% of all tasks. Our average configuration time dropped from 1325ms to 1117ms (-208ms).

For our closed-source Gradle Enterprise project, we’ve only just started converting it to the new API. We only avoid about 64% of all tasks. Our average configuration time dropped from 1636ms to 1467ms (-169ms).

We expect other builds to see a similar reduction when many tasks are avoided when they are not needed. Some builds may see a larger impact if they are much larger (hundreds of subprojects) or configure very expensive tasks. We’re working with prominent plugin authors to use the new API, so most builds will see the task avoidance benefits.

Why a new API? #

Unfortunately, the existing API could not be adapted to meet our new requirements:

• Task instances should not be created immediately. Many existing APIs return Task and we could not break binary compatibility.
• Tasks should not be created or configured unless needed. Many existing APIs could be adapted to delay creation or configuration, but this would have silently broken many builds in hard to diagnose ways.

The new API is designed to work along side the existing API, so tasks created with the existing API are visible to the new API and vice versa. Builds can gradually migrate to the new API, but mixing the existing API will negate some of the benefits of the new API. Any existing API that requires a Task instance will force tasks created by the new API to be created as if they were created via the existing API.

The new API is also intended to be similar enough to the existing API that migration is easy.

Migrating from the existing Gradle Tasks API to the new API #

Our user manual chapter provides a reference for mapping existing API to new API.

As a quick and dirty reference:

• tasks.create(...) becomes tasks.register(...)
• tasks.withType(SomeType) { } becomes tasks.withType(SomeType).configureEach { }
• tasks.all { } becomes tasks.configureEach { }
• tasks.getByName(...) becomes tasks.named(...)
• For the Groovy Gradle DSL, there is not a replacement for task foo(...) { } that uses the new API.

Please keep in mind that configuration blocks that are used with the new API are not guaranteed to execute all of the time.

In the existing API, if you were to use tasks.withType(...):

plugins {
    id "java"
}

tasks.withType(JavaCompile) { javaCompile ->
    println "Hello, " + javaCompile.name
}

Running gradle help or gradle build or gradle compileJava would log “Hello, compileJava” and “Hello, compileTestJava”. The tasks that are configured do not change depending on what needs to be executed.

If we instead use the new API tasks.withType(...).configureEach:

plugins {
    id "java"
}

tasks.withType(JavaCompile).configureEach { javaCompile ->
    println "Hello, " + javaCompile.name
}

We see the behavior is very different. Gradle configures more or less of the build depending on what is required for execution.

• Running gradle help will not log any “Hello” messages. Gradle is able to avoid configuring the JavaCompile tasks because they are never executed.
• Running gradle build will log “Hello, compileJava” and “Hello, compileTestJava”. Gradle has to configure the JavaCompile tasks because they’re both executed.
• Running gradle compileJava will log only “Hello, compileJava”. Gradle only has to configure the compileJava task because it’s executed. Task compileTestJava is not executed, so it is not created or configured.

Your Feedback Wanted #

Included in Gradle Enterprise 2018.3, you can see the progress of using the new task API in your build scans. For many of the built-in Gradle plugins, we have switched to using the new APIs. For builds using Java, you may see a large number of tasks that are never configured already.

This Gradle Tasks API is a work-in-progress, but we believe that it is complete enough to replace the existing API in all scenarios. We plan to incorporate any feedback into the next release, which will contain more documentation and examples for using the new API. Our intention is to make this API production ready as soon as possible.

Please try this API out in your plugins and builds and let us know what you think. We’ve included a section in the user manual to collect data for a performance comparisons between the existing and new APIs. We would love to see your results and hear what you think works well, what’s confusing, and what is missing that would block you from using this new API over the existing one. You can leave feedback on this Gradle issue.

Discuss