November 15th Bay Area Gradle Users Meetup

Everyone has an opportunity to engage with the Gradle team online through a variety of channels, but nothing beats meeting people face to face. If you are around the Silicon Valley area on November 15th, you can meet three of the team at the Bay Area Gradle Users meetup along with an expert user and build master from LinkedIn.

We have two great talks lined up, the first of which introduces you to an exciting new feature within Gradle—composite builds—while the second shows you how to get more out of your build scans with custom data.

Details:

  • Who? Hans Dockter, Szczepan Faber (LinkedIn), Luke Daley, Etienne Studer
  • When? 6:00pm PT on November 15, 2016
  • Where? LinkedIn, 605 W. Maude, Sunnyvale, CA
  • RSVP: If you plan to attend, please register beforehand via Eventbrite

We hope to see you there!

Hans Dockter and Szczepan Faber on Composite Builds

Many of you will be familiar with Gradle’s multi-project build support which allows you set up dependencies between projects, e.g. where the output of one project—say, a JAR file—is required by another. But this only works if the projects are part of the same multi-project build.

In this talk, Hans and Szczepan will explain how Gradle’s new support for composite builds extends this behavior to projects that are otherwise independent of one another. Want to test a local fix to a library one of your projects depends on? Now you can—without having to publish a new version of that library. Composite builds also enable structuring your projects in new ways, since you no longer need to keep all projects in one repository or directory hierarchy.

Hans won’t just talk about the topic, he’ll also show you how the feature works in practice. You’ll come away with a firm understanding of the value of composite builds and how you might put them to use in your own projects.

Luke Daley and Etienne Studer on Customizing Build Scan Data

Build scans already provide deep insights into your build by reporting key metrics and data. These are incredibly useful on their own, but Gradle is designed around the understanding that no two builds are the same. That’s why build scans allow you to add custom tags, links, and values from your builds. These custom annotations can help you bring out insights that are very specific to your build and to the environment in which your build is run.

In this talk, Luke and Etienne will show examples of how you can use this feature to add extra value to your build scans. The aim is to sow the seeds of inspiration for your own builds using a feature that you might otherwise overlook.

The Road to Gradle Script Kotlin 1.0

Five months ago we announced the first pre-release of Gradle Script Kotlin, and we thought now would be a good time to review the progress we’ve made since. We have shipped eight additional pre-releases during that time, and the road to 1.0 is looking clearer every day. So let’s take a look at the ground we’ve covered so far and where we’re going from here, shall we?

v0.1.0

As you may recall, this is what our hello-world sample looked like at the time of our first release:

import org.gradle.api.plugins.*
import org.gradle.script.lang.kotlin.*

apply<ApplicationPlugin>()

configure<ApplicationPluginConvention> {
    mainClassName = "samples.HelloWorld"
}

repositories {
    jcenter()
}

dependencies {
    "testCompile"("junit:junit:4.12")
}

Oh, that annoying org.gradle.script.lang.kotlin.* import! The publicly condemned, IDE unfriendly, string-based "testCompile" dependency configuration! And of course—for those souls brave enough to have tried them—the infamous generateKtsConfig and patchIdeaConfig tasks required to get Kotlin-based build scripts working in IDEA. These were early days, no doubt, and they brought with them a few rough edges.

But despite its flaws, the programming language and IDE experience in 0.1.0 was already so good it got us hooked. As for the rough edges, we could already see ways to smooth them out, which led to the release of 0.2.0 one month later.

v0.2.0

With implicit imports and a tooling-friendly alternative to string-based dependency configurations, hello-world 0.2.0 started looking clean and concise:

apply<ApplicationPlugin>()

configure<ApplicationPluginConvention> {
    mainClassName = "samples.HelloWorld"
}

repositories {
    jcenter()
}

dependencies {
    testCompile("junit:junit:4.12")
}

Seamless project imports meant that Kotlin-based builds in IDEA started working out of the box, and the days of mistyping generateKtsConfig and patchIdeaConfig were no more.

Perhaps most importantly, 0.2.0’s support for build script dependencies and external plugins made Gradle Script Kotlin a viable choice for many real-world projects.

v0.3.0

0.3.0 was a major milestone for the project, as it was the first version to be included in a production Gradle distribution—Gradle 3.0, no less!

And 0.3.0 was all about that Kotlin! The new Kotlin 1.1-M01 compiler, support for Kotlin-based plugins and buildSrc directories plus some sugary Kotlin-Groovy interoperability primitives:

gradle.buildFinished(closureOf<BuildResult> {
    println("$action finished") // $action refers to BuildResult.getAction()
})

With Gradle 3.0 out the door, the #gradle channel of the public Kotlin Slack saw an increase in participation which helped us greatly in prioritizing the work that would come next.

v0.3.1

We noticed people struggling with the lack of a more type-safe and IDE-friendly way of configuring dependencies, so 0.3.1 came with a much-improved dependencies DSL:

dependencies {

    default(group = "org.gradle", name = "foo", version = "1.0") {
        isForce = true
    }

    compile(group = "org.gradle", name = "bar") {
        exclude(module = "foo")
    }

    runtime("org.gradle:baz:1.0-SNAPSHOT") {
        isChanging = true
        isTransitive = false
    }

    testCompile(group = "junit", name = "junit")

    testRuntime(project(path = ":core")) {
        exclude(group = "org.gradle")
    }
}

The upgrade to Kotlin 1.1-dev-2053 notably enhanced the performance of code assistance within IDEA and thanks to a valuable member of the community the first Gradle Script Kotlin Android sample was published.

v0.3.2

With 0.3.2 we decided to tackle the dreaded it problem head-on via runtime code generation of Kotlin extensions. What is the dreaded it problem? Take the use of copySpec as an example. Prior to 0.3.2, one would have written:

copySpec {
    it.from("src/data")
    it.include("*.properties")
}

This syntax didn’t read very well, and was a departure from the fluid, readable DSL Gradle has long been known for. But never fear—with 0.3.2 it was gone:

copySpec {
    from("src/data")
    include("*.properties")
}

v0.3.3 and v0.4.0

The recently-released versions 0.3.3 and 0.4.0 shipped the first of a series of improvements to multi-project builds including the ability to define custom build logic using Kotlin in buildSrc.

0.4.0 is available now and will ship with the forthcoming Gradle 3.2 distribution.

Toward v1.0.0

What’s next, you ask? Here are some of the highlights of our upcoming releases in three key areas:

  1. Performance: Faster project configuration via caching of compiled build scripts (#31).
  2. Usability: Type-safe accessors for extensions and conventions contributed by plugins (#159); Comprehensive documentation (#106).
  3. Convenience: Declarative and tooling-friendly application of plugins, a.k.a., the plugins block (#168).

Altogether, here’s how we envision the hello-world sample will look in Gradle Script Kotlin 1.0:

plugins {
    application
}

application {
    mainClassName = "samples.HelloWorld"
}

repositories {
    jcenter()
}

dependencies {
    testCompile("junit:junit:4.12")
}

How does that look to you? We’d love to hear what you think.

A big thanks to everyone that’s been along for the ride so far, and if you’re just getting started with Gradle Script Kotlin, welcome!

Now Open: GitHub Issues for Gradle

Gradle has been an open-source project since its inception, but as a team we haven’t always lived up to the spirit of modern open-source collaboration. For example, we haven’t made it easy for folks to stay abreast of what we’re working on, and we haven’t had a clear and simple process for users to submit feature requests or bugs against a proper issue tracker.

We’re very happy to announce that all of that is changing today. We’ve opened up GitHub Issues on the Gradle repository, and what follows are the immediate, medium-, and long-term changes we’re making to put—and keep!—the needs of the Gradle community front and center.

Effective Immediately

✔︎ You can now submit feature requests and bugs through GitHub Issues. We’ve put together a simple set of guidelines in the form of an issue template so you’ll be presented with them each time you submit a new issue. We’ll label issues that don’t follow these guidelines as not-actionable and we’ll add a comment asking for what’s missing. To keep things clean, we’ll close these issues after a week of inactivity.

✔︎ Actionable issues you submit will be prioritized alongside other Gradle improvements. Low-cost changes that benefit a large number of users are the easiest to justify, so please add a 👍  reaction to issues that matter to you. Like many other GitHub projects, we’ll use these reactions as a simple kind of voting system.

✔︎ We’ll manage issue priority and workflow using ZenHub. If you’re not already familiar, ZenHub adds a number of features to GitHub Issues through its clever browser extension. It allows us to group related issues into epics and to visualize everything via a kanban board. Installing and using ZenHub is by no means a requirement, but do check it out if you’d like to get an additional level of insight about what we’re working on.

✔︎ Issues that are well-suited for community contribution will be labeled help-wanted. Take a look—you’ll see there are already a few there.

✔︎ Pull requests will be acknowledged within a week and reviews of them will be done through GitHub’s built-in pull request reviews. Please continue to follow the contribution guidelines—in a project as large and widely-used as Gradle, it’s crucial to be rigorous and get things right.

Medium-Term Transition

Open JIRA issues will be migrated to GitHub, and no new issues will be put into JIRA after the release of Gradle 3.2. We’ll ask for examples and/or clarification on JIRA issues that are not actionable so that they can be prioritized.

To draw a clearer roadmap, we’ll add and maintain high-level feature epics and mark them with milestones representing future Gradle releases.

Our bugs forum will no longer be necessary and will be made read-only. For general usage questions and potential bugs for which no reproducible test case or build scan can be provided, please continue to use the help/discuss forum.

Long-Term Unity

Once we’ve made the changes detailed above, we’ll continue looking for ways we can open up Gradle ecosystem to even greater community contribution. We’re really excited about this, and we hope you will be too.

We’d love to hear your feedback—let us know what you think in the comments on this post or talk to us @Gradle on Twitter. Your suggestions, questions, and of course, issues are most welcome.

Introducing Composite Builds

It’s not every day that we get to announce a feature that revolutionizes several software engineering workflows, but today is that day. Composite builds, a new feature in Gradle 3.1, enables an entirely new dimension in project organization.

Composite builds are a way to join multiple independent Gradle builds and build them together. The brevity of that statement does not fully convey all of the new possibilities, so let me show you how this will make your life as a developer a lot easier.

Joining projects

Many organizations split their code base into several independent projects, each having a dedicated repository and release cycle. Integration between the projects is managed using binary dependencies, e.g. JAR files published to a binary repository like Artifactory. This approach has many advantages, but can be inefficient when trying to rapidly develop and test changes that affect two or more of these projects at once.

Imagine for a moment that you are fixing a bug in a Java library that your application depends on. Your workflow probably looks something like the following:

  1. Change the library
  2. Publish the library to a local repository
  3. Add the local repository to your application’s repositories
  4. Change your application’s dependency to the new library version
  5. Test your application
  6. Repeat until the problem is fixed or you lose your mind

With composite builds, you can short-circuit this workflow by including the library’s build into your application’s build. Gradle will then automatically replace the binary dependency on the library with a project dependency—meaning that changes you make to the library become available to the application instantaneously:

The same approach works for plugins that your project depends on. You can now include a locally checked-out version of a plugin into your project’s build and get into the same kind of tight development loop between them:

The new includeBuild() API in settings.gradle even lets you write a Gradle build that dynamically includes other builds if they are available on the local file system. You could then import this composite into your IDE and do cross-repository refactoring or debugging.

Splitting Monoliths

Organizations that want to avoid the integration pains of multiple repositories tend to use a “monorepo”—a repository containing all projects, often including their dependencies and necessary tools. The upside is that all code is in one place and downstream breakages become visible immediately. But this convenience can come at the cost of productivity: a given developer will usually work only on a small part of a monorepo, but will still be forced to build all upstream projects, and that can mean a lot of waiting and wasted time. Likewise, importing large monorepo projects into an IDE often results in an unresponsive and overwhelming experience.

With composite builds, you can break your monorepo up into several independent builds within the same repository. Developers can work with the individual builds to get fast turnarounds or work with the whole composite when they want to ensure that everything still plays well together:

If you’re planning to move from a monolithic application to multiple independent ones, composite builds now offer a seamless migration strategy.

This is just the beginning

We plan to add a number of improvements to composite builds in upcoming releases:

  • Targeting tasks in an included build from the command line
  • Richer dependency substitution API with support for custom publications
  • Executing included builds in parallel
  • Integration with Gradle’s continuous build capabilities
  • Out of the box support for composite builds in IntelliJ and Eclipse

Of course, nothing is more important than feedback from real world usage. So please give composite builds a try in your own projects or have a look at the samples. Let us know about any problems, suggestions and cool things you built with it on the Gradle Forum.

Introducing Build Scans

A few months ago at this year’s Gradle Summit conference, we announced a new part of the Gradle platform called Gradle Cloud Services. In this post, I want to introduce you to the first of these services—the Gradle Build Scan Service—and the build scans it makes possible.

What is a build scan?

A build scan is a representation of data captured as you run your build. The Build Scan Plugin does the work of capturing the data and sending it to the Build Scan Service. The service then transforms the data into information you can use and share with others. Here’s a quick example of using a build scan to investigate a failure:

Publishing and Viewing a Build Scan

It’s all about that link! Here—click it for yourself: https://gradle.com/s/a5e3ngolykv3w.

As you can see, the information that scans provide can be a big help when troubleshooting, collaborating on, or optimizing the performance of your builds. For example, with a build scan in the mix, it’s no longer necessary to copy and paste error messages or include all the details about your environment each time you want to ask a question on Stack Overflow or the Gradle Forum. Instead, just include a link to your latest build scan. It contains much, if not all, of the information the person answering your question might need to know. It’ll save you both time, and they’ll probably thank you for it.

Who’s using them?

We’re excited that a number of prominent open source projects like Hibernate and JUnit 5 have already integrated build scans into their workflow. You can take a look through sample scans from each of these projects at gradle.com/explore.

Open Source Gradle Build Scans

Put build scans to use for yourself

If you’re new to build scans, now is a great time to start using them. We’re continually rolling out new features, and we’ll cover each of them in subsequent posts. In the meantime, you can learn how to enable build scans for your existing projects via our getting started instructions, or get up and running with a sample project by cloning our quick start repository and following the steps in its README.

Happy scanning, and we look forward to your feedback!

Hello, Again

Welcome to the new Gradle blog. On behalf of the whole team, it’s my pleasure to write this first post and share a bit about what we’re up to.

Mainly, we’re putting together this new blog because we want to make sure users find out about the most important developments in Gradle-land. Our team has grown rapidly over the last couple years, and as a result we’ve shipped many new features and improvements. All too often, though, we find that would-be users never hear about them. This recent tweet provides a perfect example:

Cristian’s question is a fair one. We first shipped Play support over a year ago; we mentioned it in our Gradle 2.6 forum announcement and release notes, and we wrote a chapter about it in our user manual. Still, Cristian—and probably many others—missed it. How is that?

The answer is pretty straightforward. Forum announcements and release notes are useful documents, but they get buried as new releases pile up. Reference documentation is important too, but our user manual has grown large, meaning that Chapter 70: Building Play Applications is easy to miss if you’re not already looking for it.

This situation isn’t surprising, nor is it unique to Gradle. As any project grows, it becomes a challenge to communicate about it effectively. In any case, we can no longer expect every current and future Gradle user to dig through past release notes or to read our user manual cover to cover simply to discover what Gradle can do. It’s on us to find better ways to get the word out.

And that’s where this new blog comes in. We’ll post here whenever there’s something new in Gradle that we don’t want you to miss. We’ll keep it focused on things we think are important and worth your time. We hope it’ll become a trusted resource—not only for you to stay up to date with Gradle, but also for us to get feedback through your comments.

A better blog isn’t a silver bullet, but we think it’s a great place to start. Indeed, it’s just the first step in a much larger effort to make working with and learning about Gradle as easy and enjoyable as possible. In the weeks to come you’ll see further improvements, including new guides and tutorials, a new gradle.org website, and a simpler process for filing bugs and feature requests.

Naturally, we’ll announce each of these changes here on the blog. To stay tuned, subscribe to the blog’s Atom feed or follow @Gradle on Twitter, where we’ll link to new posts as we publish them.

Gradle 3.0 M2: Initial Java 9 Support and Performance Improvements

The second milestone of Gradle 3.0 has just been released, and this version comes with initial support for Java 9!

It means that Gradle now runs properly when executed on the latest Java 9 EAP builds, but also that you can build and run tests using early versions of JDK 9. It is however important to understand that while you can compile and test applications with JDK 9, we do not support modules, nor any JDK 9 specific compile options (like -release or -modulepath) yet. However we would gladly appreciate any feedback with your own projects.

More performance improvements

This milestone is also a good moment to check out our latest and greatest performance improvements. It’s always better to perform measurements on real life builds, so the example below uses the Golo programming language as a guinea pig, and compares the execution time of a clean build of this project. The left pane is using Gradle 2.12 while the right pane is using Gradle 3.0 M2 with a “hot” daemon:

As you can see having the daemon by default makes your builds significantly snappier, although the performance improvements we’ve made since Gradle 2.12 go beyond just using the daemon. For those of you who were already enabling it in previous versions of Gradle, you should also see better performance, as this next screencast shows:

Since Gradle 2.12, we’ve made significant progress that can be summarized in a few lines:

  • configuration time is now faster, meaning that the time it will take from the moment you invoke a Gradle task and the moment the task is actually executed is much shorter. especially apparent on large multi-module builds.
  • execution with the daemon has been optimized, meaning that in Gradle 3.0 having it enabled by default, you will immediately benefit from faster builds
  • build script caching has been reworked so that subsequent builds are not only faster to configure, but also that builds running concurrently will no longer hang. This is particularly important for non-isolated builds running on a CI server

As an illustration of those improvements, we tried to execute gradle help with the daemon on on the Apereo CAS project, which consists of a large multiproject builds, which typically greatly benefits from those improvements. Again, the left side is using Gradle 2.12, while the right side uses 3.0 M2:

Last but not least, we also took a look at the rare cases where Gradle was still slower than Maven and fixed those. The following screencast is an illustration of what you can expect from Gradle 3.0 compared to Maven. This project features a build with 25 subprojects, each having around 200 files and unit tests. Then we ask both Gradle and Maven to assemble it without running tests.

Ultimately, one of the biggest differences between Gradle and Maven is that Gradle is aware of all input/outputs of tasks. As such, it’s smart enough to know about when it has to do something or not. So when we execute the same tasks again, it will not re-execute them if nothing changed:

Check out our performance guide

Having high performance builds is key to build happiness! As such, we focus heavily on performance improvements to Gradle itself. However, there are also many things that users can do to make their builds faster. To that end, we’re currently writing a performance guide, and we invite everyone to take a look at it. It’s in draft form at the moment, but already contains many valuable hints about how to make your Gradle builds even snappier. Please do give it a read, and we’d love to hear your feedback via the guide’s GitHub Issues.

Gradle 3.0 M1: Unleash the Daemon!

The first milestone release toward Gradle 3.0 has just been published, and among many smaller improvements, it contains two major features we’d like to get your feedback on.

The first feature is support for writing Gradle build scripts in Kotlin, and you can read all about it in last month’s Kotlin meets Gradle blog post. While still in the early stages of development, this functionality is now available out of the box with Gradle 3.0 M1, and we’d love to hear what you think. See the Gradle Script Kotlin 0.1.0 release notes for complete details and getting started instructions.

The second feature is that the Gradle Daemon is now enabled by default. This is a big deal, and I want to focus on it for the rest of this post.

What is the Gradle Daemon?

The Gradle Daemon is a long-running background process—a “warm JVM”—that Gradle can connect to from the command line or from within your IDE. The daemon keeps track of all kinds of information about your project’s build and makes that information available instantaneously. The result is that Gradle doesn’t have to recalculate all that information each time you run gradle build or gradle test or any other task. And the benefit to you, of course, is that your build times get a whole lot faster.

The daemon has been available since way back in Gradle 0.9, but until now it’s been an opt-in feature, meaning that users have had to provide the --daemon option at the command line or otherwise set Gradle’s org.gradle.daemon property to true in order to take advantage of it. This approach worked perfectly well, but only if you knew about it. And while many Gradle users do know about the daemon, and make use of it every day, we know from interacting with our community that the majority of Gradle users do not yet take advantage of the daemon.

So if you haven’t been familiar with the daemon until now, or if you’ve struggled to make your colleagues aware of this feature, don’t worry—you’re not alone. It’s for these reasons that we’ve made enabling the daemon by default a key part of Gradle 3.0.

Unleashing the Daemon

Over the years, the daemon has become an advanced and highly capable piece of software. But enabling it by default isn’t as simple as just “flipping a switch”. This change is profound—it means that from 3.0 forward, every Gradle user will interact with the daemon on every build unless they explicitly opt out. That’s a whole lotta builds, and with numbers like that, it means that even the tiniest bugs and most obscure corner cases are likely to surface where they haven’t before. This is why we’ve invested significant engineering effort over the last several months to ensure the daemon is up to this new challenge—and it’s why we’re so eager to get your feedback on this first milestone.

Trying out 3.0 M1 and its auto-enabled daemon is easy. Just update your Gradle wrapper using the following command and then run your builds as you normally would:

gradle wrapper --gradle-version 3.0-milestone-1

And of course, if you have any problems, comments or questions, let us know!

The Daemon in action

The short asciicast below demonstrates upgrading a project’s build from Gradle 2.13 to 3.0 M1 and shows off the resulting performance gains side-by-side in a before-and-after comparison. As you’ll see, the results are pretty dramatic—the project’s clean build time is cut by more than half:

There’s more performance goodness to come

If you run Gradle builds at all, there’s a good chance you run them dozens of times every day. This means that every second counts, and that’s why enabling the daemon by default is just one of a number of performance-related efforts we’ve been up to. Indeed, consistent with the other meaning of the word daemon, you might say we’ve been hell-bent on making Gradle the fastest and most capable build system in the world. There’s plenty more work to do to achieve that goal, and your feedback will help every step of the way. To that end, we hope you’ll check out the 3.0 M1 release notes, try it out for yourself, and let us know how it goes. Thanks!

Kotlin Meets Gradle

Many readers will be familiar with JetBrains’ excellent Kotlin programming language. It’s been under development since 2010, had its first public release in 2012, and went 1.0 GA earlier this year.

We’ve been watching Kotlin over the years, and have been increasingly impressed with what the language has to offer, as well as with its considerable uptake—particularly in the Android community.

Late last year, Hans sat down with a few folks from the JetBrains team, and they wondered together: what might it look like to have a Kotlin-based approach to writing Gradle build scripts and plugins? How might it help teams—especially big ones—work faster and write better structured, more maintainable builds?

The possibilities were enticing.

Because Kotlin is a statically-typed language with deep support in both IDEA and Eclipse, it could give Gradle users proper IDE support from auto-completion to refactoring and everything in-between. And because Kotlin is rich with features like first-class functions and extension methods, it could retain and improve on the best parts of writing Gradle build scripts—including a clean, declarative syntax and the ability to craft DSLs with ease.

So we got serious about exploring these possibilities, and over the last several months we’ve had the pleasure of working closely with the Kotlin team to develop a new, Kotlin-based build language for Gradle.

We call it Gradle Script Kotlin, and Hans just delivered the first demo of it onstage at JetBrains’ Kotlin Night event in San Francisco. We’ve published the first pre-release toward version 1.0 of this work today, along with open-sourcing its repository at github.com/gradle/gradle-script-kotlin.

So what does it look like, and what can you do with it? At a glance, it doesn’t look too different from the Gradle build scripts you know today:

build.gradle.kts

But things get very interesting when you begin to explore what’s possible in the IDE. You’ll find that, suddenly, the things you usually expect from your IDE just work, including:

  • auto-completion and content assist
  • quick documentation
  • navigation to source
  • refactoring and more

The effect is dramatic, and we think it’ll make a big difference for Gradle users. Now, you might be wondering about a few things at this point—like whether existing Gradle plugins will work with Gradle Script Kotlin (yes, they will), and whether writing build scripts in Groovy is deprecated (no, it’s not). You can find complete answers to these and other questions in the project FAQ. Do let us know if you have a question that’s not answered there.

Of course, all this is just the beginning. We’re happy to announce that Kotlin scripting support will be available in Gradle 3.0, and we’ll be publishing more information about our roadmap soon. In the meantime, there’s no need to wait—you can try out Gradle Script Kotlin for yourself right now by getting started with our samples.

And we hope you do, because we’d love your feedback. We’d love to hear what you think, and how you’d like to see this new work evolve. You can file issues via the project’s GitHub Issues and please come chat with us in the #gradle channel of the public Kotlin Slack.

I’d like to say a big thanks to my colleague Rodrigo B. de Oliveira for the last few months of working together on this project—it’s been a lot of fun! And a big thanks to the Kotlin team, in particular Ilya Chernikov and Ilya Ryzhenkov for being so responsive in providing us with everything we needed in the Kotlin compiler and Kotlin IDEA plugin. Onward!

Performance is a Feature

At Gradle Inc., we take build performance seriously. While we bundle performance improvements into every Gradle release, we’ve kicked off a concerted effort called a performance burst from Gradle 2.13 in order to make building software faster and more enjoyable for all of our users. In this blog post, we will explore how we approach performance issues, as well as what improvements to expect in the 2.13 release and beyond.

The fastest thing to do is nothing

Building software takes time, which is why the biggest performance improvement is cutting steps out of it entirely. That’s why, unlike traditional build tools such as Maven or Ant, Gradle focuses on incremental builds. Why would you ever run clean when you don’t need to? For some developers, running clean became a conditioned response to a broken build tool. Gradle doesn’t have such an issue: aware of all inputs and outputs of a task, it is reliably capable of handling incremental builds. Most builds will be incremental, and that’s why we focus so heavily on optimizing this case. One way that we accomplish this is through the Gradle daemon.

The Gradle daemon can dramatically improve your build performance by allowing build data to persist in memory between build invocations and avoiding JVM startup times on each build. The daemon is a hot JVM hosting the Gradle runtime, making it possible to run subsequent builds much faster: instead of spawning a new JVM for each build, we can benefit from all the goodness of having a cached JVM – in particular, we realize a strong benefit from JIT (just in time compilation). While turning on the daemon has a cost for the first build, the amount of time that you will gain for each subsequent build more than offsets the initial cost. In Gradle 2.13, we focused our improvements when the daemon is activated, and we’re preparing to enable this by default in Gradle 3.0. Other performance improvements we’ve implemented will benefit all users, independently of whether they use the daemon or not (and if you don’t use the daemon yet, we strongly encourage you to try it out!).

As you can read in our release notes, we’ve emphasized several categories of performance improvements:

  • reducing the build configuration time, that is to say, reducing the fixed cost of creating and configuring a Gradle build
  • reducing the test execution time; i.e., reducing the overhead of Gradle compared to just executing tests in an IDE
  • improving the performance of importing a project in an IDE
  • reducing communication latency between the interactive Gradle client and the daemon process

Reducing configuration time

Here’s an idea of the improvement you can expect:

many empty.png

So the example above yields a typical performance test metric: we’re comparing the average execution time of a build when we run gradle help for a project that contains a lot of subprojects (10000 projects). You can see that when we started optimizing configuration time, the master branch was slower than Gradle 2.7. Now, Gradle 2.13 is faster than ever! We have measured up to 25% reduction on our own builds! However, more than the improvement, it’s how we get to this that is important. Improving performance is a process, and here is how it works.

Performance test suite

The Gradle sources contain a sub-project dedicated to performance tests. This test suite is very particular, and allows us:

  • to compare the performance of the master branch with previous releases of Gradle
  • to compare various build scenarios against a single version of Gradle

So typically, in the example above, we’re comparing the average execution time of a build, when we run gradle help, in a specific scenario (an empty build with 10000 sub-projects), and compare it with previous Gradle releases. It’s worth noting that this performance test suite is executed daily, allowing us to catch performance regressions very early in the development phase.

Writing a performance test scenario

So how, in practice, do we write a performance test? It all starts with a scenario we want to test. For example, we want to make sure that we reduce the duration of test execution. The first step is then to write a build template that will let us test Gradle against this scenario. And a template has various parameters: the number of sub-projects, the number of (test) classes in source, external dependencies, … This let us generate sample Gradle builds that are used to measure performance. Of course, those performance test builds are generated with Gradle.

All the graphs you see below were generated using fully automated performance tests, and aimed at testing specific scenarios. Should you find a performance issue with Gradle, this is a great way to get started: create a new template, then send us a pull request to show the problem. Of course, all our performance tests are regular test cases, which means that we can fail the build if we introduce a regression.

Since Gradle 2.13 is primarily a performance-enhancing release, let’s focus on some of the improvements.

Gradle vs Maven

In this scenario, we are comparing the time it takes to execute gradle clean test vs mvn clean test. As we mentioned earlier, cleaning is not necessary in Gradle, but we do it here for the sake of comparison against Maven, and to assess the “cold build” time. Here are the results:

gradle vs maven clean build.png

At the end of February, Maven and Gradle were comparable. Since then, the new performance improvements in Gradle 2.13 have resulted in a 10% speedup! You can notice that the graph contains some glitches: on April 2nd, you can see that the time considerably increased. However, it increased in both scenarios: Maven and Gradle. So what you need to keep in mind when reading such graphs is that results are relative between them for a same date. This is important because:

  • we could change the templates between two executions of the performance build, resulting in an increase or decrease of the build time.
  • we could change hardware between two executions, leading to the same side effects

Profiling is better than guessing

So how did we manage to improve this? First of all, once a scenario is written and performance tests running, we need to profile the builds. For that purpose, we’re using different tools, from YourKit Java Profiler to Java Mission Control, the JIT logs or simply good old System.out.println statements. In the end, we try to identify what is causing slowdown and write a document summarizing our findings. Those documents are all public, and you can find them in our GitHub repository. Once we’ve identified hotspots and written down the profiling results, we extract stories for improvement and actually go to the implementation phase. This “profiling to stories” phase is very important, because while a profiler will be very helpful in identifying hotspots, it will be no help when it comes to interpreting the results: often, rewriting an algorithm can be much more efficient than trying to optimize a SAX parser…

Optimizing the communication between the daemon and the client

As we explained, we’re primarily (but not only) focusing on improving performance when the daemon is activated. One issue with the daemon is that you have a forked JVM. When you run gradle, the client process, the one from the command-line, starts communicating with a long-living process, the daemon, which is effectively executing the build. And typically, to see the logs as the build is running, you need to forward events from the daemon to the client. Before 2.13, this communication was synchronous. This means that the log messages were sent synchronously between the daemon and the client. This was inefficient, because we were blocking on network I/O where we could actually perform some build operations. In 2.13, not only is communication asynchronous, but we also optimized the protocol that is used to communicate between the client and the daemon and how the client responds to these events.

Forked processes start up faster

Another improvement that was made is visible in the following scenario:

gradle vs maven cleanTest test.png

This scenario is “unfair” to Gradle, and meant to compare what happens when we just want to re-execute the tests. As you may know, when running mvn test, Maven will re-execute the tests even if nothing changed. Gradle does nothing in that case, because everything is “up-to-date”. So to emulate the behavior of Maven, we need to clean-up the test results so that we re-execute the tests and re-generate the reports. As you can see, in this scenario, Gradle was significantly slower than Maven. Now, it is faster, while doing also more work: Gradle not only runs the tests, but also generates 3 types of reports: a binary one, an XML one (for CI integration) and eventually an HTML report (for use by us, poor humans, but you can disable this behavior.) Gradle 2.12 is 15% slower in this scenario, and a large amount of improvement has been done by optimizing the classpath of the forked JVMs used for tests. In 2.12, almost the whole Gradle classpath was used on forked VMs, when in reality we just need a subset of Gradle classes (basically to communicate between the forked VM and the daemon). By optimizing this classpath, we can now reduce classpath scanning and significantly improve the time it takes to execute tests. If you ever noticed a “pause” when Gradle was about to execute tests, it has now gone!

Reports are generated in parallel

Part of the improvement on test execution is also obtained thanks to parallel generation of reports. As we explained, Gradle generates more reports than Maven by default. This is usually what you want, because when you’re developing an application and run tests locally, having to decipher XML test reports can be very frustrating. With Gradle 2.13, now, the HTML and XML reports are generated in parallel, which significantly reduces the time required before starting the test suite of the next project. The more modules your project has, the more likely you will see a significant reduction in build duration.

Improving build startup time

Faster script compilation

When executing Gradle builds for the first time, you can see, as part of the “configuration” phase, that Gradle is actually compiling the build scripts. Despite being scripts, Gradle build files are written in Groovy and are nevertheless compiled to bytecode. This is time consuming, but has been optimized by the Gradle team. In particular, Gradle has to compile the scripts several times, with different classpaths, in order to compile scripts that contain references to remote resources such as plugins.

In Gradle 2.13, we changed the way Gradle scripts are compiled, and optimized two scenarios:

  • running several builds concurrently from the same directory (this often happens on CI). Before this, the “script cache” that Gradle uses was locked during the execution of a build, so if a build script was changed during the execution of a build, all concurrent builds were locked until the first one finishes.
  • re-use build scripts independently of their location. Imagine that you have multiple projects using the same remote scripts. This is typically the case in corporate environments, where a script defines some credentials, conventions, or plugins to be used in all builds of the company. Then, each project had to compile the script before being able to use it. Gradle 2.13 changed that, and now compiles script based on their actual contents (and classpath) rather than their location. It means that if you have 2 projects which have the same build files but in different locations, the script will only be compiled once. However, to be able to report build errors on the correct build file, we’re also using a “relocation technique”, which takes a compiled script class and remaps it to an actual script file so that errors are reported correctly.

Optimized classpath

Another work that has been done in 2.13 is improving the classpath of Gradle, so that services are located faster. When you have a lot of jars on classpath, ordering is important, and the number of classes is important. Even if you “only” gain 10ms, it can lead to significant differences when builds are often executed, in particular from the IDE, which leads to the last area of improvement we worked on in 2.13.

Bugfixes

Sometimes, improving performance is a matter of serendipity. We recently discovered that some performance tests were executing significantly faster on our CI server than locally, but were unsure of the cause. After doing some profiling, we realized that the code to propagate properties from the various gradle.properties files to the actual Project was very inefficient: the more properties you had in your various gradle.properties file, the longer it would take to start the build! We identified the problem and fixed this.

Faster IDE integration

The Tooling API typically allows IDE vendors to integrate Gradle. This is what we do with Buildship. It has very specific needs and in particular, it has to be both backwards and forward compatible, meaning a certain version of the TAPI can execute Gradle builds for both older and newer versions of Gradle. Of course, a developer would only benefit from the latest improvements by using both the latest version of the Tooling API and Gradle, but it leads to interesting architecture.

In this case, the Tooling API heavily relies on reflection to invoke methods. In Gradle 2.13, we significantly improved caching, which led to spectacular results:

tapi.png

This scenario illustrates how long it takes to import, typically, a 500 sub-projects build into Eclipse. While it took 25s with the 2.12 version of the Tooling API, it’s now only 10s. And you can even see more spectacular results in IntelliJ IDEA, where they are using “custom models”. Imports/synchronizing projects would then be orders of magnitude faster.

There’s more to come!

We cannot close this blog post without illustrating what we mean by “doing nothing is better”. In the Maven vs Gradle examples above, we’ve tried to “emulate” the behavior of Maven with Gradle. Here is, typically, the graph that you would get when running proper incremental builds with Gradle. That is to say that you open and edit several files from different sub-modules then re-execute the tests. Remember, with Gradle, you no longer have to clean, but we were fair and didn’t clean with Maven either:

maven vs gradle incremental.png

Yes, Gradle is almost 6x as fast in this scenario. So now, imagine doing this 10, 100 times a day, multiplied by the number of developers in your company. And realize how much money it is.

Thanks for reading this, and don’t worry: there’s more to come, stay in touch for more performance improvements in Gradle 2.14!