Introducing TestKit: A Toolkit for Functionally Testing Gradle Build Logic

January 26, 2016

Ben Muschko

New Features

Introduction

Automated testing is a necessary prerequisite for enabling software development practices like refactoring, Continuous Integration and Delivery. While writing unit, integration and functional tests for application code has become an industry norm, it is fair to say that testing for the build automation domain hasn’t made its way into the mainstream yet.

But why is it that we don’t apply the same proven practice of testing to build logic? Ultimately, build logic is as important as application code. It helps us to deliver production software to the customer in an automated, reproducible and reliable fashion. There might be many reasons to skip testing; however, one of the reasons that stands out is the data definition format used to formulate build logic. In the past, writing tests for XML-based build logic definitions was a daunting, almost impossible task without the right tooling.

In this regard, Gradle makes your life easier. Build code can be structured properly, organized based on functional boundaries, and developed as actual class implementations with the help of concepts like custom tasks and binary plugins. Automated testing of build logic becomes approachable, and when combined with the appropriate tooling is easily attainable.

Meet TestKit #

One way to test build logic is to declare and execute it the same way as the end user would. In practice, this means creating a build script, adding the configuration you want to test and executing it with the Gradle runtime. The outcome of the build, such as the console output, the executed tasks, and produced artifacts, can be inspected and verified against expected assertions. This type of testing is commonly referred to as functional testing.

Let’s have a look at an example. In the following build script, we apply the Java plugin.

build.gradle #

apply plugin: 'java'

Executing this build script with the compileJava task should produce class files for the Java source files found in the directory src/main/java. As an end user, we’d expect these class files to be located in the directory build/classes/main. Of course, you could verify this behavior by executing the given build script manually with the Gradle command and inspect the output directory. I hope the last sentence gave you the itch. We are automation engineers, so obviously we’ll want to automate as much as we can.

Meet the Gradle TestKit: a toolkit for executing functional tests in an automated fashion. TestKit is bundled with Gradle starting with version 2.6 and is available to be used in your projects now.

Using TestKit #

There are typically two different use cases for adding TestKit to a project.

1. Cross-version compatibility testing. You want to verify if a build script is compatible with a specific Gradle version. Organizations often apply this technique in preparation for a Gradle version upgrade of an existing build or when multiple versions of Gradle must be supported by the same build logic.

2. Custom build logic testing. You want to test if your custom task or plugin behaves as expected under certain conditions that resemble the real-world usage by a build script author. A typical example could be: “If a user applies this plugin and configures a property of my exposed extension, then a provided task should observe a specific runtime behavior and produce the output x when executed.” On top of this scenario cross-version compatibility could play a role as well.

Given the last example, let’s have a look how we can implement the scenario with the TestKit API. Note that the following test class uses the Spock test framework.

BuildLogicFunctionalTest.groovy #

import org.gradle.testkit.runner.GradleRunner
import static org.gradle.testkit.runner.TaskOutcome.*
import org.junit.Rule
import org.junit.rules.TemporaryFolder
import spock.lang.Specification

class BuildLogicFunctionalTest extends Specification {
   @Rule final TemporaryFolder testProjectDir = new TemporaryFolder()
   File buildFile

   def setup() {
       buildFile = testProjectDir.newFile('build.gradle')
   }

   def 'produces class files when compiling Java source code'() {
       given:
       buildFile << "apply plugin: 'java'"

       when:
       def result = GradleRunner.create()
           .withProjectDir(testProjectDir.root)
          .withArguments('compileJava')
          .build()

       then:
       result.task(':compileJava').outcome == SUCCESS
       new File(testProjectDir.root, 'build/classes/main').exists()
   }
}

Even if you haven’t used Groovy or Spock before, it becomes apparent how easy it is to formulate a functional test case with the help of TestKit.

Tell Me More About TestKit #

The previous code example uses Spock for implementing a test case. If you are not familiar with Spock or prefer a different test framework, you can still use TestKit. By design, TestKit is test framework-agnostic. It’s up to you to pick the test framework you are most comfortable with whether that’s JUnit, TestNG or any other test framework out there in the wild.

For test scenarios that require you to execute a test with multiple Gradle distributions, e.g. in the context of cross-version compatibility testing, TestKit exposes API methods for providing the appropriate Gradle distribution information. You can either point to a local installation of Gradle, a distribution identified by version on a server hosted by Gradle Inc. or a distribution identifiable by URI.

As you execute the test, you might also want to step through the build logic under test for debugging purposes from the IDE of your choice. TestKit allows you to execute tests in debug mode to track down unexpected test runtime behavior.

What’s on the Roadmap for TestKit #

There’s more to come for TestKit. In the future, we want to make it even more convenient to use the TestKit API. You can read all about it in the design document. Let us know if you are interested in contributing! We’d love to see TestKit evolve.

Discuss