Improvements in the Build Configuration Input Tracking

September 1, 2023

Mikhail Lopatkin

New Features

Introduction

The configuration cache is a feature that significantly improves build performance by caching the result of the configuration phase and reusing it for subsequent builds. Using the configuration cache, the Gradle Build Tool can skip the configuration phase entirely when nothing that affects the build configuration, such as build scripts, has changed.

In Gradle 8.1, the configuration cache became stable and recommended for adoption. Stability, in this case, means that the behavior is finalized, and all breaking changes follow the Gradle deprecation process. In general, if something works now, it will continue to work the same way—unless the behavior is buggy and may lead to incorrect builds. While some features are not yet implemented, most users can already benefit from the speed-ups brought by the configuration cache.

Build configuration inputs #

When writing configuration-cache-compatible build logic or plugins, it is important to understand what Gradle considers build configuration inputs and how they affect cache invalidation.

The concept is similar to task caching:

1. Everything that contributes to the build logic execution is an input.
2. The configuration phase is an action.
3. The resulting task execution graph is the output.

Examples of the build configuration inputs are:

• Build scripts and their dependencies (plugins and libraries).
• Environment variables and system properties read at configuration time.
• Files read by the build logic or plugins at configuration time.
• Information about the file system (existence of files, directory structure) obtained at configuration time.
• The output of external processes executed at configuration time.

When build logic or a plugin reaches out to something “environmental” at configuration time, the obtained data becomes part of the configuration cache fingerprint. Next time the build runs, Gradle checks if the values of the inputs are still the same—it re-reads environment variables, system properties, files, and re-runs external processes. Then, it compares them with what was seen during the first run when the cache entry was stored. If everything matches, the cache entry is reused, and the task graph is loaded from the cache and executed without running the configuration phase.

It is important to correctly detect all inputs because otherwise, the build using the cache can be incorrect—its result won’t match the non-cached one. For example, if build logic checks for the existence of a file at configuration time and configures a task differently depending on this, then the configuration cache should be invalidated if the file appears or disappears.

To ensure correctness, Gradle employs several techniques:

• Provide APIs for build authors to explicitly specify values as coming from the environment.
• Forbid uses of incompatible APIs when configuration cache is enabled.
• Rewrite configuration logic bytecode to intercept environment access.

We continuously improve build input tracking with every release so that Gradle can reliably detect more types of inputs. These newly discovered inputs usually do not negatively affect existing builds utilizing the configuration cache. The build still reuses the cache when the new input remains unchanged. In that sense, detecting new kinds of inputs is not a breaking change but more of a bug fix that eliminates false cache hits. However, certain patterns may lead to a slightly different user experience, requiring a few additional non-cacheable runs before the cached configuration can be used in all subsequent builds. For example, the initial build may create some files after their absence has been recorded, making the cache entry invalid for the next run. Such behavior can be problematic for short-lived environments like ephemeral CI builds.

Temporarily ignoring build configuration inputs #

Adapting plugins and build logic to avoid unnecessary cache invalidations takes time. However, the input can be ignored for some scenarios without affecting the cache correctness. As an interim solution, alongside the newly introduced input detection, Gradle provides a way for build users to suppress it with Gradle properties, on a per-input basis whenever possible.

Plugin authors can document the recommended suppressions while working on an update¹, but there is no way to apply suppressions from the plugin code. As with the general Gradle deprecation process, the option to ignore the input will be available at least until the next major release.

Below we discuss two major examples of configuration input checks that improve configuration cache correctness, how they can be temporarily disabled, and how to properly fix build logic or plugin code to work with them.

Use case 1: File API tracking #

Gradle 8.1 introduced tracking of File.exists, File.isDirectory, File.list, and similar file system-querying calls. This helps to ensure correct build results in case the shape of the task graph depends on the presence of a file, directory, etc.

However, the new check might be problematic for plugins that use the “check then write” pattern:

if (!someFile.exists()) {
    someFile.createNewFile();
}

That’s because when the checked file doesn’t exist, Gradle records its absence as a configuration input and stores this in the cache. Upon the next build run, Gradle discovers that the file now exists, and therefore the cache has to be invalidated, even though the result of the build configuration is likely the same. Input detection cannot infer how checking the existence of a file affects build logic. In this case, the build or plugin author has to explicitly declare their intentions by using the appropriate Gradle API.

Temporarily disabling file API tracking #

While waiting for the updated plugin, build users can disable these checks for selected files with the org.gradle.configuration-cache.inputs.unsafe.ignore.file-system-checks property. We don’t recommend doing so without assessing the impact first. Note that the check was added in Gradle 8.1, and disabling is only possible in Gradle 8.3 onward.

Fixing plugins or build logic to work with API tracking #

Let’s see how some uses of file API can be rewritten in a configuration-cache compatible way.

Reading and writing an optional configuration file #

Suppose the plugin in question reads a configuration file. The plugin creates the file with a default configuration if it doesn’t exist. The user can modify it if needed. A value from the file is then used to conditionally register a task:

fun loadConfiguration(configFile: File): Properties {
    val result = Properties()
    if (!configFile.exists()) {
        result["some.property"] = "default value"
        configFile.bufferedWriter().use {
            result.store(it, "")
        }
        return result
    }
    configFile.bufferedReader().use { result.load(it) }
    return result
}

if (loadConfiguration().getProperty("should.register.task").toBoolean()) {
    tasks.register("someTask") {
        // ...
    }
}

In this case, the next build run will not reuse the configuration cache entry because the configuration file, which was absent at check time, is now present. This is a perfect use case for a ValueSource:

abstract class PropertiesValueSource
    : ValueSource<Properties, PropertiesValueSource.Params> {

    interface Params : ValueSourceParameters {
        val configFile: RegularFileProperty
    }

    override fun obtain(): Properties {
        val configFile = parameters.configFile.asFile.get()
        // The remaining code of creating/loading a file can be left intact.
        val result = Properties()
        if (!configFile.exists()) {
            result["some.property"] = "default value"
            configFile.bufferedWriter().use {
                result.store(it, "")
            }
            return result
        }
        configFile.bufferedReader().use { result.load(it) }
        return result
    }
}

val propsProvider = providers.of(PropertiesValueSource::class) {
    parameters.configFile = layout.projectDirectory.file("config.properties")
}

if (propsProvider.get().getProperty("should.register.task").toBoolean()) {
    tasks.register("someTask") {
        // ...
    }
}

File checks, reads, and writes made inside the ValueSource implementation aren’t recorded as configuration inputs. Instead, when checking the cache fingerprint, the whole ValueSource is recomputed, and the cache entry is only invalidated if the returned value changes.

Suppressing this type of input can be dangerous unless the user never modifies the configuration file.

Ensuring the file exists before the execution phase #

Sometimes plugins or build logic create files at the configuration phase to make sure these files are available to the tasks running at the execution phase.

val someFile = file("createMe.txt")
if (!someFile.exists()) {
    someFile.writeText("Some initial text")
}

abstract class Consumer : DefaultTask() {
    @get:InputFile
    @get:PathSensitive(PathSensitivity.NONE)
    abstract val inputEnsured: RegularFileProperty

    @TaskAction
    fun action() {
        println(inputEnsured.asFile.get().readText())
    }
}

tasks.register<Consumer>("consumer") {
    inputEnsured = someFile
}

Creating files at configuration time isn’t well supported with the configuration cache, especially when the file is then used at execution time. Gradle doesn’t track if files are created, deleted, or written at configuration time, so changes to such files may not invalidate the configuration cache. Creating the file with a task that other tasks depend on is a better solution.

abstract class EnsureFile : DefaultTask() {
    @get:OutputFile
    abstract val ensured: RegularFileProperty

    @TaskAction
    fun action() {
        val file = ensured.asFile.get()
        if (!file.exists()) {
            file.writeText("Some initial text")
        }
    }
}

val ensureFile by tasks.registering(EnsureFile::class) {
    ensured = layout.buildDirectory.file("createMeTask.txt")
}

tasks.register<Consumer>("consumer") {
    inputEnsured = ensureFile.flatMap { it.ensured }
}

Using files for cross-process locks #

Files can be used for cross-process lock protocols. For example, to ensure that concurrent builds do not overwrite some shared resource. When the locks are used at configuration time, a lock file can become an input to the configuration.

inline fun withFileLock(lockFile: File, block: () -> Unit) {
    if (!lockFile.exists()) {
        lockFile.createNewFile()
    }

    val channel = FileChannel.open(lockFile.toPath(),
        StandardOpenOption.READ,
        StandardOpenOption.WRITE,
        StandardOpenOption.CREATE
    )
    channel.use {
        channel.tryLock()?.use {
            block()
        }
    }
}

Both checking the file existence and opening the FileChannel for reading (by using open(..., StandardOpenOption.READ, ...)) make the lock file an input. The second run gets no cache hit because the result of the exists check changes.

When implementing the locking protocol, the content of the lock file typically doesn’t matter, so there is no need to open the file for reading. A race between checking for file existence and creating it is unavoidable if done separately (as shown above), so the check isn’t necessary either. After removing both the existence check and the READ option, the file is no longer an input:

inline fun withFileLockNoInput(lockFile: File, block: () -> Unit) {
    val channel = FileChannel.open(lockFile.toPath(),
        StandardOpenOption.WRITE,
        StandardOpenOption.CREATE
    )
    channel.use {
        channel.tryLock()?.use {
            block()
        }
    }
}

Suppressing this type of input is typically benign.

File access in third-party libraries #

Third-party libraries used in plugin implementations may also access files. The libraries often do not depend on the Gradle API, which makes targeted fixes unlikely.

Plugin authors can either wrap uses of such libraries in ValueSources (if the library-provided value has to be obtained at configuration time) or use the Worker API with classloader or process isolation to run the library code (if there’s no need in communicating back anything but success/failure). However, the latter is more suitable for task actions than the configuration phase.

Use case 2: Tracking inputs while storing the task graph #

Another change to the build configuration input detection is planned for Gradle 8.4.

When storing the computed task graph in the cache, Gradle resolves configurations and flattens provider chains, replacing them with the computed values if possible. User code, particularly dependency resolution callbacks and provider {} calculations, runs when doing this. Before Gradle 8.4, inputs accessed while storing the computed task graph were ignored, which could lead to false cache hits.

For example, consider a task that only runs if the file is present:

tasks.register("onlyIfFileExists") {
    val fileExistsProvider = provider { file("runTask.txt").exists() }

    onlyIf { fileExistsProvider.get() }

    doLast {
        println("File exists!")
    }
}

When running on Gradle 8.3, the absence of the “runTask.txt” file is cached in the configuration cache, so even if the file is added later, the cache isn’t invalidated, and the task is still skipped. Gradle 8.4 fixes this, and the configuration cache is invalidated correctly when the file is added.

Temporarily disabling store-time inputs #

As with file API tracking, users may temporarily opt out of the new behavior by setting the Gradle property org.gradle.configuration-cache.inputs.unsafe.ignore.in-serialization to true. Note that there is no way to ignore only a subset of inputs accessed that way.

Fixing plugins or build logic to work with store-time inputs #

To avoid unwanted inputs at the store time, build and plugin authors can use the same techniques as the other configuration-time code. Another option is to replace the Callable-based provider with a ValueSource-based or built-in one, which may improve the configuration cache hit rate if the provider’s value is only queried at execution time.

The example above can be rewritten to avoid invalidating the configuration cache at all:

tasks.register("onlyIfFileExists") {
    val fileExistsProvider = providers.fileContents(layout.projectDirectory.file("runTask.txt")).asBytes

    // fileContents provider has value present only if the file exists.
    onlyIf { fileExistsProvider.isPresent }

    doLast {
        println("File exists!")
    }
}

Conclusion #

Detecting build configuration inputs is critical for the correctness of the builds that use the configuration cache. Gradle continues to improve in this area but allows existing builds to temporarily opt out from the new behavior if it incurs “benign” cache misses until build logic and plugins can be adequately fixed.

Let us know if you think Gradle doesn’t recognize a build configuration input or lacks APIs to support your use case in a configuration-cache compatible way by opening a Gradle issue. You can also contact us on our forums or in the #configuration-cache channel of the Gradle Community Slack.

Discuss