Introducing file system watching
Table of Contents
Introduction
This is the first installment in a series of blog posts about incremental development–the part of the software development process where you make frequent small changes. We will be discussing upcoming Gradle build tool features that significantly improve feedback time around this use case.
This blog post has been updated after the feature has become production-ready in Gradle 6.7.
In Gradle 6.5, we have introduced an experimental feature called file system watching that significantly accelerates incremental builds. When enabled, it allows Gradle to keep what it has learned about the file system in memory between builds instead of polling the file system on each build. This significantly reduces the amount of disk I/O needed to determine what has changed since the previous build.
Since Gradle 6.7 this feature is ready for production use. We are planning to enable the feature by default in a later release.
How does it work? #
In order to determine whether Gradle needs to execute a task, it needs to check if any of its input and output files have changed since the last build. The daemon stores this information about the file system till the end of the current build in memory in what we call the virtual file system.
Without file system watching, the daemon does not know what happens to the file system in-between builds, and therefore it must discard all the collected information about the file system at the end of each build. When file system watching is enabled, the daemon instructs the operating system to notify it about changes on disk. Since the daemon knows what changed since the previous build, it can re-use the information in the virtual file system for all the unchanged locations, avoiding unnecessary disk I/O.
Gradle ships with the necessary integration for the recent versions of Linux, macOS, and Windows. Other operating systems are not supported.
File system watching in action #
You can enable file system watching by passing the command line switch --watch-fs or setting a Gradle property.
Once it’s enabled, the Gradle daemon will do the following:
- Start watching the file system for changes.
- Keep the virtual file system during and between builds.
Note: we are enabling verbose logging to see the effect of file system watching.
$ ./gradlew --watch-fs -Dorg.gradle.vfs.verbose=true :core:testClasses
Starting a Gradle Daemon
BUILD SUCCESSFUL in 24s
254 actionable tasks: 1 executed, 253 up-to-date
Received 11 file system events for current build while watching 1 hierarchies
Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
As you can see, the Gradle daemon learned quite a bit about the file system during this first build. All that information is only available to the daemon used to run this particular build.
Now, let’s make a change to a source file. Gradle will detect that change, and update the information about the source file in the virtual file system. Then run another build:
$ ./gradlew --watch-fs -Dorg.gradle.vfs.verbose=true :core:testClasses
Received 14 file system events since last build while watching 1 hierarchies
Virtual file system retained information about 34796 files, 3838 directories and 128 missing files since last build
BUILD SUCCESSFUL in 6s
254 actionable tasks: 3 executed, 251 up-to-date
Received 117 file system events for current build while watching 1 hierarchies
Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
Note that the daemon received some events since the last build and updated parts of the virtual file system. Though most of the information remains intact.
Build time improvements #
The practical impact of the feature depends on a number of factors, but in general it should result in a significant reduction of build time for incremental builds. For example, in the Santa Tracker Android project we’ve seen the following improvement in the build time for a small change:
In this instance we ran the same build many times, and on average we saw ~150ms (or 20%) faster execution times with file system watching enabled.
Try out file system watching #
If you like to see how your project benefits from file system watching, here is how you can try it out.
First, make sure you run Gradle 6.7 or later.
In order to enable file system watching, you need to pass --watch-fs on the command-line.
Alternatively, add
org.gradle.vfs.watch=true
to the gradle.properties in the project directory or in the Gradle user home, so you don’t need to pass the command-line option on every build.
That’s it: the next build will run with file system watching enabled.
Note: in Gradle 6.5 and 6.6 the experimental feature can be enabled via org.gradle.unsafe.watch-fs=true.
Keep in mind that you will only see performance improvements when consecutive builds on the same daemon have the feature enabled.
If you run into any problems, check out the troubleshooting section in the user manual. If you still have problems contact us via the Gradle community Slack.
We also would love to hear about the improvements you saw in your build. Please share your success stories in the Gradle community Slack. If you want to benchmark your build, you can do it easily with Gradle profiler by following the instructions in this repository.
This is not the end of the story regarding fast feedback for local incremental builds. We have further improvements scheduled for the following Gradle releases!