Introducing file-system watching
This is the first part of a series of blog posts about incremental development, the part of the software development process where the developer makes frequent small changes to the code and rebuilds them locally, typically in an IDE. We will be discussing upcoming features in the Gradle build tool that significantly improve performance around this use case.
In Gradle 6.5, we are introducing an experimental feature called file-system watching that significantly accelerates incremental builds. When enabled, it allows Gradle to keep what it has learned about the file-system in memory between builds instead of polling the file-system on each build. This significantly reduces the amount of disk I/O needed to determine what has changed since the previous build.
File-system watching is currently experimental, and it comes with some limitations. We plan to make it production-ready in the next couple of Gradle releases and then it will be enabled by default.
How does it work?
In order to determine whether Gradle needs to execute a task, it needs to check if any of its input and output files have changed since the last build. The daemon stores this information about the file-system till the end of the current build in memory in what we call the virtual file-system.
Without file-system watching, the daemon does not know what happens to the file-system in-between builds, and therefore it must discard all the collected information about the file-system at the end of each build. When file-system watching is enabled, the daemon instructs the operating system to notify it about changes on disk. Since the daemon knows what changed since the previous build, it can re-use the information in the virtual file-system for all the unchanged locations, avoiding unnecessary disk I/O.
Gradle ships with the necessary integration for the recent versions of Linux, macOS, and Windows. Other operating systems are not supported.
File-system watching in action
You can enable file-system watching by passing the command line switch
--watch-fs or setting a Gradle property.
Once it’s enabled, the Gradle daemon will do the following:
- Start watching the file-system for changes.
- Keep the virtual file-system during and between builds.
$ ./gradlew --watch-fs :core:testClasses Starting a Gradle Daemon Watching the file system is an incubating feature. Spent 10 ms registering watches for file system events Virtual file system retained information about 0 files, 0 directories and 0 missing files since last build BUILD SUCCESSFUL in 24s 254 actionable tasks: 1 executed, 253 up-to-date Received 11 file system events for current build Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
In the current experimental state of the feature, Gradle gives plenty of information about what happened. As you can see, the Gradle daemon learned quite a bit about the file-system during this first build. All that information is only available to the daemon used to run this particular build.
Now, let’s make a change to a source file. Gradle will detect that change, and update the information about the source file in the virtual file-system. Then run another build:
$ ./gradlew --watch-fs :core:testClasses Watching the file system is an incubating feature. Received 14 file system events since last build Virtual file system retained information about 34796 files, 3838 directories and 128 missing files since last build BUILD SUCCESSFUL in 6s 254 actionable tasks: 3 executed, 251 up-to-date Received 117 file system events for current build Virtual file system retains information about 34797 files, 3845 directories and 128 missing files till next build
Note that the daemon received some events since the last build and updated parts of the virtual file-system. Though most of the information remains intact.
Build time improvements
The practical impact of the feature depends on a number of factors, but in general it should result in a significant reduction of build time for incremental builds. For example, in the Santa Tracker Android project we’ve seen the following improvement in the build time for a small change:
In this instance we ran the same build many times, and on average we saw ~150ms (or 20%) faster execution times with file-system watching enabled.
Try out file-system watching
If you like to see how your project benefits from file-system watching, here is how you can try it out.
First, make sure you run Gradle 6.5 or later.
In order to enable file-system watching, you need to pass
--watch-fs on the command-line.
gradle.properties in the project directory or in the Gradle user home, so you don’t need to pass the command-line option on every build.
That’s it: the next build will run with file-system watching enabled.
Keep in mind that you will only see performance improvements when consecutive builds on the same daemon have the feature enabled.
We also would love to hear about the improvements you saw in your build. Please share your success stories in the Gradle community Slack. If you want to benchmark your build, you can do it easily with Gradle profiler by following the instructions in this repository.
This is not the end of the story regarding fast feedback for local incremental builds. We have further improvements scheduled for the following Gradle releases!