Good Neighbors: How to Reduce Maven Central Traffic from Gradle Builds

Calendar
Louis Jacomet
General
With Maven Central having to deal with an ever growing load, this blog entry explains how you can ensure your Gradle project is not part of the problem.

Table of Contents

Introduction

Maven Central is a vital resource for Java, Kotlin, and other JVM communities. It provides the most popular repository for open source libraries, developer tools, documentation, and other artifacts. This service is heavily used not only by developers but also by CI/CD pipelines and automation tools.

While it’s best practice to use a caching proxy for Maven Central, not everyone does. This results not only in excessive traffic costs and build delays but also exposes your builds and pipelines to more slowdowns and failures due to throttling mechanisms to protect Maven Central.

As Brian Fox, Sonatype’s CTO, wrote in this blog post in June 2024, “83% of the total bandwidth of Maven Central is being consumed by just 1% of the IP addresses. Further, many of those IPs originate from some of the world’s largest companies… we will start to work with our providers to implement throttling mechanisms aimed at the extremely heavy consumers, which are effectively abusing a community resource.” As this work progresses, more organizations will face throttling, including those using Gradle, especially if they rely heavily on Maven Central without a caching proxy in place.

While Gradle currently does not have a built-in equivalent of Maven’s mirrorOf feature to route all dependency resolution through a caching proxy, organizations can deploy a configuration that has the same effect and is compatible with existing Gradle releases. This is possible thanks to Gradle’s flexibility and rich programming model.

ℹ️ What about the Gradle Plugin Portal?
In this blog post, we showcase configuration changes that also impact the use of the Gradle Plugin Portal for resolving plugin dependencies. This is also a community resource provided for free by Gradle Inc. and, just like Maven Central, it needs to manage its workload and bandwidth usage. Note that the Plugin Portal also uses rate limiting for artifact resolution to cope with the increasing load.

Using an internal artifact repository #

Having a caching Maven repository proxy for Maven Central is a standard practice for organizations that want to curate artifacts and reduce network traffic costs. Multiple implementations of such repositories are provided by different vendors or open-source organizations, and most of them are compatible with Maven Central.

Configuring Gradle projects to use the organization’s repository instead of Maven Central #

In Gradle, you must explicitly define one or more repositories for resolving dependencies. To do this, declare the appropriate repositories in your settings script:

// in settings.gradle.kts
pluginManagement {
    repositories {
        // Replace Gradle's default for plugin resolution by internal repository
        maven {         
            url = uri("https://company/com/proxy-repository")
        }
    }
}

plugins {
    // Declare settings plugins here
}

dependencyResolutionManagement {
    // Make sure projects do not add other repositories by accident
    repositoriesMode = RepositoriesMode.FAIL_ON_PROJECT_REPOS
    // Define the internal repository as the only repository to use
    repositories {
        maven {         
            url = uri("https://company/com/proxy-repository")
        }
    }
}

// Other settings content

In the example above, replace the URL with your organization’s repository location. This snippet is an opinionated way of defining repositories for internal projects that leverages the centralized repository declaration feature. A project using the configuration above will not send requests to Maven Central, although the organization’s repository itself might have to do so to cache resources.

This declaration is compatible with Gradle 6.8 and above. For more information on repository declaration syntax and requirements, refer to the Gradle documentation.

Overriding the URLs for Maven Central and the Plugin Portal repositories #

Modifying the build file is not always feasible. For example, when building open-source projects within your own infrastructure, you may want to globally override repository configurations defined in the build.

Here is an init script that will replace the Maven Central and Gradle Plugin Portal URLs in all projects that use them:

// init.gradle.kts
apply<InternalRepositoryPlugin>()

class InternalRepositoryPlugin : Plugin<Gradle> {
    override fun apply(gradle: Gradle) {
        val canBeMirrored: Spec<MavenArtifactRepository> =
            Spec { r -> r.getName().equals(ArtifactRepositoryContainer.DEFAULT_MAVEN_CENTRAL_REPO_NAME)
                    || r.getName().equals("Gradle Central Plugin Repository") }

        val useMirror: Action<MavenArtifactRepository> = Action {
            val mirrorUrl: String = "https://company/com/proxy-repository"
            setUrl(mirrorUrl)
        }
        val configureMirror: Action<RepositoryHandler> = Action {
            withType(MavenArtifactRepository::class.java)
                .matching(canBeMirrored)
                .configureEach(useMirror)
        }
        gradle.beforeSettings(Action {
            configureMirror.execute(getBuildscript().getRepositories())
            configureMirror.execute(getPluginManagement().getRepositories())
        })
        // Remove the settingsEvaluated part if you are using Gradle <6.8
        gradle.settingsEvaluated(Action {
            configureMirror.execute(getDependencyResolutionManagement().getRepositories())
        })
        gradle.beforeProject(Action {
            configureMirror.execute(getBuildscript().getRepositories())
        })
        gradle.afterProject(Action {
            configureMirror.execute(getRepositories())
        })
    }
}

In the above example, replace mirrorUrl with the URL of your internal repository. There are several options to deploy this init script; see the Gradle documentation for options.

The above init script is fully compatible with Gradle 6.8 and above. The gradle.settingsEvaluated section needs to be removed for Gradle 6.0 to 6.7 included. Lower versions are not supported.

Support for easier mirroring #

The example above demonstrates how Gradle’s flexibility and programming model can resolve build challenges, even if no out-of-the-box feature exists. The code is non-trivial because Gradle distinguishes between plugins and project repositories and has a central and local way of declaring both of these.

Admittedly, addressing the mirroring use case could be easier. At the moment, Gradle does not support defining repository mirrors, similar to Apache Maven’s support for Mirrors. We are monitoring an open feature request in gradle/gradle #27808. Firstly, we plan to improve the documentation on how to inject repository configuration from outside the build. We will also evaluate, based on community reactions to this post and the upcoming documentation update, if a core feature is needed.

Troubleshooting #

To troubleshoot dependency resolution and caching, you can use our free Build Scan service. Build Scans analyze the effective dependency of the builds and the performance of artifact resolution and downloads.

Each dependency indicates from which URL it was resolved:

Build Scan dependency origin

And you can get a global view of all the network traffic from your build:

Build Scan networking view

To learn more about enabling Build Scan, refer to the Gradle documentation.

Summary #

Reducing traffic to Maven Central isn’t just about improving build performance; it’s about building resilient, efficient, and sustainable development pipelines. Whether you’re working on open-source projects or managing enterprise-scale builds, using an artifact repository can dramatically cut down redundant downloads, minimize the risk of throttling, and save on network costs.

While not every project can adopt all solutions, even small improvements can make a meaningful difference. And with the growing pressure on public infrastructure like Maven Central, it’s increasingly important for the community to follow best practices and reduce unnecessary load.

If you haven’t already, consider reviewing your organization’s current dependency resolution setup, and if you’re affected by the lack of repository mirroring in Gradle, lend your voice to Gradle issue #27808.

See also #

Discuss

Related Posts