Gradle Plugin Resolution Outage Postmortem

January 20, 2022

On January 12th 2022, Gradle users were experiencing issues resolving plugins from the Gradle Plugin Portal because of the outage of JCenter that the Plugin Portal depends on for some of the functionality.

This postmortem gives a timeline of the outage, describes the effect on Gradle users, and what actions were taken to further reduce the Gradle Plugin Portal’s reliance on JCenter.

Finally, we will also discuss how you can protect your build against such outages.

Users affected by JCenter outages for project dependencies should also refer to our original blog post about the shutdown of JCenter.

Outage timeline

4.30pm UTC, January 12th Outage begins: Gradle users start noticing that plugin resolution is failing in builds.
4.56pm UTC, January 12th Incident opened: The outage is acknowledged on the Gradle status page.
6pm UTC, January 12th: We identify a hotfix that will allow the Gradle Plugin Portal to ignore 5xx error codes from JCenter and resolve plugins it hosts entirely. However, the hotfix cannot be built and deployed immediately because building the Plugin Portal code requires resolving some plugins from the Plugin Portal itself and JCenter, which failed.
8.44pm UTC, January 12th First fix is deployed: We update the build of the Plugin Portal to not rely on JCenter at all by using repository content filtering and supported metadata sources. The first fix is deployed.
8.58pm UTC, January 12th Second fix is deployed: The first fix turns out to be insufficient. We develop and deploy a second fix that allows us to change the repository to which the Plugin Portal redirects. We configure the Plugin Portal to use our internal mirror of JCenter.
9.43pm UTC, January 12th Outage is resolved: The existing mirror is too sparsely populated as it was not used much, which means most builds still fail to resolve plugins. We update the mirror repository to redirect to Maven Central as well. This results in most dependencies being properly resolved and unblocked most builds. Plugins that rely on artifacts that were only available on JCenter still fail to be resolved.
6.35am UTC, January 13th Incident is closed: JCenter is back online. The mirror repository is changed back to mirror JCenter only and the Gradle Plugin Portal remains configured to use the mirror repository.

Effects of outage on builds

This outage showed a few ways that builds may fail when the Plugin Portal cannot serve all requests:

Builds that use dynamic versions for plugins could not resolve the list of versions.
Builds that run on ephemeral CI agents or in new environments could download plugin artifacts, but they could not download dependencies for the plugins from JCenter.
Plugins that missed artifacts requested by Gradle on the Plugin Portal failed to resolve because JCenter would serve 5xx errors instead of 404, which Gradle considers an error instead of a missing artifact.

Reducing the chance of future outages

Issues with resolving Gradle plugins have a high impact on Gradle builds. Below we describe the steps we are taking to reduce the likelihood and severity of potential future outages.

Immediate changes

Following this incident, the Gradle Plugin Portal now uses a JCenter mirror hosted by Gradle instead of JCenter directly. This should shield users from short JCenter outages for libraries that have been cached by the mirror. We saw another short outage of JCenter over the weekend and this did not appear to impact Gradle Plugin Portal users.

Background information

The link between the Gradle Plugin Portal and JCenter is largely historical.

Before the announcement of the sunset of Bintray and JCenter, you could publish a Gradle plugin in two different ways:

To a Bintray repository. Plugin authors could then ask for the plugin to be included on the Plugin Portal.
To the Plugin Portal. Plugin authors could use the plugin-publish plugin to upload their plugins directly.

The Gradle Plugin Portal relies on the following from JCenter:

List of available versions: Plugins that were published to Bintray and then on the Plugin Portal still need to expose all of their versions to plugin users.
Missing artifacts: Files like Gradle Module Metadata, checksums and signature files were not always supported by the Plugin Portal. These files may have been published directly to JCenter and not the Plugin Portal.
Plugin dependencies: Dependencies used by a plugin, like Guava, are not deployed to the Plugin Portal but need to be reachable as if they are, which is what the redirect offers.

We discontinued and removed the integration with Bintray when Bintray shutdown in May 2021, but we kept redirecting to JCenter since JFrog said that JCenter would remain read-only indefinitely. We are now reconsidering that decision and will move to reduce and eliminate the Plugin Portal’s dependency on JCenter.

The Gradle team is aiming to find a solution that will have as little impact as possible on builds, but we know that zero impact will not be possible.

We will provide a future update on how we will be replacing JCenter on the Plugin Portal and how that may affect your builds.

Strengthening your build infrastructure

Build engineers can strengthen their build infrastructure by decoupling it from online services like JCenter and the Gradle Plugin Portal.

By setting up a mirror of the Plugin Portal, you can cache artifacts closer to your build infrastructure and developers.

This can provide multiple benefits:

Improved performance
Better control of used dependencies for security and auditing purposes
Isolation from outages

If you’re already using custom repositories, you can also leverage repository content filtering to make sure that you fetch build and project dependencies from a particular repository.

Feedback

Let us know if you have any questions on our forums or Gradle Community Slack.