Testing Astra 6 v Fable 5.1 on a Gradle docs bug

September 9, 2026

Laura Kassovic

General

Introduction

On the afternoon of September 7th, claude-fable-5-1 was handed a git repository, a GitHub issue, and fifty turns to fix it. Just over three hours later, gpt-6-astra finished the same assignment in another worktree. Same issue. Same constraints. Different CLI wrapped around each model: claude-code for Fable, codex for Astra.

Both Agents fixed the bug. Both results were graded as successes by an independent judge. But these results were quite different.

You would be forgiven for wondering why an engineering blog is running a two-model bakeoff on a single documentation bug, in a year when everybody and their intern has published an AI coding benchmark. Here’s the honest answer: we wanted to know which one actually did the better job, as a side quest of the Agentic Gradle project. But “better” was never going to mean pass-or-fail. Both of these agents passed; if that were the whole story, this post would be four sentences long. What actually separates a fix worth merging from a fix worth sending back is cost, speed, and whether the result is something we would want to maintain, and those three do not all point the same direction, as you are about to see.

So: one bug, two agents, and a very literal stopwatch. Here is what this looks like when you actually read the agents’ transcripts.

Real talk up front: this is n=1. One run per model, one judge, one bug. That’s a thin base for anything you’d call a leaderboard, and I’ll say so again at the end, but it didn’t stop me from landing on an opinion by the time I’d finished reading both transcripts.

The bug, briefly #

Issue #34751, filed August 21st by our very own cobexer, is the kind of bug that is very easy to describe and mildly annoying to fix. In the Kotlin DSL API docs, a type like Attribute<Integer> linked to the Java 21 Javadoc. It should have linked to Java 17, because Gradle targets Java 17.

The reason is almost embarrassingly mundane once you see it. Gradle is built with a JDK 21 toolchain but targets JDK 17, which is a completely normal thing for a build tool to do (you want the newer compiler, not necessarily the newer bytecode floor). The Kotlin DSL reference docs are generated by Dokka, and Dokka’s Java interop layer derives its jdkVersion from the toolchain used to compile the project. Toolchain is 21, so Dokka assumed 21, so every JDK type in the Kotlin DSL reference linked to Java 21’s Javadoc.

The Java API docs (the userguide, generated separately) already get this right, through a small property chain: jvmVersion flows into minJdkVersion in platforms/documentation/docs/build.gradle.kts. Nobody had wired that same property through to the plugin that configures Dokka for the Kotlin DSL reference, GradleKotlinDslReferencePlugin.java. One plugin knew the right answer. The other plugin never asked the first the question.

That is the entire bug. A property that exists, correctly wired in one place, and was simply never passed to a second place that needed it. I mention this not to bury the lede but because it is worth knowing exactly how small the fix actually is before you watch two agents spend a combined twenty-six minutes finding it.

The rules of engagement #

We run these comparisons through an internal harness we call gradle-eval which is built on top of Inspect AI (we will share more about it in a later post). Both agents received the identical prompt:

You are working on the official gradle/gradle repo, master branch. First, initialize a git repository: run git init, then git add ., then git commit -m “initial” to create a base commit. Then create a branch named fix/issue-34751 to work on a documentation issue. Before you get started on the fix, read the documentation README in the repo located at platforms/documentation/docs/README.md. Then you work on solving #34751 — “Kotlin DSL docs link to Java 21 for platform types.” at https://github.com/gradle/gradle/issues/34751.

Both got the same snapshot of gradle/gradle (commit 8606a2c1), the same limits (50 turns, 5 million tokens, 45 minutes of wall clock), network access, and a JDK 21 container. Both claude-fable-5-1 and gpt-6-astra were wrapped in their native CLIs, not some harness-provided abstraction over “a model.” That distinction matters more than it sounds like it should, and I will come back to it, because a decent chunk of what makes this comparison interesting has nothing to do with the underlying models at all.

A PASS required the fix to land in the actual Dokka/Kotlin DSL configuration, resolve the Java 17 versus 21 mismatch by wiring the reference back to the same source the userguide already uses, and compile. Both agents were scored against that rubric by an LLM judge. Both scored PASS. If you stopped reading right here, you would conclude the two systems are interchangeable for this task. You would be leaving the interesting 90% of the transcript on the table.

What Fable 5.1 did with fourteen minutes #

claude-fable-5-1, running inside claude-code v2.1.263, took 861 seconds and 28 turns. It did the setup commands, read the docs README as instructed, and then, instead of going straight to the file, fetched the GitHub issue over the network. Three separate WebFetch calls: the issue itself, and then two follow-ups pulling in Dokka’s own source to understand how the Java adapter actually derives jdkVersion.

That last part is the interesting decision. It found that Dokka’s adapter sets jdkVersion using configureEach with a plain .set(...) call, not .convention(...). That is a small distinction with real consequences in Gradle’s lazy configuration model: a configureEach block runs for every object added to a container, in registration order, and a later set() call simply overwrites whatever came before it, where a convention() would have politely stepped aside for anyone who set their own value. Knowing that ordering rule tells you exactly where in the Gradle build-logic code you can safely put an override and have it actually stick.

Having established that, claude-fable-5-1 built a standalone Dokka 2.2.0 reproduction project with a JDK 21 toolchain, confirmed the bug reproduced (Java 21 links), applied its fix, and confirmed the fix worked (Java 17 links), all before touching the real fix location in gradle/gradle. Then it patched GradleKotlinDslReferencePlugin.java, added a new Spock/TestKit test asserting both Dokka source sets report 17 when the toolchain is 21, ran that test (failed once on a missing environment variable, fixed it, passed), and for good measure re-ran the existing GradleReleaseNotesPluginTest to make sure nothing else moved. It left the branch uncommitted, ready for review.

Thirty-nine Bash calls, three WebFetch calls, one TaskOutput call (an Inspect-injected tool, not something either CLI ships natively), 25,845 output tokens, and 9,693 of those were reasoning tokens spent narrating its own thought process in the transcript.

What Astra 6 did with twelve minutes #

gpt-6-astra, running inside codex v0.153.4, took 723 seconds and 48 turns to reach the same PASS. It had to configure a git identity first (codex@openai.com), then did the same setup commands, read both the docs README and CONTRIBUTING.md, and immediately grepped the codebase for minJdkVersion. That single grep found the property declaration in build.gradle.kts line 94 and its trace back to Javadocs.getMinJdkVersion(). From there it read GradleKotlinDslReferencePlugin.java in full, read Javadocs.java for the property declaration, searched for where Dokka source sets expose a jdkVersion property, applied a three-line fix, and compiled it with ./gradlew :build-logic:documentation:compileJava. It then manually verified that the Integer link on the generated docs page pointed at Java 17. No new test. It left the fix uncommitted with a note that it was “ready for review.”

Every single one of those steps, including the git commands, the reads, the grep, and the compile, went through one tool: exec. Forty-seven calls to it. No dedicated fetch tool, no network calls at all; gpt-6-astra relied entirely on what already existed in the checked-out repository and succeeded anyway.

The two fixes, side by side #

Here is claude-fable-5-1’s version, added inside each of the two register() blocks that configure Dokka’s source sets:

             spec.getIncludes().from(extension.getSourceRoot().file("kotlin/Module.md"));
+            configureJdkVersion(extension, spec);
             configureSourceLinks(project, extension, spec);

with a new private method and a doc comment explaining why:

    /**
     * Links JDK types to the Javadoc of the minimum JDK version Gradle supports,
     * rather than the JDK used to build Gradle.
     * <p>
     * By default, the Dokka Gradle Plugin derives {@code jdkVersion} from the Java toolchain
     * used to build the project, which would make the Kotlin DSL reference link to Javadoc
     * of a newer JDK than Gradle actually requires.
     */
    private static void configureJdkVersion(GradleDocumentationExtension extension,
            DokkaSourceSetSpec spec) {
        spec.getJdkVersion().set(extension.getJavadocs().getMinJdkVersion());
    }

And here is gpt-6-astra’s, a single configureEach sitting above both register() calls:

         NamedDomainObjectContainer<DokkaSourceSetSpec> dokkaSourceSets = getDokkaExtension(project).getDokkaSourceSets();
+        dokkaSourceSets.configureEach(spec -> {
+            spec.getJdkVersion().set(extension.getJavadocs().getMinJdkVersion());
+        });
         NamedDomainObjectProvider<DokkaSourceSetSpec> javaApi = dokkaSourceSets.register("java_api", spec -> {

Thirteen lines added versus three. Functionally, today, these produce identical output: there are exactly two Dokka source sets in this plugin, java_api and kotlin_dsl, and both approaches set jdkVersion on both of them before Dokka’s own configureEach (which also uses .set(), remember) has a chance to overwrite it with the toolchain version. Gradle runs register() actions after container-level configureEach actions that were registered earlier, so claude-fable-5-1’s per-register placement wins the same way gpt-6-astra’s earlier configureEach wins. Two different levers, same ordering outcome, for now.

“For now” is doing real work in that sentence. gpt-6-astra’s configureEach applies to every source set this container will ever hold, including ones nobody has written yet. If a future contributor adds a third Dokka source set to this plugin for some other purpose, it silently inherits minJdkVersion whether that is what they wanted or not, with no comment anywhere nearby explaining that this is happening to them. claude-fable-5-1’s version only touches the two source sets it explicitly configures, in the same place it already configures everything else about them, which happens to also be the existing style of the surrounding method. Both fixes solve today’s bug. One of them is quietly betting on the container never growing a third member.

The numbers, because we tracked them #

The token story is the one I keep coming back to. gpt-6-astra produced four times fewer output tokens per turn while taking nearly twice as many turns, and it still burned 56% more total tokens than claude-fable-5-1, almost entirely on cache reads. That is not a contradiction, it is a description of two different places to spend compute. A reasoning model working through a codex-style exec loop appears to do a great deal of its thinking in a scratchpad that never becomes visible output; only 991 of its output tokens were tagged as reasoning, versus 9,693 for claude-fable-5-1, which narrates its exploration directly into the transcript you can read. Every one of those 47 exec calls also re-sent a growing conversation history back through the model, which is what shows up as cache reads rather than fresh input. Terser final output, more expensive to arrive at it. Neither number tells you which model is “smarter”; they tell you two different CLIs made two different bets about where to spend the token budget, and this harness happened to be instrumented well enough to show us the receipt.

The tool vocabulary difference is worth a paragraph of its own, because it is easy to misattribute to the harness and it isn’t the harness’s doing at all. Our evaluator hands both agents the same minimal configuration: model, sandbox mode, working directory. No tool allowlist, no permission mode, nothing extra granted to one and withheld from the other. claude-code in that configuration runs with permissions bypassed entirely, so it shows up with its full native palette: Bash, Read, Write, Edit, Glob, Grep, WebFetch, and friends. codex ships with exactly one tool, exec, a general-purpose shell escape hatch that every read, grep, git command, and compile step gets routed through. That is not a restriction we imposed; it is the deliberate design of each CLI, and it is the single biggest reason these two transcripts read so differently even when the underlying task is identical. The one capability gap that actually mattered here is WebFetch: claude-fable-5-1 used it to go read Dokka’s own source before writing a fix, and gpt-6-astra, with no equivalent tool available, got to a correct fix anyway using only what was already sitting in the checkout.

Which one we would actually merge #

Short version: claude-fable-5-1’s, no argument.

The fix is idiomatic for this file. It configures jdkVersion inside the same register() blocks that already configure everything else about each source set, which is exactly how the rest of wireInArtificialSourceSet is written, so a reviewer skimming the method sees one more line that looks like all the others rather than a new pattern grafted on top. The Javadoc comment explains why the override exists, which matters here specifically because the reason (Dokka silently defaults to the build toolchain, not the target version) is not something the next person touching this file would necessarily know to go looking for. And it shipped with a genuine TestKit integration test: apply the real plugin, set toolchain to 21 and minJdkVersion to 17, assert both source sets report 17. That test passed, and the pre-existing GradleReleaseNotesPluginTest still passed alongside it.

gpt-6-astra’s fix would come back from code review with two comments attached. First: no test. The fix is three lines and correct, and gpt-6-astra even eyeballed the rendered docs page to confirm the link now points at Java 17, but “I looked at it and it seemed right” is not a regression test, and any reviewer on this repository is going to ask for one before merging a fix to generated reference documentation, which is exactly the kind of thing that regresses silently. Second: the configureEach sits above both register() calls and therefore governs a container that currently has two members and might not stay at two forever, with no comment flagging that a future third source set inherits this behavior by default. Today that is equivalent to the more scoped fix. It is a wider net than the bug needed, cast without a label on it.

The part that isn’t in either diff #

Here is the meta-observation I keep turning over. claude-fable-5-1 didn’t reach for the first plausible fix. It went and read Dokka’s own source to understand why the toolchain version was winning, specifically that the adapter uses configureEach with set() rather than convention(), and used that understanding to reason about where an override would actually take effect before writing one line of code. It then built a disposable repro project just to watch the bug happen and then stop happening, which is a slower path to the same three-line insight and also exactly the kind of thing a careful senior engineer does before touching build logic they don’t fully trust yet.

gpt-6-astra went grep, read, write, compile, verify, in that order, and arrived at a defensible fix in less wall-clock time with far less visible narration. Faster is not wrong. But the transcript gives you no way to tell whether choosing configureEach over the more scoped per-register approach was a considered tradeoff or simply the first pattern that occurred to it, because there is no reasoning on the page to check it against. That absence of narration is itself information, just not the kind you can put a checkmark next to.

The caveat, spelled out twice because it matters #

This is one run, on one bug, judged once (then overanalyzed by me). Both models ran against the identical snapshot, the identical prompt, the identical limits, and the same 45-minute clock. I’m not going to pretend that settles which lab builds the better model in general; one documentation bug does not crown a universal champion, and running this exact scenario again tomorrow might yield a different verdict. What I will stand behind is that on this task, under these constraints, claude-fable-5-1 did the more thorough, more maintainable job, and that’s worth saying plainly instead of hiding behind a hedge.

One more scope note. This ran single-shot: no follow-up, no chance for either agent to course-correct, which isn’t how we actually review AI-written changes before they ship to gradle/gradle. The instinct behind running it at all — measure it, don’t vibe it — is the same one running through everything else Agentic Gradle ships. This was the smallest, easiest place to try it out. The next ones will matter more.

If you have any feedback for the Agentic Gradle team, say “hi” to us in the #agentic-gradle channel on the Gradle Community Slack. In the meantime, claude-fable-5-1’s branch is getting turned into a real PR — Javadoc comment and all.

Discuss