Introducing Exemplar for Automated Samples Testing

Calendar
Eric Wendelin
General

Table of Contents

Introduction

This post introduces a new library called Exemplar. The goal of Exemplar is to ensure that users get outputs that you expect them to see. It handles sample discovery, normalization (semantically equivalent results, perhaps from different environments), and flexible output verification. It invokes any command-line tool in the environment to be invoked. You can also invoke curl, for example, to verify service API responses.

Gradle uses this library to verify examples in docs and guides, and remove boilerplate from integration tests.

Exemplar can be configured using a JUnit test runner (recommended) or using its APIs. See examples below and in the Exemplar GitHub repo.

Exemplar Information Flow

Use cases for Exemplar #

It’s important that documentation samples are accurate. Imagine the frustration, trying to learn something new and having the examples fail to work.

We think Exemplar works best as a documentation checking mechanism. It is not meant to substitute other forms of integration testing however.

Exemplar is unique because it allows discovery of samples embedded in structured documents, as well as OutputNormalizers (bundled and custom ones), conveniences for using sample projects in “more traditional” integration tests (via @UsesSample("path/to/sample")), and allows samples to be programmatically modified (for extra environment setup, perhaps) before execution (SampleModifier).

Exemplar is focused on logging and exit code outputs, and verifying other side effects is cumbersome (checking created files may require extra commands). Furthermore, although Exemplar can be used for any tool, it only has APIs for JVM projects.

Sample project testing primer #

The sample-discovery library in Exemplar will consider any directory containing a *.sample.conf file under the samples root as a sample project.

A sample config file is written in HOCON and tells sample-check how to run the sample project. It can be simple:

executable: echo
args: "Hello World"

… or more complex, with multiple steps:

commands: [{
  executable: gradle
  args: originalInputs incrementalReverse
  expected-output-file: originalInputs.out
  allow-additional-output: true
}, {
  executable: gradle
  args: --quiet removeOutput incrementalReverse
  expected-output-file: incrementalTaskRemovedOutput.out
  allow-disordered-output: true
}]

See the samples configuration reference in the docs to learn what options are available.

A JUnit Test then declares which @SampleRunner and other aspects of samples tests.

@RunWith(GradleSamplesRunner.class)
@SamplesRoot("src/docs/samples")
@SamplesOutputNormalizers({JavaObjectSerializationOutputNormalizer.class, FileSeparatorOutputNormalizer.class})
public class SamplesIntegrationTest {}

Executing this test will discover and run external (that is, not embedded) sample projects and generate a JUnit test report as you’d expect.

Embedded samples testing primer #

Exemplar can discover samples within Asciidoctor source files using Asciidoctor’s AST. Here’s an example that uses SampleModifiers as well to setup Samples’ environments as well.

.Sample title
====
[.testable-sample]       (1)
=====
.hello.rb                (2)
[source,ruby]            (3)
-----
puts "hello, #{ARGV[0]}" (4)
-----

[.sample-command]        (5)
-----
$ ruby hello.rb world    (6)
hello, world             (7)
-----
=====
====

Please read the docs for embedded samples to learn about the Asciidoctor elements required to allow this to work.

We run this test using the EmbeddedSamplesRunner pointing to the docs/ root directory where Exemplar will look for documentation files.

@RunWith(EmbeddedSamplesRunner.class)
@SamplesRoot("docs")
@SampleModifiers({SetupEnvironmentSampleModifier.class, ExtraCommandArgumentsSampleModifier.class})
public class EmbeddedSamplesIntegrationTest {}

Executing this test will extract the code from Asciidoctor into a temporary project, run it, and verify it matches the declared output in the doc.

Adopt and contribute #

The best way to get started is to read the Exemplar docs and check out Exemplar’s samples and sample tests.

Exemplar is at version 0.6 at the time of writing this post, which means that the APIs may change before v1.0. That said, given that the API has gone through some revisions already and has been adopted by Gradle, breaking changes are unlikely before version 1.0.

Here are some of the things that might be good expansions of the library, and I’d love your help if you’re eager to file ideas or submit pull requests to the gradle/exemplar GitHub repo.

  • More standard normalizers such as date, time, or duration normalization
  • Generate structured metadata for search indexing
  • Allow expected output to be inlined in sample config files
  • Samples discovery embedded in Markdown or other documentation formats (#2938)

Please reach out to me (@eriwen on Twitter) if you’d like guidance adopting or contributing to this project.

Discuss

Related Posts