cue-unity / unity

unity - run experiments/regression tests on CUE modules

unity is a tool used to run experiments/regression tests on various CUE modules using different versions of CUE. The repository that contains the unity tool (this repository) is also a corpus of CUE modules against which unity is run. unity is based in part on the ideas behind Rust's crater.

unity test is currently the only implemented command.

The main features of unity test are:

• simple specification of tests and expectations via testscript test scripts
• running of Go tests for those packages in a project that use the CUE Go API (coming soon)
• cross platform (currently only Linux tested) support for running unity
• can be run locally or triggered to run as a GitHub Actions workflow via the cueckoo command
• by default runs in "safe" mode where tests are run in a Docker container on a volume-mounted copy of the CUE module under test
• multiple ways of specifying CUE versions to test against (see below)
• a --corpus mode for running unity across the git submodules of a project. The unity project itself uses this mode to test the corpus under https://github.com/cue-unity/unity/tree/main/projects
• an --update flag to update golden files in a project when a cmp-esque comparison in a testscript test script fails
• easy addition of new projects to the unity corpus

How do I add my project to the unity corpus?

Simple!

• Add a new git submodule;
• Create a PR;
• Request a review;
• Wait for the CI tests for pass;
• We will merge, everyone will benefit!

For projects which define their own cue.mod/tests manifest, you can use the PR where cue-unity/example was added to the corpus as an example.

Where a project cannot (yet) define such a manifest, the vector project PR provides an example of how to also define an overlay.

Motivation

CUE is currently missing:

• cue test (see https://cuelang.org/issue/209)
• full dependency management via modules (see https://cuelang.org/issue/434)
• a module discovery site/API similar to pkg.go.dev

Hence:

• we don't have a good way of discovering CUE modules in the wild
• module authors don't have a way to assert correctness of their CUE configurations with respect to a given version of CUE, either via cmd/cue commands or via the cuelang.org/go/... package APIs

unity is intended as a stepping stone towards these missing features. It will also:

• allow the CUE project to move more quickly with changes/performance improvements, because we have a corpus of code against which to regression test/experiment
• allow analysis of how CUE is being used in the wild, to see whether aspects of CUE can be improved (e.g. better cue vet rules, or different package/input modes)

Using unity

unity test is currently the only implemented command. unity test works in two modes: project mode (default) or --corpus mode which are described in detail below. As a quick start:

git clone https://github.com/cue-unity/unity
cd unity
git submodule update -f --init --recursive

# Run unity in corpus mode using overlays testing base versions declared in
# corpus projects
go run github.com/cue-unity/unity/cmd/unity test --corpus --overlay overlays

# As above, but also test against the source version of CUE found in the
# directory /path/to/cuesrc
go run github.com/cue-unity/unity/cmd/unity test --corpus --overlay overlays /path/to/cuesrc

# Allow unity to update cmp failures in test assertions when the second
# argument refers to a file within the testscript archive
go run github.com/cue-unity/unity/cmd/unity test --update --corpus --overlay overlays /path/to/cuesrc

Project mode

In project mode, unity runs within the context of a project that declares the unity manifest within the cue.mod/tests directory. The unity manifest is CUE package value that satisfies the #Manifest definition. The cue-unity/example project declares such a manifest:

package tests

Versions: ["go.mod", "v0.3.0-beta.5"]

Via such a manifest a project declares the latest versions of CUE against which its configurations are known to be correct, or more precisely against which its unity tests are known to pass.

The cue.mod/tests directory also contains a number of testscript test scripts. Again, considering the cue-unity/example project, it defines a basic eval.txt test script as follows:

# Verify that eval works as expect

cue eval ./...
cmp stdout $WORK/stdout.golden

-- stdout.golden --

x: 5

Every such test script is run:

• within a Docker container, unless --unsafe is provided
• within a clean working directory, referred to as $WORK (see the testscript documentation for more details)
• with a minimal environment (see the testscript documentation for more details)
• with a copy of the repository containing the CUE module under test available at $WORK/repo
• with all files in the test script archive expand to $WORK
• with an initial working directory of $WORK/repo/path/to/module for convenience

Hence the above example test script makes a copy of cue-unity/example available at $WORK/repo. The script has an initial working directory of $WORK/repo, because the cue-unity/example CUE module is defined at the root of that project repository. Therefore, the script runs cue eval ./... in the context of a copy of the CUE module under test. The golden file stdout.golden is extracted to $WORK, hence the comparison cmp stdout $WORK/stdout.golden needs to specify the full path to stdout.golden because the working directory is $WORK/repo.

Here is the output from running unity within the cue-unity/example project:

$ go run github.com/cue-unity/unity/cmd/unity test --verbose commit:91abe0de26571ef337559580442f990ded0b32f9
--- PASS: eval/go.mod

        WORK=$WORK
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=/no-home
        TMPDIR=$WORK/tmp
        devnull=/dev/null
        /=/
        :=:
        exe=

        # Verify that eval works as expect (0.043s)
        > cue eval ./...
        [stdout]
        x: 5

        > cmp stdout $WORK/stdout.golden
        PASS
--- PASS: eval/v0.3.0-beta.5

        WORK=$WORK
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=/no-home
        TMPDIR=$WORK/tmp
        devnull=/dev/null
        /=/
        :=:
        exe=

        # Verify that eval works as expect (0.042s)
        > cue eval ./...
        [stdout]
        x: 5

        > cmp stdout $WORK/stdout.golden
        PASS
--- PASS: eval/commit:91abe0de26571ef337559580442f990ded0b32f9

        WORK=$WORK
        PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        HOME=/no-home
        TMPDIR=$WORK/tmp
        devnull=/dev/null
        /=/
        :=:
        exe=

        # Verify that eval works as expect (0.043s)
        > cue eval ./...
        [stdout]
        x: 5

        > cmp stdout $WORK/stdout.golden
        PASS

First, the two base versions declared as supported in the cue-unity/example manifest are run. Then, the command line version commit:91abe0de26571ef337559580442f990ded0b32f9 is also tested, which is a reference to a commit referenced by the master branch of the CUE project.

Any project that uses CUE can also use unity as part of its own testing/CI regime. For example, the Play with Go project runs unity as part of its GitHub workflows. This fits nicely with the fact that the Play with Go project is also part of the unity corpus.

--corpus mode

In --corpus mode, unity tests all of the git submodules of a repository in project mode. Taking the unity repository itself as an example corpus:

$ go run github.com/cue-unity/unity/cmd/unity test --corpus --overlay overlays --nopath refs/changes/41/8841/3
testing projects/github.com/play-with-go/play-with-go against version go.mod
testing projects/github.com/cue-unity/example against version go.mod
testing projects/github.com/cue-unity/example against version v0.3.0-beta.5
testing projects/github.com/timberio/vector against version v0.3.0-beta.5
testing projects/github.com/TangoGroup/cfn-cue against version v0.3.0-beta.5
testing projects/github.com/play-with-go/play-with-go against version refs/changes/41/8841/3
testing projects/github.com/cue-unity/example against version refs/changes/41/8841/3
testing projects/github.com/timberio/vector against version refs/changes/41/8841/3
testing projects/github.com/TangoGroup/cfn-cue against version refs/changes/41/8841/3

In this case the base versions declared as supported by each project in the corpus are tested first. Then the command line specified refs/changes/41/8841/3 is also tested. This is a reference to a CL that was in progress at the time (since merged).

Specifying CUE versions

unity supports different ways of specifying the CUE version against which to test:

• go.mod - the version of CUE resolved via the Go module in which the CUE module under test is found
• $semver - any official CUE (pre)release, e.g. v0.3.0-beta.5
• /path/to/cue - an absolute path to the location of a Go module where cuelang.org/go can be resolved (this could be the CUE project itself)
• PATH - use the cue command found on your PATH. This binary must be compiled for the operating system and architecture of the target Docker image if you are running in normal/safe mode
• commit:$hash - a commit on the master branch of the CUE project, e.g. commit:a0e19707b99d8e76caf3234c42761a73d0fb85f7
• $CLref - a CUE project Gerrit CL patchset reference, e.g. refs/changes/21/8821/3

FAQ

Please see the wiki FAQ.