TackleTest: Automated Unit and UI Test Generation

This repository contains a Python-based command-line interface (CLI) to TackleTest, which provides automated test-generation capabilities for Java unit testing and end-to-end testing of web applications. TackleTest-Unit (supported by the CLI command tkltest-unit) automatically generates unit-level test cases for Java applications. TackleTest-UI (supported by the CLI command tkltest-ui), automatically generates end-to-end test cases for web applications that exercise the application under test via its user interface.

These CLI commands invoke core test generators, whose codebase resides in the related tackle-test-generator-core repository.

1. Overview
2. Installing and running the CLI
3. TackleTest CLI in action on sample apps
4. TackleTest-Unit documentation: quick-start guide, detailed user guide, configuration options
5. TackleTest-UI documentation: quick-start guide, detailed user guide, configuration options
6. Known tool issues
7. Additional resources (demos, presentations, blog)

Overview

TackleTest-Unit automatically generates unit test cases for Java applications. Each generated test case focuses on exercising the visible methods of an application class. Test generation can be done on a specific set of application classes or the entire application.

Among its many features, TackleTest-Unit:

• Supports automatic generation of test assertions: tests can be generated with or without assertions
• Combines different coverage metrics and generates coverage reports: the supported coverage metrics include conventional code coverage criteria, such as statement and branch coverage, as well as a novel type-based combinatorial coverage coverage of Java methods
• Accommodates developer-written unit test suites: it can perform coverage-driven augmentation and directed test generation to enhance the coverage achieved by a given test suite
• Supports multiple build tools: Maven, Gradle, and Ant
• Integrates two opens-source test-generation tools: EvoSuite and Randoop

For more details of TackleTest-Unit, see the quick-start guide and the detailed user guide.

TackleTest-UI supports automatic generation of end-to-end test cases for web applications. These test cases exercise the application under test via its user interface and drive execution through different application tiers. Some of the features of TackleTest-UI include

• Model-based generation of test cases for web applications: infers state-transition model of the webapp under test and generates test cases from paths in the model
• Integration with the state-of-the-art web-crawling tool Crawljax
• Generation two test suites using the TestNG framework: one using Crawljax API and the other using Selenium API

For more details of TackleTest-UI, see the quick-start guide and the detailed user guide.

Installing and running TackleTest CLI

The CLI command can be installed locally to be run, or it can be run in a Docker container, in which case the various dependencies (e.g., Python, Java, Maven) need not be installed.

Using a published Docker image

The simplest way is to run the CLI using a published Docker image. For each released version of TackleTest, the docker image (tagged with the version number) is published on the GitHub Container Registry.

To run TackleTest from docker, the directory containing the TackleTest configuration file for the application under test (AUT)---along with the AUT classes and library dependencies, in the case of TackleTest-Unit---has to be mounted onto the container. Suppose that the directory containing the AUT configuration file is /home/user/aut. To mount this directory and run TackleTest with the AUT configuration file, use the following commands (for unit and UI test generation, respectively) after pulling the image from the registry:

docker run --rm -v /home/user/aut:/app/tackle-test-cli ghcr.io/konveyor/tackle-test-generator-cli:latest tkltest-unit --config-file /app/tackle-test-cli/tkltest_config.toml --verbose generate ctd-mplified'

docker run --rm -v /home/user/aut:/app/tackle-test-cli ghcr.io/konveyor/tackle-test-generator-cli:latest tkltest-ui --config-file /app/tackle-test-cli/tkltest_config.toml --verbose generate'

Note that, in these commands, the configuration file is specified as an absolute path in the container (/app/tackle-test-cli/tkltest_config.toml). You can also specify a relative path, provided that path is relative to /app/tackle-test-cli in the container.

The results of test generation or execution in the container are available under /home/user/aut on the host machine (/app/tackle-test-cli in the container).

For convenience in using the TackleTest via Docker, you can create a workspace directory for TackleTest (e.g., /home/user/tkltest-workspace) and create the following aliases:

alias tkltest-unit='docker run --rm -v /home/user/tkltest-workspace:/app/tackle-test-cli ghcr.io/konveyor/tackle-test-generator-cli:latest tkltest-unit'

alias tkltest-ui='docker run --rm -v /home/user/tkltest-workspace:/app/tackle-test-cli ghcr.io/konveyor/tackle-test-generator-cli:latest tkltest-ui'

With these aliases set, you can simply use tkltest-unit and tkltest-ui commands instead of the long docker run ... command. If you have the TackleTest CLI repo cloned, you could use the CLI repo as the workspace directory. If you are using a TackleTest image tagged other than latest, remember to substitute latest with the specific version tag in the docker commands.

Note: For running tkltest-unit via Docker, the option java_jdk_home (see tkltest-unit configuration options) should be left empty because the environment variable JAVA_HOME is set in the container envrionment. Providing a value for java_jdk_home will override the JAVA_HOME setting in the container, resulting in failure.

Installing a released version with all dependencies included

• Download and unzip a released version of TackleTest with all dependencies included (i.e., a release archive file named *-all-deps.tgz or *-all-deps.zip)

• Install TackleTest CLI by following the command installation instructions

Installing from a repo snapshot

• Clone the repo or download codebase zip

• Download all library dependencies by following the library download instructions

• Install TackleTest CLI by following the command installation instructions

After the CLI has been installed, try out the tkltest-unit and tkltest-ui commands:

tkltest-unit --help
tkltest-ui --help

Notes on upgrading an installation:

• If you download a newer TackleTest release, follow the command installation instructions and check the TackleTest version by running the command tkltest-unit -v or tkltest-ui -v.
• If you upgrade the repo snapshot, please be sure to rerun the download script to pull in the updated published library dependencies for the TackleTest Core components.

TackleTest CLI in action on sample apps

To see the TackleTest-Unit in action on a sample Java application, set JAVA_HOME to your JDK installation and run the command:

tkltest-unit --config-file ./test/data/irs/tkltest_config.toml --verbose generate ctd-amplified

This command will take a few minutes to complete. The --verbose option allows to view its progress in the standard output. Note that during test sequence initialization phase, the output is not printed to the standard output but rather to log files. You can open those log (via tail -f <logfile>) to view the progress during this phase.

To see the TackleTest-UI in action on a sample web application, follow these steps:

1. Make sure that the sample web applications, which are included as Git submodules, are checked out:

   git submodule init
   git submodule update
   

   Alternatively, you can get the sample web applications from this repo.

2. Deploy one of the sample web applications, addressbook or petclinic, in Docker:

   cd test/ui/data/webapps/addressbook && ./deploy_app.sh start && cd ../../../../../
   

   The addressbook webapp can be opened at http://localhost:3000/addressbook/

   cd test/ui/data/webapps/petclinic && ./deploy_app.sh start --build && cd ../../../../../
   

   The petclinic webapp can be opened at http://localhost:8080/

3. Run UI test generation on the app:

   tkltest-ui --config-file ./test/ui/data/addressbook/tkltest_ui_config.toml --verbose generate
   

   or

   tkltest-ui --config-file ./test/ui/data/petclinic/tkltest_ui_config.toml --verbose generate
   

This command will take a few minutes to run; during its execution, the Chrome browser will be started in headless mode to launch the petclinic webapp for crawling and test generation. The --verbose option redirects crawl logs to a file called <app-nane>_crawljax_runner.log. You can view the progress of crawling via the command tail -f <logfile>.

Known Tool Issues

TackleTest-Unit

1. On apps with native UI (e.g., swing), the tool can sometimes get stuck during sequence execution (even though it uses a Java agent for replacing calls to GUI components); as a workaround, users can exclude UI-related classes from the set of test targets.

2. Coverage in JEE apps could be low because of limited JEE mocking support.

3. A known issue on Windows OS is that TackleTest might exceed Windows limit of 260 characters for a file system folder path name length. Tackle-test mimics the structure of the application under its output directory, to enable generating tests in the same package as the class under test, and gaining access to all its non-private members and methods. If your application has a deep package hierarchy, these paths might exceed the 260 characters length limit. For Windows 10, there are online instructions available on how to enable long paths and avoid this limitation.

TackleTest-UI

1. The form-fill actions performed by Crawljax have some limitations. Crawljax does not have information about disabled form fields and attempts to perform actions on them (e.g., via Selenium sendKeys method) without encountering exceptions. This can result in such actions occuring also in the generated test cases, but these tests would still pass (the action on a disabled field essentially becomes a NOP). Crawljax also does not perform any verification after performing the form-fill actions or analysis to associate form fields with submit buttons (other than the association provided in form data specification). Aside from the specified forms, Crawljax simply performs all form-fill actions available on a web page before performing any clicks.

2. Currently, support is available for only the Chrome browser (in normal and "headless" modes); support for Firefox will be added soon.

Additional Resources

TackleTest-Unit resources

• Article on TackleTest
• Short demo of TackleTest (5 min)
• Detailed TackleTest presentation and demo from Konveyor meetup (48 min)
• Slide deck on TackleTest