Rainforest-cli

A command line interface to interact with Rainforest QA.

This is the easiest way to integrate Rainforest with your deploy scripts or CI server. See our documentation on the subject.

The CLI uses the Rainforest API which is documented at https://app.rainforestqa.com/docs

Installation

Docker

$ docker pull gcr.io/rf-public-images/rainforest-cli
$ docker run gcr.io/rf-public-images/rainforest-cli --version
Rainforest CLI version 2.15.1 - build: docker

Brew

If you are on OSX and use Brew, you can install the CLI with the command:

brew install rainforestapp/public/rainforest-cli

Binaries

Get the CLI binaries from our download page and follow the instructions.

The CLI will check for updates and automatically update itself on every use unless the global flag --skip-update is used.

Migrating from our old CLI

The previous version of our CLI is deprecated and parts of it no longer work. To upgrade, follow our migration guide.

Basic Usage

To authenticate against Rainforest you'll need your API token which you can get from your integrations settings page.

Pass the token into the CLI through the RAINFOREST_API_TOKEN environment variable or with the global flag --token.

CLI commands are formatted as follows:

rainforest [global flags] <command> [command-specific-flags] [arguments]

Options

Running Tests

Run all tests in the foreground and report.

rainforest run all

Run all your tests in the background and exit the process immediately.

rainforest run all --bg

Run all tests with tag 'run-me' and abort previous in-progress runs.

rainforest run --tag run-me --conflict abort

Run all tests and generate a junit xml report.

rainforest run all --junit-file results.xml

Run individual tests in the foreground and report.

rainforest run <test_id1> <test_id2>

Run a run group.

⚠️ This uses the configuration defined in the run group (environment, browsers, crowd, location). If you wish to run tests from a run group without using the run group's configuration, you will need to use the Rainforest API directly, passing a run_group_id parameter to the POST /runs endpoint. ⚠️

rainforest run --run-group <run_group_id>

Rerunning Failed Tests

rainforest rerun <failed_run_id>

The failed_run_id argument is optional. If none is passed in, the CLI will look for a run ID in the RAINFOREST_RUN_ID environment variable.

Creating and Managing Tests

Create new Rainforest test in RFML format (Rainforest Markup Language).

rainforest new

You may also specify a custom test title or file name.

rainforest new "My Awesome Title"

Validate your tests for syntax and correct RFML ids for embedded tests. Use the --token options or RAINFOREST_API_TOKEN environment variable to validate your tests against server data as well.

rainforest validate

Validate RFML syntax of a specified file. This command just validates RFML syntax for more complex validation including checking embedded tests id correctness and existence of potential circural dependiences in tests use general command.

rainforest validate /path/to/test/file.rfml

Upload tests to Rainforest

rainforest upload

Upload a specific test to Rainforest

rainforest upload /path/to/test/file.rfml

Remove RFML file and remove test from Rainforest test suite.

rainforest rm /path/to/test/file.rfml

Download all tests from Rainforest

rainforest download

Download tests filtered by tags, site, and smart folder

rainforest download --tag foo --tag bar --site-id 123 --folder 456

Download specific tests based on their id on the Rainforest dashboard

rainforest download 33445 11232 1337

Running Local RFML Tests Only

If you want to run a local set of RFML files (for instance in a CI environment), use the run -f option:

rainforest run -f [FILES OR FOLDERS]

run -f accepts any number of files and folders as arguments (folders will be recursively checked for *.rfml files). All embedded tests must be included within FILES OR FOLDERS.

There is a specific metadata option in RFML files for run -f: # execute: true|false, which indicates whether a test should be run by default (defaults to true). Embedded tests that are not usually run directly should specify # execute: false.

The following options are specific to run -f or behave differently:

• --tag TAG_NAME: only run tests that are tagged with TAG_NAME (which can be a comma-separated list of tags). Note that this filters within local RFML files, not tests stored on Rainforest. Tests that are not tagged with TAG_NAME will not be executed but may be still be uploaded if they are embedded in another test.
• --exclude FILE: exclude the test in FILE from being run, even if # execute: true is specified.
• --force-execute FILE: execute the test in FILE even if # execute: false is specified.

Run-level setting options (--browsers, --environment_id, etc) behave the same for run -f. Other test filtering options (such as --run-group, --site, etc) cannot be used in conjunction with run -f.

Viewing Account Specific Information

See a list of all of your sites and their IDs

rainforest sites

See a list of all of your environments and their IDs

rainforest environments

See a list of all of your smart folders and their IDs

rainforest folders

See a list of all of your browsers and their IDs

rainforest browsers

See a list of all of your features and their IDs

rainforest features

See a list of all of your run groups and their IDs

rainforest run-groups

To generate a junit xml report for a test run which has already completed

rainforest report <run-id> --junit-file rainforest.xml

Updating Tabular Variables

Upload a CSV to create a new tabular variables.

rainforest csv-upload --import-variable-name my_variable PATH/TO/CSV.csv

Upload a CSV to update an existing tabular variables.

rainforest csv-upload --import-variable-name my_variable --overwrite-variable PATH/TO/CSV.csv

Uploading Mobile Apps

Upload a mobile app to Rainforest.

rainforest mobile-upload --site-id <site_id> --environment-id <environment_id> PATH/TO/mobile_app.ipa

• --site-id SITE_ID - The site ID of the app you are uploading. You can see a list of your site IDs with the sites command.
• --environment-id ENVIRONMENT_ID - The environment ID of the app you are uploading. You can see a list of your environment IDs with the environments command.
• --app-slot SLOT - An optional flag for specifying the app slot of your app, if your site-environment contains multiple apps. Valid values are from 1 to 100, and the default value is 1.

Options

Global

• --token <your-rainforest-token> - your API token if it's not set via the RAINFOREST_API_TOKEN environment variable
• --skip-update - Do not automatically check for CLI updates

Writing Tests

Rainforest Tests written using RFML have the following format

#! [RFML ID]
# title: [TITLE]
# start_uri: [START_URI]
# tags: [TAGS]
# site_id: [SITE ID]
# browsers: [BROWSER IDS]
# feature_id: [FEATURE_ID]
# state: [STATE]
# [OTHER COMMENTS]

[ACTION 1]
[QUESTION 1]

# redirect: [REDIRECT FLAG]
- [EMBEDDED TEST RFML ID]

Action with an embedded screenshot: {{ file.screenshot(./relative/path/to/screenshot.jpg) }}
Response with an embedded file download: {{ file.download(./relative/path/to/file.txt) }}

[ACTION 3]
[QUESTION 3]

... etc.

Required Fields:

• RFML ID - Unique identifier for your test. For newly generated tests, this will be a UUID, but you are free to change it for easier reference (for example, your login test might have the id login_test).
• TITLE - The title of your test.
• START_URI - The path used to direct the tester to the correct page to begin the test.
• ACTION 1, ACTION 2, ... - The directions for your tester to follow in this step. You must have at least one step in your test.
• QUESTION 1, QUESTION 2, ... - The question you would like your tester to answer in this step. You must have at least one step in your test.

Optional Fields:

• SITE ID - Site ID for the site this test is for. You can find your available site IDs with the sites command. Sites can be configured at https://app.rainforestqa.com/settings/sites.
• BROWSER IDS - Comma separated list of browsers for this test. You can reference your available browsers with the browsers command. If left empty or omitted, your test will default to using your account's default browsers.
• TAGS - Comma separated list of your desired tags for this test.
• FEATURE_ID - Feature ID for the feature that this test is a part of. You can find your available feature IDs with the features command.
• STATE - State of the test. Valid states are enabled, disabled and draft.
• OTHER COMMENTS - Any comments you'd like to save to this test. All lines beginning with # will be ignored by Rainforest unless they begin with a supported data field, such as tags or start_uri.
• REDIRECT FLAG - A true or false flag to designate whether the tester should be redirected. The default value is true. This flag is only applicable for embedded tests and the first step of a test.
• EMBEDDED TEST RFML ID - Embed the steps of another test within the current test using the embedded test's RFML ID.

For more information on embedding inline screenshots and file downloads, see our examples.

For more information on test writing see the documentation.

Command Line Options

Popular command line options are:

• --browsers ie8 or --browsers ie8,chrome - specify the browsers you wish to run against. This overrides the test level settings. Valid browsers can be found in your account settings.
• --tag TAG_NAME - filter tests by tag. Can be used multiple times for filtering by multiple tags.
• --site-id SITE_ID - filter tests by a specific site. You can see a list of your site IDs with rainforest sites.
• --folder ID/--filter ID - filter tests in specified folder.
• --feature ID - filter tests in a feature.
• --run-group ID - run/filter based on a run group. When used with run, this trigger a run from the run group; it can't be used in conjunction with other test filters.
• --environment-id - run your tests using this environment. Otherwise it will use your default environment
• --conflict OPTION - use the abort option to abort any runs in progress in the same environment as your new run. use the abort-all option to abort all runs in progress.
• --bg - creates a run in the background and rainforest-cli exits immediately after. Do not use if you want rainforest-cli to track your run and exit with an error code upon run failure (ie: using Rainforest in your CI environment). Cannot be used together with --max-reruns.
• --crowd [default|automation|automation_and_crowd|on_premise_crowd] - select automation or your crowd of testers (for clients with on premise testers). For more information, contact us at [email protected].
• --wait RUN_ID - wait for an existing run to finish instead of starting a new one, and exit with a non-0 code if the run fails. rainforest-cli will exit immediately if the run is already complete.
• --fail-fast - return an error as soon as the first failed result comes in (the run always proceeds until completion, but the CLI will return an error code early). If you don't use it, it will wait until 100% of the run is done. Has no effect with --bg and cannot be used together with --max-reruns.
• --custom-url - specify the URL for the run to use when testing against an ephemeral QA environment. This will create a new environment for the run.
• --git-trigger - only trigger a run when the last commit (for a git repo in the current working directory) has contains @rainforest and a list of one or more tags. E.g. "Fix checkout process. @rainforest #checkout" would trigger a run for everything tagged checkout. This over-rides --tag and any tests specified. If no @rainforest is detected it will exit 0.
• --description "CI automatic run" - add an arbitrary description for the run.
• --release "1a2b3d" - add an ID to associate the run with a release. Commonly used values are commit SHAs, build IDs, branch names, etc.
• --flatten-steps - Use with rainforest download to download your tests with steps extracted from embedded tests.
• --test-folder /path/to/directory - Use with rainforest [new, upload, export]. If this option is not provided, rainforest-cli will, in the case of 'new' create a directory, or in the case of 'upload' and 'export' use the directory, at the default path ./spec/rainforest/.
• --junit-file - Create a junit xml report file with the specified name. Must be run in foreground mode, or with the report command. Uses the rainforest api to construct a junit report. This is useful to track tests in CI such as Jenkins or Bamboo.
• --run-id - Only used with the report command. Specify a past rainforest run by ID number to generate a report for.
• --import-variable-csv-file /path/to/csv/file.csv - Use with run and --import-variable-name to upload new tabular variable values before your run to specify the path to your CSV file.
• --import-variable-name NAME - Use with run and --import-variable-csv-file to upload new tabular variable values before your run to specify the name of your tabular variable. You may also use this with the csv-upload command to update your variable without starting a run.
• --single-use - Use with run or csv-upload to flag your variable upload as single-use. See --import-variable-csv-file and --import-variable-name options as well.
• --max-reruns - If set to a value > 0 and a test fails, the CLI will re-run failed tests a number of times before reporting failure. If --junit-file <filename> is also used, the JUnit reports of reruns will be saved under <filename>.1, <filename>.2 etc. Cannot be used together with --fail-fast.

Support

Email [email protected] if you're having trouble using the CLI or need help with integrating Rainforest in your CI or development workflow.

Contributing

1. Fork it
2. Create a feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Release process

Check the .circleci/config.yml for the latest, but currently merging to master will build and deploy to the following Equinox channels:

Development + release process is:

Development PR

1. Branch from master
2. Do work
3. Open PR against master
4. Merge to master

Changelog PR

1. Branch from master to update CHANGELOG.md to include the commit hashes and release date
2. Update the version constant in rainforest-cli.go following semantic versioning
3. Merge to master

Release

1. Release via GitHub Releases
2. Update https://github.com/rainforestapp/homebrew-public to use the latest stable build url from equinox

Releasing a beta version

Simply tag a commit with an alpha or beta version and push to trigger the deploy_beta CI job:

git tag vX.Y.Z-alpha.N # or vX.Y.Z-beta.N
git push origin vX.Y.Z-alpha.N