Rasa Train-Test Model GitHub Action

The GitHub action to run easily rasa train and rasa test in the CIs.

Input Arguments

In order to pass the input parameters to the GH action, you have to use the with argument in a step that uses the GH action, e.g.:

jobs:
  my_first_job:
    name: My first job
    runs-on: ubuntu-latest
    steps:
      # Checks-out GitHub repository
      # more: https://github.com/actions/checkout
      - uses: actions/checkout@v2

      # Run rasa train and rasa test
      - name: My first step
        uses: RasaHQ/rasa-train-test-gha@main
        with:
          test_type: nlu

Input	Description	Default
`rasa_version`	The Rasa Open Source version used to run test and train	`latest-full`
`rasa_image`	Custom Rasa Docker image. Useful if you use Rasa with custom Python modules.	`none`
`rasa_train`	Run `rasa train`	`true`
`rasa_test`	Run `rasa test`	`true`
`data_validate`	Validates domain and data files to check for possible mistakes	`true`
`data_validate_args`	Additional arguments passed to the `rasa data validate` command	`none`
`fine_tune`	Fine-tune an existing model with new training dataset	`false`
`workspace`	The root directory containing your Rasa Open Source project	`${{ github.workspace }}`
`train_type`	The types of training (available types: `core`/`nlu`/`all`)	`all`
`train_args`	Additional arguments passed to the `rasa train` command	`none`
`test_type`	The types of tests to run (available types: `core`/`nlu`/`all`)	`all`
`test_nlu_args`	Additional arguments passed to the `rasa test nlu` command	`none`
`test_core_args`	Additional arguments passed to the `rasa test core` command	`none`
`publish_summary`	Publish tests summary as a PR comment	`true`
`github_token`	GitHub Token - required to add a comment with summary	`none`
`configuration`	Model configuration file	`config.yml`
`model`	Path to a file with a model. Use existing model instead of training a new one	`none`
`cross_validation`	Switch on cross validation mode. Any provided model will be ignored	`false`
`configuration_name`	Configuration name used in summary. If not provided a file name is used	`none`
`data_name`	Data name used in summary. If not provided a directory name is used	`default`
`compare_report`	Path to a report that will be used to compare results	`none`
`result_directory`	Directory name where results are stored in	`results`
`report_directory`	Directory name where reports are stored in	`reports`
`gomplate_image`	Custom gomplate image. Useful if you use custom gomplate image	`hairyhenderson/gomplate`
`tmpfs_directory`	The directory location where tmpfs mounts	`/.config`

Outputs

The list of available output variables:

Output	Description
`report`	Return report as JSON

Report

The GH action generates two reports, a report with a summary of run tests, the report is available as JSON and CSV file. The example of a report generated by the GH action can be found here.

Example Usage

In the example below, we are using the Rasa Demo data.

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}
            # ...

The GitHub action by default adds a PR comment with summary (the summary comment can be disabled by setting the publish_summary input argument to false):

Use JSON report in GH workflow

The GH action returns a JSON report as an output. The following example shows how to use the output in a GH workflow.

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              id: action
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}

            # We have to convert the output to JSON by using fromJSON built-in function
            # more: https://docs.github.com/en/free-pro-team@latest/actions/reference/context-and-expression-syntax-for-github-actions#fromjson
            # syntax: fromJSON(steps.action.outputs.report).<data_name>[<configuration_name|configuration>]
            # example: fromJSON(steps.action.outputs.report).default['config.yml']
            - name: Check output
              if: fromJSON(steps.action.outputs.report).default['config.yml'].intent_classification.accuracy >= 0.8
              run: |
                echo "I'm doing extra work..."
                echo ${{ fromJSON(steps.action.outputs.report).default['config.yml'].intent_classification.accuracy }}

Advanced examples

Compare results with the base report

It possible to compare results to the other report, the feature is useful for example to see that a model is better than before changes. The difference against the report that we compare to is included in brackets.

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}
                # A path to the report that we want to compare to
                compare_report: 'report_to_compare.json'
                test_type: 'nlu'
            # ...

Use the existing model

It's possible to use the existing model instead of training a new one.

jobs:
    test:
        # ...
        steps:
            # ...
            - name: Download Rasa X model
              run: |
                wget https://github.com/RasaHQ/rasa-x-demo/blob/0.33.0/models/model.tar.gz?raw=true \
                  -O test_model.tar.gz

            - name: Train and Test Rasa Demo
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}
                # If a file with the model is provided, training is disabled automatically
                model: test_model.tar.gz
            # ...

Fine-tune the existing model

You can fine-tune an existing model with new training dataset. Please see incremental training for more details.

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}
                # By default, the number of epoches is defined in model configuration
                # You can shorten the epoches by using the --epoch-fraction flag
                fine_tune: 'true'
                train_args: >-
                  --epoch-fraction 0.5
                # ...

Cross-validate NLU model

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}
                # Switch on cross validation mode. Any provided model will be ignored
                cross_validation: 'true'
                # Number of cross validation folds (cross validation only)
                # All available arguments: rasa test nlu --help
                test_nlu_args: '--folds 3'
                test_type: 'nlu'
            # ...

In a case where the cross-validation mode is enabled, a summary published as a PR comment includes Intent Cross-Validation Results and Entity Cross-Validation Results, e.g.

Upload results and reports as artifacts

jobs:
    train_and_test:
        # ...
        steps:
            # ...
            - name: Train and Test Rasa Demo
              id: action
              uses: RasaHQ/rasa-train-test-gha@main
              with:
                # List of available tags: https://hub.docker.com/r/rasa/rasa/tags
                rasa_version: '2.0.0-full'
                # In order to add a PR comment with summary
                # a GH Token has to be pass to the GH action
                github_token: ${{ secrets.GITHUB_TOKEN }}

            # The 'actions/upload-artifact' action to upload files
            # More: https://github.com/actions/upload-artifact
            - uses: actions/upload-artifact@v2
              with:
                name: rasa-demo-cfg
                path: |
                  results
                  reports

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

RasaHQ / rasa-train-test-gha

Programming Languages

Labels

Projects that are alternatives of or similar to rasa-train-test-gha