All Projects → mmcc007 → Sylph

mmcc007 / Sylph

Licence: gpl-3.0
Runs Flutter integration tests on real devices in cloud.

Programming Languages

dart
5743 projects

Projects that are alternatives of or similar to Sylph

Movieapp
🎬 MovieApp is a Flutter application built to demonstrate the use of modern development tools with best practices implementation like Modularization, BLoC, Dependency Injection, Dynamic Theme, Cache, Shimmer, Testing, Flavor, CI/CD, etc.
Stars: ✭ 117 (-8.59%)
Mutual labels:  ci-cd, integration-testing
ecosystem-ci
Automate issue discovery for your projects against Lightning nightly and releases.
Stars: ✭ 41 (-67.97%)
Mutual labels:  integration-testing, ci-cd
Drone Gae
Drone plugin for managing deployments and services on Google App Engine (GAE)
Stars: ✭ 96 (-25%)
Mutual labels:  ci-cd
Nginx Builder
A tool to build deb or rpm package of required Nginx version from the source code, with the ability to connect third-party modules. Nginx parameters are set in the yaml configuration file.
Stars: ✭ 123 (-3.91%)
Mutual labels:  ci-cd
Julie
A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!
Stars: ✭ 104 (-18.75%)
Mutual labels:  ci-cd
Hsac Fitnesse Fixtures
An environment to define and run integration tests. It contains Fitnesse fixture (base) classes and a baseline FitNesse installation.
Stars: ✭ 99 (-22.66%)
Mutual labels:  integration-testing
Frisby
Frisby is a REST API testing framework built on Jest that makes testing API endpoints easy, fast, and fun.
Stars: ✭ 1,484 (+1059.38%)
Mutual labels:  integration-testing
Autotag
Git repository version tagging tool
Stars: ✭ 90 (-29.69%)
Mutual labels:  ci-cd
Testcontainers Rs
A library for integration-testing against docker containers from within Rust.
Stars: ✭ 124 (-3.12%)
Mutual labels:  integration-testing
Fabric8 Platform
Generates the distribution of the fabric8 microservices platform
Stars: ✭ 105 (-17.97%)
Mutual labels:  ci-cd
Automation Arsenal
Curated list of popular Java and Kotlin frameworks, libraries and tools related to software testing, quality assurance and adjacent processes automation.
Stars: ✭ 105 (-17.97%)
Mutual labels:  integration-testing
Quiz
Example real time quiz application with .NET Core, React, DDD, Event Sourcing, Docker and built-in infrastructure for CI/CD with k8s, jenkins and helm
Stars: ✭ 100 (-21.87%)
Mutual labels:  ci-cd
Kontinuous
The Kubernetes Continuous Integration & Delivery Platform (CI/CD) 🔄
Stars: ✭ 115 (-10.16%)
Mutual labels:  ci-cd
Gaucho
A Python CLI tool for Rancher's API
Stars: ✭ 96 (-25%)
Mutual labels:  ci-cd
Kubetest
Kubernetes integration testing in Python via pytest
Stars: ✭ 122 (-4.69%)
Mutual labels:  integration-testing
Aws Workflows On Github
Workflows for automation of AWS services setup from Github CI/CD
Stars: ✭ 95 (-25.78%)
Mutual labels:  ci-cd
Pytest Monitor
Pytest plugin for analyzing resource usage during test sessions
Stars: ✭ 105 (-17.97%)
Mutual labels:  integration-testing
Gest
👨‍💻 A sensible GraphQL testing tool - test your GraphQL schema locally and in the cloud
Stars: ✭ 109 (-14.84%)
Mutual labels:  integration-testing
React Native Circleci Orb
A CircleCI Orb to Simplify Testing your React Native App
Stars: ✭ 126 (-1.56%)
Mutual labels:  integration-testing
Next Page Tester
DOM integration testing for Next.js
Stars: ✭ 122 (-4.69%)
Mutual labels:  integration-testing

pub package Build Status Build status codecov

Sylph

A sylph is a mythological invisible being of the air. Wikipedia

Sylph

Sylph is a command line utility for running Flutter integration and end-to-end tests on pools of real iOS and Android devices in the cloud. Sylph runs on mac, linux and windows and also in a CI environment.

Sylph works with AWS Device Farm for up to hundreds of Android and iOS devices in a single run.

Installation

pub global activate sylph

Usage

sylph

or, if not using the default config file:

sylph -c <path to config file>

General usage:

usage: sylph [--help] [--config <config file>] [--devices <all|android|ios>] [--verbose]

sample usage: sylph

-c, --config=<sylph.yaml>          Path to config file.
                                   (defaults to "sylph.yaml")

-d, --devices=<all|android|ios>    List devices available in cloud.
                                   [all, android, ios]

-v, --verbose                      Noisy logging, including all shell commands executed.
-h, --help                         Display this help information.

Dependencies

AWS CLI

Install AWS Command Line Interface (AWS CLI)

MacOS/Linux:

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

Windows:

pip install  awscli 

For alternative install options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

AWS CLI Credentials

Configure the AWS CLI credentials:

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

For alternative configuration options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Test AWS CLI

Confirm AWS CLI is installed and configured correctly by running an AWS command. For example, the following command should generate output:

aws devicefarm list-projects

Configuration

Configuration information is passed to Sylph using a configuration file. The default config file is called sylph.yaml:

# Config file for Flutter tests on real device pools.
# Auto-creates projects and device pools if needed.
# Configures android and ios test runs.
# Builds app, uploads and runs tests.
# Then monitors tests, returns final pass/fail result and downloads artifacts.
# Note: assumes the 'aws' command line utility is logged-in.
# Note: to build the debug iOS app, certain environment variables are required.

# sylph config
tmp_dir: /tmp/sylph
artifacts_dir: /tmp/sylph_artifacts
# local timeout per device farm run
sylph_timeout: 720 # seconds approx
# run on ios and android pools concurrently (for faster results)
concurrent_runs: true

# device farm config
project_name: App Integration Tests
default_job_timeout: 10 # minutes, set at project creation

device_pools:

  - pool_name: android pool 1
    pool_type: android
    devices:
      - name: Google Pixel 2
        model: Google Pixel 2
        os: 8.0.0

  - pool_name: ios pool 1
    pool_type: ios
    devices:
      - name: Apple iPhone X
        model: A1865
        os: 11.4

test_suites:

  - test_suite: example tests 1
    main: test_driver/main.dart
    tests:
      - test_driver/main_test.dart
    pool_names:
      - android pool 1
      - ios pool 1
    job_timeout: 15 # minutes, set per job, over-rides default job timeout above

Multiple test suites, consisting of multiple tests, can be run on each device in each device pool. The 'main' app must include a call to enableFlutterDriverExtension().

Device pools can consist of multiple devices. Devices in a device pool must be of the same type, iOS or Android.

Note: If running on linux or windows, tests can only be run on Android devices. To run tests on both Android and iOS use a mac CI provider.

Building an iOS debug app

To build a testable iOS app locally, that can run on any real device in the cloud, the following environment variable must be present:

  • TEAM_ID
    This is the Developer Portal Team ID. It is of the form 'ABCDEFGHIJ'.

A check is made before the start of a run to confirm this environment variable is present.

Note: if not running on an iOS pool this environment variable is not required.

Populating a device pool

To add devices to a device pool, pick devices from the list provided by

sylph -d android
or
sylph -d ios

and add to the appropriate pool type in sylph.yaml. The listed devices are devices currently available on Device Farm.

Configuration Validation

The sylph.yaml is validated to confirm the devices are available on Device Farm and tests are present before starting a run.

If running on an iOS pool, the iOS-related environment variables must be defined.

Configuring Flavors

A reference flavor app can be found in example/flavors. It is taken from https://github.com/flutter/flutter/tree/master/dev/integration_tests/flavors and works for android and iOS. It is currently recommended that you follow this pattern when implementing flavors in your app.

To enable testing on a flavor add the following to your sylph.yaml:

flavor: <name of flavor>

Configuring a CI Environment for Sylph

In addition to running from the command line, Sylph also runs in a CI environment.

AWS CLI Credentials for CI

The following AWS CLI credentials are required in a CI environment:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

For details on other credentials see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

iOS builds

To build the iOS app, the provisioning profile and certificate must be installed on the CI build machine. To install these dependencies, Fastlane's match is used. Sylph will detect it is running in a CI environment (using the 'CI' environment variable), and will install fastlane and fastlane scripts. The scripts are used to install the dependencies using Fastlane's match. The iOS build can then complete as normal.

The following environment variables are required by a CI build to use Fastlane match:

  • PUBLISHING_MATCH_CERTIFICATE_REPO
    This is the location of the private match repo. It expects an ssh-based url. For example, ssh://[email protected]/private_repos/match.git
  • MATCH_PASSWORD
    This is the password that was used to encrypt the git repo's contents during match setup.

For details on how to setup Match see:
https://docs.fastlane.tools/actions/match/

The following are required by sylph in a CI environment to connect to the match host. The match host is running a ssh server that connects to the git server which serves the match repo. This configuration is required so that PUBLISHING_MATCH_CERTIFICATE_REPO will work via ssh:

  • SSH_SERVER
    This is used to configure the CI's ssh client to find the match host. For example, private.mycompany.com.
  • SSH_SERVER_PORT
    This is used to configure the CI's ssh client to find the match host's ssh port. For example, 22.

As with running from the command line, the following environment variable is also required by Sylph in a CI environment:

  • TEAM_ID
    This is the Developer Portal Team ID. It is of the form 'ABCDEFGHIJ'.

Note: if not running on an iOS pool all iOS-related environment variables are not required.

Sample environment variables for Travis-CI

For example, when Sylph is run on Travis-CI the following environment variables are used:

secret variables

See .travis.yml for running Sylph on Travis-CI.

Note: the Travis-CI build uses pre-configured AWS CLI values in .aws/config.

Upgrade

To upgrade, simply re-issue the install command

$ pub global activate sylph

To check the version of Sylph currently installed:

pub global list

Live demo

To see Sylph in action in a CI environment, a demo of the example app is available.

The log of the live run on mac and linux is here:
https://travis-ci.com/mmcc007/sylph

The resulting artifacts are here:
https://github.com/mmcc007/sylph/releases
(includes a video of test running on device)

To view a similar run on windows:
https://ci.appveyor.com/project/mmcc007/sylph

Contributing

When contributing to this repository, please feel free to discuss via issue or pull request.

Issues and pull requests are welcome.

Your feedback is used to guide where development effort is focused. So feel free to create as many issues and pull requests as you want. You should expect a timely and considered response.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].