GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → dotnet → xharness

dotnet / xharness

Licence: MIT license
C# command line tool for running tests on Android / iOS / tvOS devices and simulators

Programming Languages

C#
18002 projects
powershell
5483 projects
shell
77523 projects

Labels

androidiosdotnettvosxunitmonotestsautomated-testingmaccatalyst

Projects that are alternatives of or similar to xharness

Mvvmcross
The .NET MVVM framework for cross-platform solutions, including Xamarin.iOS, Xamarin.Android, Windows and Mac.
Stars: ✭ 3,594 (+2821.95%)
Mutual labels:  tvos, mono
testable
QT/QML Test Runner and Utilities
Stars: ✭ 53 (-56.91%)
Mutual labels:  tests
bUnit
bUnit is a testing library for Blazor components that make tests look, feel, and runs like regular unit tests. bUnit makes it easy to render and control a component under test’s life-cycle, pass parameter and inject services into it, trigger event handlers, and verify the rendered markup from the component using a built-in semantic HTML comparer.
Stars: ✭ 857 (+596.75%)
Mutual labels:  xunit
hasura-node-monolith-example
Example of a monolithic web application using Hasura GraphQL Engine + Node.js + Next.js
Stars: ✭ 25 (-79.67%)
Mutual labels:  mono
perseus
Perseus is a set of scripts (docker+javascript) to investigate a distributed database's responsiveness when one of its three nodes is isolated from the peers
Stars: ✭ 49 (-60.16%)
Mutual labels:  tests
jdi-dark
Powerful Framework for Backend Automation Testing on Java (Rest, Soap, WebSocket)
Stars: ✭ 36 (-70.73%)
Mutual labels:  automated-testing
go-recipes
🦩 Tools for Go projects
Stars: ✭ 2,490 (+1924.39%)
Mutual labels:  tests
fingerprintjs-pro-ios
Official iOS/tvOS agent & SDK for accurate device identification, created for the Fingerprint Pro identification API.
Stars: ✭ 35 (-71.54%)
Mutual labels:  tvos
TvOSTextViewer
Light and scrollable view controller for tvOS to present blocks of text
Stars: ✭ 45 (-63.41%)
Mutual labels:  tvos
Juicer
Juicer is a generic animation / motion library for macOS & iOS & tvOS written in Swift
Stars: ✭ 13 (-89.43%)
Mutual labels:  tvos
openjpeg-data
Test files for the OpenJPEG libraries and utilities
Stars: ✭ 37 (-69.92%)
Mutual labels:  tests
XunitContext
Extends xUnit to expose extra context and simplify logging
Stars: ✭ 127 (+3.25%)
Mutual labels:  xunit
EntityFrameworkCore.AutoFixture
A library aimed to minimize the boilerplate required to unit-test Entity Framework Core code using AutoFixture and in-memory providers.
Stars: ✭ 31 (-74.8%)
Mutual labels:  tests
Futures
Lightweight promises for iOS, macOS, tvOS, watchOS, and Linux
Stars: ✭ 59 (-52.03%)
Mutual labels:  tvos
Dot-Net-Ecosystem
Welcome to the GitHub repository of the .NET Ecosystem. This repository contains the examples for the Pluralsight course: The .NET Ecosystem: The Big Picture. You can download a copy and follow along in the course.
Stars: ✭ 34 (-72.36%)
Mutual labels:  mono
TVGemist
An *Unofficial* Uitzending Gemist application for  TV
Stars: ✭ 23 (-81.3%)
Mutual labels:  tvos
KAI
KAI is a distributed computing model written in modern C++ and is cross-plaftorm. Using custom language translators and an executor, KAI provides full reflection, persistence and cross-process communications without having to modify existing source code. KAI Comes with an automated, generational tricolor garbage collector, and Console- and Windo…
Stars: ✭ 13 (-89.43%)
Mutual labels:  tests
ScaledFont
ScaledFont - Using custom fonts with dynamic type
Stars: ✭ 50 (-59.35%)
Mutual labels:  tvos
Kattis-Demos-Testing
Kattis demo solutions with unit testing
Stars: ✭ 14 (-88.62%)
Mutual labels:  automated-testing
MinimalApi
ASP.NET Core 7.0 - Minimal API Example - Todo API implementation using ASP.NET Core Minimal API, Entity Framework Core, Token authentication, Versioning, Unit Testing, Integration Testing and Open API.
Stars: ✭ 156 (+26.83%)
Mutual labels:  xunit
View All Similar Projects ➔

XHarness

This repo contains the code to build the XHarness dotnet tool and a TestRunner library that makes running unit tests in mobile platforms easier.

What is XHarness

XHarness is primarily a command line tool that enables running xUnit like tests on Android, Apple iOS / tvOS / WatchOS / Mac Catalyst, WASI and desktop browsers (WASM). It can

  • locate devices/emulators
  • install a given application, run it and collect results uninstalling it after,
  • perform the operations above as part of one command or separately if need be,
  • handle application crashes by collecting crash dumps (symbolicate),
  • use different types of connection modes (network, USB cable),
  • output test results in various different formats from text to xUnit/NUnit XML
  • install Apple Simulator runtimes (different versions of iOS, tvOS...).

System requirements

The tool requires .NET 6 or later to be run. It is packaged as a dotnet tool command and can be installed using the dotnet tool CLI.

  • The Apple scenarios require you to run the tool on MacOS with full Xcode installation
  • Android scenarios are supported on Linux, macOS and Windows systems
  • Browsers scenarios are supported on Linux systems

Try the tool out quickly

If you want to test the tool quickly, following script will install the required .NET SDK and the XHarness tool locally in the current folder.

# Using bash on Linux/MacOS
curl -L https://aka.ms/get-xharness | bash -
# Using PowerShell on Windows
iex ((New-Object System.Net.WebClient).DownloadString('https://aka.ms/get-xharness-ps1'))

You can delete the folder after you're done, nothing is installed in your system.

Installation and usage

To install the latest version of the tool run (in bash):

dotnet tool install Microsoft.DotNet.XHarness.CLI                                                   \
    --global                                                                                        \
    --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json \
    --version "1.0.0-prerelease*"

Or run (in PowerShell):

dotnet tool install Microsoft.DotNet.XHarness.CLI                                                   `
    --global                                                                                        `
    --add-source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json `
    --version "1.0.0-prerelease*"

You can get a specific version from the dotnet-eng feed where it is published. So far, we are in preview so omitting the version will fail to locate a stable version of the tool and fail the installation so a specific version has to be supplied.

To run the tool, use the xharness command. The tool returns one of the exit codes listed here (ExitCode.cs). The tool always expects the platform (android/apple/browser) followed by a command. To get an up-to-date set of commands, please run xharness help.

Applications run via the apple test command require a TestRunner inside of the iOS/tvOS app bundle to work properly. The apple run command, on the other hand, doesn't expect the TestRunner and only runs the application and tries to detect the exit code. Detection of exit code might not work across different iOS versions reliably.

* See the Test Runners section.

Example:

xharness android state

To list all the possible commands, use the help command:

xharness help

To get help for a specific command or sub-command, run:

xharness help apple
xharness help apple test

Other settings

There are other settings which can be controlled via environmental variables and are primarily meant for build pipeline scenarios:

  • XHARNESS_DISABLE_COLORED_OUTPUT - disable colored logging so that control characters are not making the logs hard to read
  • XHARNESS_LOG_WITH_TIMESTAMPS - enable timestamps for logging
  • XHARNESS_LOG_TEST_START - log test start messages, useful to diagnose when tests are hanging. Currently only works for WebAssembly
  • XHARNESS_MLAUNCH_PATH - local path to the mlaunch binary when developing XHarness (when not using as .NET tool)

Arcade/Helix integration

In case your repository is onboarded into Arcade you can use the Arcade Helix SDK to run XHarness jobs over Helix. More on how to do that is described here.

Examples

To run an iOS/tvOS app bundle on a 64bit iPhone Simulator:

xharness apple test           \
    --app=/path/to/an.app     \
    --output-directory=out    \
    --target=ios-simulator-64

or the same can be achieved via the shorthand versions of the same options:

xharness apple test -a=/path/to/an.app -o=out -t=ios-simulator-64

The out dir will then contain log files such as these:

iPhone X (iOS 13.3) - created by xharness.log   # logs from the Simulator
test-Simulator_iOS64.log                        # logs from the tool itself
test-ios-simulator-64-20200430_025916.xml       # test results in XML format

Example for Android apk:

xharness android test                                    \
    --output-directory=out                               \
    --package-name=net.dot.System.Numerics.Vectors.Tests \
    --app=/path/to/test.apk

Output directory will have a file with dump from logcat and a file with tests results.

Test Runners

The repository also contains several TestRunners which are libraries that can be bundled inside of the application and execute the tests. The TestRunner detects and executes unit tests inside of the application. It also connects to XHarness over TCP connection from within the running app bundle and reports test run results/state.

There is a library Microsoft.DotNet.XHarness.DefaultAndroidEntryPoint.Xunit that provides default logic for Android test app entry point. It is possible to use DefaultAndroidEntryPoint from there for the test app by providing only test result path and test assemblies. Other parameters can be overrided as well if needed.

Currently we support Xunit and NUnit test assemblies but the Microsoft.DotNet.XHarness.Tests.Runners supports implementation of custom runner too.

Development instructions

When working on XHarness, there are couple of neat hacks that can improve the inner loop. The repository can either be built using regular .NET, assuming you have new enough version:

dotnet build XHarness.sln

or you can use the build scripts build.sh or Build.cmd in repository root which will install the correct .NET SDK into the .dotnet folder. You can then use

./.dotnet/dotnet build XHarness.sln

You can also use Visual Studio 2019+ and just F5 the Microsoft.DotNet.XHarness.CLI project.

ADB, mlaunch

In order for XHarness to work, you will need ADB (for Android) and mlaunch (for anything Apple). These are executables that go with the packaged .NET xharness tool.

The easiest way to get these at the moment for development purposes is to build the CLI project and they will be downloaded.

dotnet build src/Microsoft.DotNet.XHarness.CLI/Microsoft.DotNet.XHarness.CLI.csproj

You can then find these dependencies in artifacts/obj/Microsoft.DotNet.XHarness.CLI/.

For iOS flows, you can further store the path to mlaunch to an environmental variable XHARNESS_MLAUNCH_PATH

export XHARNESS_MLAUNCH_PATH='[xharness root]/artifacts/obj/Microsoft.DotNet.XHarness.CLI/mlaunch/bin/mlaunch'

and you won't have to specify the --mlaunch argument.

Running E2E tests

In case you want to test your changes in XHarness, you can run E2E tests located in /tests/integration-tests. These usually download some pre-built application and send it to our "test cloud" called Helix together with an XHarness version built from your sources. There, XHarness executes the app on a device/simulator.

To run the E2E tests, you can find a script in tools/ that will build everything and create the cloud job for you:

./tools/run-e2e-test.sh Apple/Simulator.Tests.proj

Troubleshooting

Some XHarness commands only work in some scenarios and it's good to know what to expect from the tool. Some Android/Apple versions also require some workarounds and those are also good to know about.

My Apple unit tests are not running

For the apple test command, XHarness expects the application to contain a TestRunner which is a library you can find in this repository. This library executes unit tests similarly how you would execute them on other platforms. However, the TestRunner from this repository contains more mechanisms that help to work around some issues (mostly in Apple platforms).

The way it works is that XHarness usually sets some environmental variables for the application and the TestRunner recognizes them and acts upon them.

The workarounds we talk about are for example some TCP connections between the app and XHarness so that we can stream back the test results.

For these reasons, the test command won't just work with any app. For those scenarios, use the apple run commands.

iOS/tvOS device runs are timing out

For some iOS/tvOS, we have problems detecting when the application exits on the real device (simulators work fine). The workaround we went with lies in sharing a random string with the application using an environmental variable RUN_END_TAG and expecting the app to output this string at the end of its run.

To turn this workaround on, run XHarness with --signal-app-end and make sure your application logs the string it reads from the env variable. Using the TestRunner from this repository will automatically give you this functionality.

Contribution

We welcome contributions! Please follow the Code of Conduct.

Filing issues

This repo should contain issues that are tied to the XHarness command line tool and the TestRunners.

For other issues, please use the following repos:

  • For .NET runtime and Base Class Library issues, file in the dotnet/runtime repo
  • For overall .NET SDK issues, file in the dotnet/sdk repo

License

.NET (including the xharness repo) is licensed under the MIT license.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].