cwl-WES

Synopsis

Microservice implementing the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification for the execution of workflows written in the Common Workflow Language (CWL).

cwl-WES is a core service of the ELIXIR Cloud & AAI project.

Description

cwl-WES (formerly: WES-ELIXIR) is a Flask/Gunicorn application that makes use of Connexion to implement the GA4GH WES OpenAPI specification. It enables clients/users to execute CWL workflows in the cloud via a GA4GH Task Execution Service (TES)-compatible execution backend (e.g., TESK or Funnel). Workflows can be sent for execution, previous runs can be listed, and the status and run information of individual runs can be queried. The service leverages cwl-tes to interpret CWL workflows, break them down into individual tasks and emit GA4GH TES-compatible HTTP requests to a configured TES instance. Access to endpoints can be configured to require JSON Web Token-based access tokens, such as those issued by ELIXIR AAI. Run information is stored in a MongoDB database.

Note that development is currently in beta stage. Check the website badge at the top of this document for a publicly available test deployment. Further test deployments can be found at the ELIXIR Cloud & AAI's resource listings.

cwl-WES is developed and maintained by the ELIXIR Cloud & AAI project, a multinational effort aimed at establishing and implementing FAIR research in the Life Sciences.

Installation

Kubernetes

See separate instructions available here.

docker-compose

Requirements

Ensure you have the following software installed:

• Docker (18.06.1-ce, build e68fc7a)
• docker-compose (1.22.0, build f46880fe)
• Git (1.8.3.1)

These are the versions used for development/testing. Other versions may or may not work. Please let us know if you encounter any issues with newer versions than the listed ones.

Instructions

Set up environment

Create data directory and required subdiretories:

mkdir -p data/cwl_wes/db data/cwl_wes/output data/cwl_wes/tmp

Clone repository:

git clone https://github.com/elixir-cloud-aai/cwl-WES.git

Traverse to app directory:

cd app

Optional: Edit/override app config

• Via the app configuration file

  vi cwl_wes/config/app_config.yaml

• Via environment variables

  A few configuration settings can be overridden by environment variables:

  export <ENV_VAR_NAME>=<VALUE>

  List of the available environment variables:

  Variable	Description
  MONGO_HOST	MongoDB host endpoint
  MONGO_PORT	MongoDB service port
  MONGO_DBNAME	MongoDB database name
  MONGO_USERNAME	MongoDB client username
  MONGO_PASSWORD	MongoDB client password
  RABBIT_HOST	RabbitMQ host endpoint
  RABBIT_PORT	RabbitMQ service port

Build & deploy

Build and run services in detached/daemonized mode:

docker-compose up -d --build

Copy FTP credentials

Create a .netrc file with credentials for accessing an FTP server:

cat << EOF > .netrc
machine <HOST>
login <USERNAME>
password <PASSWORD>
EOF

Don't forget to replace <HOST>, <USERNAME> and <PASSWORD> with real values.

If you do not know what to put here, creating an empty file .netrc with, e.g., touch .netrc will be fine for testing purposes.

Copy the file into the running worker container(s):

for cont in $(docker ps --all | grep wes-worker | cut -f1 -d" "); do
    docker cp .netrc "${cont}:/tmp/user"
done

Use service

Visit Swagger UI:

firefox http://localhost:8080/ga4gh/wes/v1/ui

Example values to start a simple CWL test workflow via the POST /runs endpoint:

workflow_params: {"input":{"class":"File","path":"ftp://ftp-private.ebi.ac.uk/upload/foivos/test.txt"}}`
workflow_type: CWL
workflow_type_version`: v1.0
workflow_url: https://github.com/uniqueg/cwl-example-workflows/blob/master/hashsplitter-workflow.cwl

Leave the rest of the values empty and hit the Try it out! button.

You can also use the service through curl. For example, to send a request to the GET /runs endpoint:

curl -X GET \
    --header 'Accept: application/json' \
    'http://localhost:8080/ga4gh/wes/v1/runs' 

Authorization

All endpoints except for GET /service-info can be configured to require the presence of a valid JWT Bearer token in a request's header. This can be enabled by setting the following paraneter in the app configuration to True:

# Security settings
security:
    authorization_required: True

To send authorized requests, you must be in possession of a valid JWT Bearer token. The app was developed against JWTs issues by ELIXIR AAI, although JWTs issued by other identity providers may work.

If you want to utilize actual compute resources in any of the ELIXIR test deployments, you must be a member of a specific ELIXIR user group. You can apply here to be added to that group (ELIXIR AAI only!). When applying, please include a brief description of why you would like to make use of our test deployments and please note that they are for fair use only (please don't try to run your production workflows on them or we will need to re-think our strategy of offering free resources for testing/developing).

To access protected endpoints via curl, you only need to add an Authorization header to your request, followed by the Bearer prefix and your JWT token, like so:

curl -X GET \
    --header 'Accept: application/json' \
    --header 'Authorization: Bearer <YOUR_TOKEN>' \
    'http://localhost:8080/ga4gh/wes/v1/runs' 

If you are making use of the Swagger UI, you will need to click on the Authorize button (which appears on the right-hand side of the top bar upon enabling the authorization_required option) of the Swagger UI website and enter your JWT token in the api_key field, preceded by the Bearer prefix:

Contributing

This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please refer to our organization's contributing guidelines if you are interested to contribute. Please mind the code of conduct for all interactions with the community.

Versioning

Development of the app is currently still in beta stage, and current versions are for internal use only. We are aiming to have a fully spec-compliant version of the app available soon. The plan is to then adopt a semantic versioning scheme in which we would shadow WES spec versioning, with added date stamp patches for any patch-level changes to our service.

License

This project is covered by the Apache License 2.0 also shipped with this repository.

Contact

The project is a collaborative effort under the umbrella of ELIXIR Cloud & AAI. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.