Bioinformaticians and data scientists, rely on computational frameworks (e.g. snakemake, nextflow, CWL, WDL) to process, analyze and integrate data of various types. Such frameworks allow scientists to combine software and custom tools of different origin in a unified way, which lets them reproduce the results of others, or reuse the same pipeline on different datasets. One of the fundamental issues is that the majority of the users execute multiple pipelines at the same time, or execute a multistep pipeline for a big number of datasets, or both, making it hard to track the execution of the individual steps or monitor which of the processed datasets are complete. panoptes is a tool that monitors the execution of such workflows.
panoptes is a service that can be used by:
- Data scientists, bioinformaticians, etc. that want to have a general overview of the progress of their pipelines and the status of their jobs
- Administrations that want to monitor their servers
- Web developers that want to integrate the service in bigger web applications
Note: panoptes is in early development stage and the first proof of concept server will support only workflows written in snakemake.
Installation
Basic installation process
Requirements
- Python>=3.6
- virtualenv
- sqlite3
Option 1: Install via pypi and run server
Create virtual environment
virtualenv -p `which python3` venvActivate virtual environment
source venv/bin/activateInstall via pypi
pip install panoptes-uiRun server
panoptesServer should run on: 127.0.0.1:5000
By default it should generate an sqlite database: .panoptes.db
Option 2: Install via conda and run server
Create conda environment
conda create --name panoptesActivate conda environment
conda activate panoptesInstall via pypi OR conda
conda install -c panoptes-organization panoptes-uiRun server
panoptesServer should run on: 127.0.0.1:5000
By default it should generate an sqlite database: .panoptes.db
Option 3: Install from source code and run server
Clone repo
git clone https://github.com/panoptes-organization/panoptes.gitEnter repo
cd panoptesCreate virtual environment
virtualenv -p `which python3` venvActivate virtual environment
source venv/bin/activateInstall all requirements
pip install .Run server
panoptesServer should run on: 127.0.0.1:5000
By default it should generate an sqlite database: .panoptes.db
Docker installation
Requirements
- docker
- docker-compose
Build and run with docker-compose
Build
docker-compose buildRun
docker-compose up -dServer should run on: http://127.0.0.1:8000
Stop
docker-compose downRun an example workflow
In order to run an example workflow please follow the instructions here
panoptes in action
panoptes API
Panoptes provides the following API endpoints:
| Endpoint | Method | Description |
|---|---|---|
/api/service-info |
GET |
Server status |
/api/workflows |
GET |
Get all workflows |
/api/workflow/<workflow-id> |
GET |
Get workflow status |
/api/workflow/<workflow-id>/jobs |
GET |
Get all jobs of a workflow |
/api/workflow/<workflow-id>/job/<job-id> |
GET |
Get job status |
/api/workflow/<workflow-id> |
PUT |
Rename a workflow Expects a dictionary with new name (e.g. {'name': 'my new workflow name'}) |
/api/workflow/<workflow-id> |
DELETE |
Delete a workflow |
/api/workflows/all |
DELETE |
Clean up database |
To communicate with panoptes the following endpoints are used by snakemake:
| Endpoint | Method | Description |
|---|---|---|
/api/service-info |
GET |
Server status (same as above) |
/create_workflow |
GET |
Get a unique id/name str(uuid.uuid4()) for each workflow |
/update_workflow_status |
POST |
Panoptes receives a dictionary from snakemake that contains: - A log message dictionary - The current timestamp - The unique id/name of the workflow. (e.g. {'msg': repr(msg), 'timestamp': time.asctime(), 'id': id}) |
Contribute
Please see the Contributing instructions.
CI server
Changes in develop or master trigger a Travis build (and runs tests)
Contact
In case the issues section is not enough for you, you can also contact us via discord