ngs-preprocess pipeline
A pipeline for preprocessing short and long sequencing reads
See the documentation »
Report Bug · Request Feature
About
ngs-preprocess is built using Nextflow, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. It is an easy to use pipeline that uses state-of-the-art software for quality check and pre-processing ngs reads of Illumina, Pacbio and Oxford Nanopore Technologies.
It wraps up the following software:
| Step | tools |
|---|---|
| SRA NBCI fetch | Entrez-direct & sra-tools |
| Illumina pre-processing | Fastp |
| Nanopore pre-processing | Porechop, pycoQC, NanoPack |
| Pacbio pre-processing | bam2fastx, bax2bam, lima, pacbio ccs |
Further reading
This pipeline has two complementary pipelines (also written in nextflow) for genome assembly and prokaryotic genome annotation that can give the user a complete workflow for bacterial genomics analyses.
Quickstart
-
Install Nextflow:
curl -s https://get.nextflow.io | bash -
Give it a try:
nextflow run fmalmeida/ngs-preprocess --help
-
Download required tools
-
for docker
# for docker docker pull fmalmeida/ngs-preprocess:v2.5 # run nextflow run fmalmeida/ngs-preprocess -profile docker [options]
-
for singularity
# for singularity # remember to properly set NXF_SINGULARITY_LIBRARYDIR # read more at https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub export NXF_SINGULARITY_LIBRARYDIR=MY_SINGULARITY_IMAGES # your singularity storage dir export NXF_SINGULARITY_CACHEDIR=MY_SINGULARITY_CACHE # your singularity cache dir singularity pull \ --dir $NXF_SINGULARITY_LIBRARYDIR \ fmalmeida-ngs-preprocess-v2.5.img docker://fmalmeida/ngs-preprocess:v2.5 # run nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
-
for conda
# for conda # it is better to create envs with mamba for faster solving wget https://github.com/fmalmeida/ngs-preprocess/raw/master/environment.yml conda env create -f environment.yml # advice: use mamba # must be executed from the base environment # This tells nextflow to load the available ngs-preprocess environment when required nextflow run fmalmeida/ngs-preprocess -profile conda [options]
-
-
Start running your analysis
nextflow run fmalmeida/ngs-preprocess -profile <docker/singularity/conda>
Documentation
Selecting between profiles
Nextflow profiles are a set of "sensible defaults" for the resource requirements of each of the steps in the workflow, that can be enabled with the command line flag -profile. You can learn more about nextflow profiles at:
- https://nf-co.re/usage/configuration#basic-configuration-profiles
- https://www.nextflow.io/docs/latest/config.html#config-profiles
The pipeline have "standard profiles" set to run the workflows with either conda, docker or singularity using the local executor, which is nextflow's default and basically runs the pipeline processes in the computer where Nextflow is launched. If you need to run the pipeline using another executor such as sge, lsf, slurm, etc. you can take a look at nextflow's manual page to proper configure one in a new custom profile set in your personal copy of ngs-preprocess config file and take advantage that nextflow allows multiple profiles to be used at once, e.g. -profile conda,sge.
By default, if no profile is chosen, the pipeline will try to load tools from the local machine $PATH. Available pre-set profiles for this pipeline are: docker/conda/singularity, you can choose between them as follows:
-
conda
# must be executed from the base environment # This tells nextflow to load the available ngs-preprocess environment when required nextflow run fmalmeida/ngs-preprocess -profile conda [options]
-
docker
nextflow run fmalmeida/ngs-preprocess -profile docker [options]
-
singularity
nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
Usage
For understading pipeline usage and configuration, users must read the complete online documentation »
Using a configuration file
All the parameters showed above can be, and are advised to be, set through the configuration file. When a configuration file is set the pipeline is run by simply executing:
nextflow run fmalmeida/ngs-preprocess -c ./configuration-fileYour configuration file is what will tell to the pipeline the type of data you have, and which processes to execute. Therefore, it needs to be correctly set up.
Create a configuration file in your working directory:
nextflow run fmalmeida/ngs-preprocess [ --get_config ]Interactive graphical configuration and execution
Via NF tower launchpad (good for cloud env execution)
Nextflow has an awesome feature called NF tower. It allows that users quickly customise and set-up the execution and configuration of cloud enviroments to execute any nextflow pipeline from nf-core, github (this one included), bitbucket, etc. By having a compliant JSON schema for pipeline configuration it means that the configuration of parameters in NF tower will be easier because the system will render an input form.
Checkout more about this feature at: https://seqera.io/blog/orgs-and-launchpad/
Via nf-core launch (good for local execution)
Users can trigger a graphical and interactive pipeline configuration and execution by using nf-core launch utility. nf-core launch will start an interactive form in your web browser or command line so you can configure the pipeline step by step and start the execution of the pipeline in the end.
# Install nf-core
pip install nf-core
# Launch the pipeline
nf-core launch fmalmeida/ngs-preprocessIt will result in the following:
Citation
To cite this tool please refer to our Zenodo tag.
This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the GPLv3.
The nf-core framework for community-curated bioinformatics pipelines.
Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.
Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.
In addition, users are encouraged to cite the programs used in this pipeline whenever they are used. Links to resources of tools and data used in this pipeline are as follows: