master	develop

RNA-Seq Analytic Pipeline for GUDMAP/RBK

Introduction

This pipeline was created to be a standard mRNA-sequencing analysis pipeline which integrates with the GUDMAP and RBK consortium data-hub. It is designed to run on the HPC cluster (BioHPC) at UT Southwestern Medical Center (in conjunction with the standard nextflow profile: config biohpc.config)

Authentication:

The consortium server used must be authentificated with the deriva authentication client, and remain authentificated till the end of the pipeline run. Prematurely closing the client will result in invalidation of the tokens, and may result in the pipeline failure. The use of long-lived "globus" tokens is on the roadmap for use in the future.

To Run:

• Available parameters:

  • --deriva active credential.json file from deriva-auth
  • --bdbag active cookies.txt file from deriva-auth
  • --repRID mRNA-seq replicate RID
  • --source consortium server source
    • dev = dev.gudmap.org (default, does not contain all data)
    • staging = staging.gudmap.org (does not contain all data)
    • production = www.gudmap.org (does contain all data)
  • --refMoVersion mouse reference version (optional, default = 38.p6.vM25)
  • --refHuVersion human reference version (optional, default = 38.p13.v36)
  • --refERCCVersion human reference version (optional, default = 92)
  • --upload option to not upload output back to the data-hub (optional, default = false)
    • true = upload outputs to the data-hub
    • false = do NOT upload outputs to the data-hub
  • -profile config profile to use (optional):
    • defaut = processes on BioHPC cluster
    • biohpc = process on BioHPC cluster
    • biohpc_max = process on high power BioHPC cluster nodes (=> 128GB nodes), for resource testing
    • aws_ondemand = AWS Batch on-demand instant requests
    • aws_spot = AWS Batch spot instance requests
  • --email email address(es) to send failure notification (comma separated) (optional):
    • e.g: --email 'Venkat.Malladi@utsouthwestern.edu,Gervaise.Henry@UTSouthwestern.edu'

• NOTES:

  • once deriva-auth is run and authenticated, the two files above are saved in ~/.deriva/ (see official documents from deriva on the lifetime of the credentials)
  • reference version consists of Genome Reference Consortium version, patch release and GENCODE annotation release # (leaving the params blank will use the default version tied to the pipeline version)
    • current mouse 38.p6.vM25 = GRCm38.p6 with GENCODE annotation release M25
    • current human 38.p13.v36 = GRCh38.p13 with GENCODE annotation release 36

• Optional input overrides

  • --refSource source for pulling references
    • biohpc = source references from BICF_Core gudmap reference local location (workflow must be run on BioHPC system)
    • datahub = source references from GUDMAP/RBK reference_table location (currently uses dev.gudmap.org)
  • --inputBagForce utilizes a local replicate inputBag instead of downloading from the data-hub (still requires accurate repRID input)
    • eg: --inputBagForce test_data/bag/Q-Y5F6_inputBag_xxxxxxxx.zip (must be the expected bag structure, this example will not work because it is a test bag)
  • --fastqsForce utilizes local fastq's instead of downloading from the data-hub (still requires accurate repRID input)
    • eg: --fastqsForce 'test_data/fastq/small/Q-Y5F6_1M.R{1,2}.fastq.gz' (note the quotes around fastq's which must me named in the correct standard [*.R1.fastq.gz and/or *.R2.fastq.gz] and in the correct order, also consider using endsForce if the endness doesn't match submitted value)
  • --speciesForce forces the species to be "Mus musculus" or "Homo sapiens", it bypasses a metadata mismatch or an ambiguous species error
    • eg: --speciesForce 'Mus musculus'
  • --endsForce forces the endness to be "se", or "pe", it bypasses a metadata mismatch error
    • eg: --endsForce 'pe'
  • --strandedForce forces the strandedness to be "forward", "reverse" or "unstranded", it bypasses a metadata mismatch error
    • eg: --strandedForce 'unstranded'
  • --spikeForce forces the spike-in to be "false", or "true", it bypasses a metadata mismatch error
    • eg: --spikeForce 'true'

• Tracking parameters (Tracking Site):

  • --ci boolean (default = false)
  • --dev boolean (default = true)

FULL EXAMPLE:

nextflow run workflow/rna-seq.nf --repRID Q-Y5JA --source production --deriva ./data/credential.json --bdbag ./data/cookies.txt --dev false --upload true -profile biohpc

Cloud Compatibility:

This pipeline is also capable of being run on AWS and DNAnexus. To do so:

AWS

• Build a AWS batch queue and environment either manually or with aws-cloudformantion
• Edit one of the aws configs in workflow/config/
  • Replace workDir with the S3 bucket generated
  • Change region if different
  • Change queue to the aws batch queue generated
• The user must have awscli configured with an appropriate authentication (with aws configure and access keys) in the environment which nextflow will be run
• Add -profile with the name aws config which was customized

DNAnexus (utilizes the DNAnexus extension package for Nextflow (XPACK-DNANEXUS))

• Follow the istructions from XPACK-DNANEXUS about installing and authenticating (a valid license must be available for the extension package from Seqera Labs, as well as a subsription with DNAnexus)
• The nf-dxapp needs to be built with a custom scm config to allow nextflow to pull the pipeline from the UTSW self-hosted GitLab server (git.biohpc.swmed.edu)

providers {
    bicf {
        server = 'https://git.biohpc.swmed.edu'
        platform = 'gitlab'
    }
}

• Follow the instructions from XPACK-DNANEXUS about launching runs. A template json file has been included (dnanexusExample.json)
  • [version] should be replaced with the pipeline version required (eg: v2.0.0)
  • [credential.json] should be replaced with the location of the credential file outpted by authentification with Deriva
  • [cookies.txt] should be replaced with the location of the cookies file outpted by authentification with Deriva for BDBag
  • [repRID] should be replaced with the replicate RID to be analized (eg: Q-Y5F6)
  • [outDir] should be replaced with the location to save local outputs of the pipeline

To generate you own references or new references:

Download the reference creation script. This script will auto create human and mouse references from GENCODE. It can also create ERCC92 spike-in references as well as concatenate them to GENCODE references automatically. In addition, it can create references from manually downloaded FASTA and GTF files.

Errors:

Error reported back to the data-hub are (they aren't thrown on the command line by the pipeline, but rather are submitted (if --upload true) to the data-hub for that replicate in the execution run submission):

Credits

This workflow is was developed by Bioinformatic Core Facility (BICF), Department of Bioinformatics

PI

Venkat S. Malladi
Faculty Associate & Director
Bioinformatics Core Facility
UT Southwestern Medical Center
orcid.org/0000-0002-0144-0564
venkat.malladi@utsouthwestern.edu

Developers

Gervaise H. Henry
Computational Biologist
Department of Urology
UT Southwestern Medical Center
orcid.org/0000-0001-7772-9578
gervaise.henry@utsouthwestern.edu

Jonathan Gesell
Computational Biologist
Bioinformatics Core Facility
UT Southwestern Medical Center
orcid.org/0000-0001-5902-3299
johnathan.gesell@utsouthwestern.edu

Jeremy A. Mathews
Computational Biologist
Bioinformatics Core Facility
UT Southwestern Medical Center
orcid.org/0000-0002-2931-1430
jeremy.mathews@utsouthwestern.edu

Please cite in publications: Pipeline was developed by BICF from funding provided by Cancer Prevention and Research Institute of Texas (RP150596).

Pipeline Directed Acyclic Graph