Commit b4b389fc authored by Jeremy Mathews's avatar Jeremy Mathews
Browse files

Make most updates except last one

parent 3a4c7a97
Pipeline #8334 failed with stages
in 16 seconds
...@@ -4,16 +4,17 @@ before_script: ...@@ -4,16 +4,17 @@ before_script:
- pip install --user pytest-pythonpath==0.7.1 pytest-cov==2.5.1 - pip install --user pytest-pythonpath==0.7.1 pytest-cov==2.5.1
- module load nextflow/20.01.0 - module load nextflow/20.01.0
- module load singularity/3.5.3 - module load singularity/3.5.3
- mkdir -p test_data/simple1 - mkdir -p test_data/cellranger1bcl
- mkdir -p test_data/simple2 - mkdir -p test_data/cellranger2bcl
- mkdir -p test_data/simple3 - mkdir -p test_data/spaceranger1bcl
- ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/* test_data/simple1/ - ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/cellranger1bcl/* test_data/cellranger1bcl/
- ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple2/* test_data/simple2/ - ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/cellranger2bcl/* test_data/cellranger2bcl/
- ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple3/* test_data/simple3/ - ln -sfn /project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/spaceranger1bcl/* test_data/spaceranger1bcl/
stages: stages:
- astrocyte - astrocyte
- simple - cellranger
- spaceranger
astrocyte_check: astrocyte_check:
stage: astrocyte stage: astrocyte
...@@ -26,11 +27,11 @@ astrocyte_check: ...@@ -26,11 +27,11 @@ astrocyte_check:
when: when:
- always - always
simple_1FC: cellranger1bcl:
stage: simple stage: cellranger
script: script:
- nextflow run workflow/main.nf -profile biohpc,cluster --ranger "cellranger" --bcl "test_data/simple1/*.tar.gz" --designFile "test_data/simple1/cellranger-tiny-bcl-simple-1_2_0.csv" --ci true - nextflow run workflow/main.nf -profile biohpc,cluster --ranger "cellranger" --bcl "test_data/cellranger1bcl/*.tar.gz" --designFile "test_data/cellranger1bcl/cellranger-tiny-bcl-simple-1_2_0.csv" --ci true
##- pytest -m simple1 ##- pytest -m cellranger1bcl
artifacts: artifacts:
name: "$CI_JOB_NAME" name: "$CI_JOB_NAME"
when: always when: always
...@@ -43,11 +44,11 @@ simple_1FC: ...@@ -43,11 +44,11 @@ simple_1FC:
when: when:
- always - always
simple_2FC: cellranger2bcl:
stage: simple stage: cellranger
script: script:
- nextflow run workflow/main.nf -profile biohpc,cluster --ranger "cellranger" --bcl "test_data/simple2/*.tar.gz" --designFile "test_data/simple2/cellranger-tiny-bcl-simple-1_2_0.csv" --ci true - nextflow run workflow/main.nf -profile biohpc,cluster --ranger "cellranger" --bcl "test_data/cellranger2bcl/*.tar.gz" --designFile "test_data/cellranger2bcl/cellranger-tiny-bcl-simple-1_2_0.csv" --ci true
##- pytest -m simple2 ##- pytest -m cellranger2bcl
artifacts: artifacts:
name: "$CI_JOB_NAME" name: "$CI_JOB_NAME"
when: always when: always
...@@ -60,11 +61,11 @@ simple_2FC: ...@@ -60,11 +61,11 @@ simple_2FC:
when: when:
- always - always
simple_3FC: spaceranger1bcl:
stage: simple stage: spaceranger
script: script:
- nextflow run workflow/main.nf -profile biohpc,cluster --ranger "spaceranger" --bcl "test_data/simple3/*.tar.gz" --designFile "test_data/simple3/spaceranger-tiny-bcl-simple-1.0.0.csv" --ci true - nextflow run workflow/main.nf -profile biohpc,cluster --ranger "spaceranger" --bcl "test_data/spaceranger1bcl/*.tar.gz" --designFile "test_data/spaceranger1bcl/spaceranger-tiny-bcl-simple-1.0.0.csv" --ci true
##- pytest -m simple3 ##- pytest -m spaceranger1bcl
artifacts: artifacts:
name: "$CI_JOB_NAME" name: "$CI_JOB_NAME"
when: always when: always
......
...@@ -3,6 +3,7 @@ ...@@ -3,6 +3,7 @@
* Add flag to select cellranger vs spaceranger * Add flag to select cellranger vs spaceranger
* Adjust versions to take into account multiple 10x-Ranger softwares * Adjust versions to take into account multiple 10x-Ranger softwares
* Remove countDesign process * Remove countDesign process
* Spaceranger V1.2.0 update
**Background** **Background**
* Update scripts to reflect spaceranger or cellranger input * Update scripts to reflect spaceranger or cellranger input
......
...@@ -42,19 +42,19 @@ To Run: ...@@ -42,19 +42,19 @@ To Run:
* Available parameters: * Available parameters:
* **-profile** * **-profile**
* what environments to run on, available: `biohpc`, `local`, `cluster`, `aws`, `ondemand`, `spot` * what environments to run on, available: `biohpc`, `local`, `cluster`, `aws`, `ondemand`, `spot`
* eg: ```-profile biohpc,cluster``` to run on BioHPC in cluster mode * eg: `-profile biohpc,cluster` to run on BioHPC in cluster mode
* eg: ```-profile aws,ondemand``` to run on AWS on a on-demand queue * eg: `-profile aws,ondemand` to run on AWS on a on-demand queue
* **--name** * **--name**
* run name, puts outputs in a directory with this name * run name, puts outputs in a directory with this name
* eg: ```--name 'test'``` * eg: `--name 'test'`
* **--ranger** * **--ranger**
* select the 10x ranger being run. * select the 10x ranger being run.
* eg: ```--ranger 'cellranger'``` to run cellranger mkfastq * eg: `--ranger 'cellranger'` to run cellranger mkfastq
* eg: ```--ranger 'spaceranger'``` to run spaceranger mkfastq * eg: `--ranger 'spaceranger'` to run spaceranger mkfastq
* **--bcl** * **--bcl**
* base call files (tarballed [*.tar] +/- gunzipping [*.tar.gz] from a sequencing of 10x single-cell expereiment, supports pigr parallelization) * base call files (tarballed [*.tar] +/- gunzipping [*.tar.gz] from a sequencing of 10x single-cell expereiment, supports pigr parallelization)
* there can be multiple basecall files, but they all will be demultiplexed by the same design file * there can be multiple basecall files, but they all will be demultiplexed by the same design file
* eg: ```--bcl '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-1_2_0.tar.gz'``` * eg: `--bcl '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-1_2_0.tar.gz'`
* **--designFile** * **--designFile**
* path to design file (csv format) location * path to design file (csv format) location
* column 1 = "Lane" (number of lanes to demultiplex, **\*** for all lanes) * column 1 = "Lane" (number of lanes to demultiplex, **\*** for all lanes)
...@@ -64,13 +64,13 @@ To Run: ...@@ -64,13 +64,13 @@ To Run:
* [Current sample barcode IDs](https://s3-us-west-2.amazonaws.com/10x.files/supp/cell-exp/chromium-shared-sample-indexes-plate.csv) * [Current sample barcode IDs](https://s3-us-west-2.amazonaws.com/10x.files/supp/cell-exp/chromium-shared-sample-indexes-plate.csv)
* can have repeated "Sample" if there are multiple fastq R1/R2 pairs for the samples * can have repeated "Sample" if there are multiple fastq R1/R2 pairs for the samples
* can be downloaded [HERE](https://git.biohpc.swmed.edu/BICF/Astrocyte/cellranger_mkfastq/blob/master/docs/design.csv) * can be downloaded [HERE](https://git.biohpc.swmed.edu/BICF/Astrocyte/cellranger_mkfastq/blob/master/docs/design.csv)
* eg: ```--designFile '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple/cellranger-tiny-bcl-simple-1_2_0.csv'``` * eg: `--designFile '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple/cellranger-tiny-bcl-simple-1_2_0.csv'`
* **--mask** * **--mask**
* add a base mask if the sequencing strategy doesn't match the requirements of 10x Genomics * add a base mask if the sequencing strategy doesn't match the requirements of 10x Genomics
* eg: ```--mask '--use-bases-mask=Y\*,I8n\*,n\*,Y\*'``` * eg: `--mask '--ignore-dual-index --use-bases-mask=Y\*,I8n\*,n\*,Y\*'`
* **--outDir** * **--outDir**
* optional output directory for run * optional output directory for run
* eg: ```--outDir 'test'``` * eg: `--outDir 'test'`
* FULL EXAMPLE: * FULL EXAMPLE:
``` ```
nextflow run workflow/main.nf -profile biohpc,cluster --name 'test' --ranger 'cellranger' --bcl '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-1_2_0.tar.gz' --designFile '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-simple-1_2_0.csv' --outDir 'test' nextflow run workflow/main.nf -profile biohpc,cluster --name 'test' --ranger 'cellranger' --bcl '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-1_2_0.tar.gz' --designFile '/project/shared/bicf_workflow_ref/workflow_testdata/cellranger/cellranger_mkfastq/simple1/cellranger-tiny-bcl-simple-1_2_0.csv' --outDir 'test'
......
...@@ -76,6 +76,7 @@ process trackStart { ...@@ -76,6 +76,7 @@ process trackStart {
"sessionId": "${workflow.sessionId}", \ "sessionId": "${workflow.sessionId}", \
"pipeline": "cellranger_mkfastq", \ "pipeline": "cellranger_mkfastq", \
"pipelineVersion": "${pipelineVersion}", \ "pipelineVersion": "${pipelineVersion}", \
"10x-ranger": ${params.ranger}, \
"start": "${workflow.start}", \ "start": "${workflow.start}", \
"astrocyte": ${params.astrocyte}, \ "astrocyte": ${params.astrocyte}, \
"status": "started", \ "status": "started", \
...@@ -213,7 +214,7 @@ process spaceranger_mkfastq { ...@@ -213,7 +214,7 @@ process spaceranger_mkfastq {
tag "${bcl.simpleName}" tag "${bcl.simpleName}"
publishDir "${outDir}/${task.process}", mode: 'copy', pattern: "{*/outs/**/*.fastq.gz}" publishDir "${outDir}/${task.process}", mode: 'copy', pattern: "{*/outs/**/*.fastq.gz}"
queue '128GB,256GB,256GBv1,384GB' queue '128GB,256GB,256GBv1,384GB'
module 'spaceranger/1.1.0:bcl2fastq/2.19.1' module 'spaceranger/1.2.0:bcl2fastq/2.19.1'
input: input:
file versions_spacerangerScript file versions_spacerangerScript
......
...@@ -14,15 +14,15 @@ import os ...@@ -14,15 +14,15 @@ import os
test_output_path = os.path.dirname(os.path.abspath(__file__)) + \ test_output_path = os.path.dirname(os.path.abspath(__file__)) + \
'/../output/' '/../output/'
@pytest.mark.simple1 @pytest.mark.cellranger1bcl
def test_simple1_mkfastq(): def test_cellranger1bcl_mkfastq():
assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0', 'outs')) assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0', 'outs'))
@pytest.mark.simple2 @pytest.mark.cellranger2bcl
def test_simple2_mkfastq(): def test_cellranger2bcl_mkfastq():
assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0-1', 'outs')) assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0-1', 'outs'))
assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0-2', 'outs')) assert os.path.exists(os.path.join(test_output_path, 'mkfastq', 'cellranger-tiny-bcl-1_2_0-2', 'outs'))
@pytest.mark.simple3 @pytest.mark.spaceranger1bcl
def test_simple3_mkfastq(): def test_spaceranger1bcl_mkfastq():
assert os.path.exists(os.path.join(test_output_path, 'spaceranger_mkfastq', 'mkfastq_spaceranger-tiny-bcl-1', 'outs')) assert os.path.exists(os.path.join(test_output_path, 'spaceranger_mkfastq', 'mkfastq_spaceranger-tiny-bcl-1', 'outs'))
...@@ -14,14 +14,14 @@ import os ...@@ -14,14 +14,14 @@ import os
test_output_path = os.path.dirname(os.path.abspath(__file__)) + \ test_output_path = os.path.dirname(os.path.abspath(__file__)) + \
'/../output/multiqc/run/' '/../output/multiqc/run/'
@pytest.mark.simple1 @pytest.mark.cellranger1bcl
def test_simple1_multiqc(): def test_cellranger1bcl_multiqc():
assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html')) assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html'))
@pytest.mark.simple2 @pytest.mark.cellranger2bcl
def test_simple2_multiqc(): def test_cellranger2bcl_multiqc():
assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html')) assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html'))
@pytest.mark.simple3 @pytest.mark.spaceranger1bcl
def test_simple3_multiqc(): def test_spaceranger1bcl_multiqc():
assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html')) assert os.path.exists(os.path.join(test_output_path, 'multiqc_report.html'))
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment