Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
A
astrocyte_example_wordcount_ignite
Manage
Activity
Members
Labels
Plan
Issues
0
Issue boards
Milestones
Iterations
Code
Merge requests
0
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Container Registry
Monitor
Service Desk
Analyze
Contributor analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Astrocyte
Workflows
BioHPC
astrocyte_example_wordcount_ignite
Commits
21c2b069
Commit
21c2b069
authored
6 years ago
by
David Trudgian
Browse files
Options
Downloads
Patches
Plain Diff
Tidy up example
parent
010a236f
Branches
Branches containing commit
Tags
Tags containing commit
No related merge requests found
Pipeline
#3815
failed with stages
Changes
4
Pipelines
2
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
.gitlab-ci.yml
+1
-2
1 addition, 2 deletions
.gitlab-ci.yml
LICENSE.md
+1
-1
1 addition, 1 deletion
LICENSE.md
README.md
+38
-8
38 additions, 8 deletions
README.md
astrocyte_pkg.yml
+1
-1
1 addition, 1 deletion
astrocyte_pkg.yml
with
41 additions
and
12 deletions
.gitlab-ci.yml
+
1
−
2
View file @
21c2b069
# Run our tests against a specific version of astrocyte
before_script
:
-
module load astrocyte/0.1.0
-
module load astrocyte
stages
:
-
check
...
...
This diff is collapsed.
Click to expand it.
LICENSE.md
+
1
−
1
View file @
21c2b069
Copyright © 201
6
. The University of Texas Southwestern Medical Center
Copyright © 201
9
. The University of Texas Southwestern Medical Center
5323 Harry Hines Boulevard Dallas, Texas, 75390 Telephone 214-648-3111
This diff is collapsed.
Click to expand it.
README.md
+
38
−
8
View file @
21c2b069
# Example Wordcount Package
# Example
Ignite
Wordcount Package
[

](https://git.biohpc.swmed.edu/BioHPC/astrocyte_example_wordcount/commits/master)
[

](https://astrocyte-test.biohpc.swmed.edu/static/docs/index.html)
[

](https://astrocyte-test.biohpc.swmed.edu/static/docs/index.html)
This is an example workflow package for astrocyte. It contains a worfklow
This is a minimal test workflow package that counts the occurences of words
in a test file. It can be used as a template to develop workflows, and as to
test the astrocyte platform. This version uses the Ignite executor for
Nextflow, so that processes all execute inside a single batch job allocation.
to count the occurences of each word in a text file.
## The Workflow
The workflow
`workflow/main.nf`
has three processes:
-
Convert all the text in the input files to uppercase
-
Split the text so each word is on a separate line
-
Sort, find and count the occurence of unique words
-
`parameters`
outputs the parameters provided to the workflow, as an example
of how workflows receive the different types of parameters that can be setup
in the
`astrocyte_pkg.yml`
file.
-
`uppercase`
converts all the text in the input files to uppercase
-
`tolines`
splits the text so each word is on a separate line
-
`wordcount`
sorts, find and count the occurence of unique words
## Parameters
There is a single parameter
`story`
. This provides 1 or more files that the
workflow should run on.
There is a single parameter which affects the workflow -
`story`
. This provides
1 or more files that the workflow should run on.
Additional parameters named
`test_xxx`
are included only to show how to define
the parameter type in the
`astrocyte_pkg.yml`
file. They are simply printed to
STDOUT by the workflow's
`parameters`
process.
## Ignite Execution
The workflow can be run using Nextflow + Apache Ignite using the
`astrocyte_cli`
tool. On the BioHPC cluster:
```
sh
module add astrocyte
astrocyte ignite-cluster astrocyte_example_wordcount_ignite
```
An example Slurm sbatch script
`workflow/ignite_batch.sh`
has been included to
provide a template for running Nextflow workflows with Ignite outside of
astrocyte.
Both the
`astrocyte_pkg.yml`
and
`ignite_batch.sh`
are set so that the workflow
will run using 4 x 32GB nodes on the BioHPC Nucleus cluster. The
`tolines`
process in the
`main.nf`
workflow has been artificially set to require 16 cpus,
and sleep for 300s so that you can examine the running workflow to see how
Ignite distributes jobs between nodes.
## Questions
...
...
This diff is collapsed.
Click to expand it.
astrocyte_pkg.yml
+
1
−
1
View file @
21c2b069
...
...
@@ -13,7 +13,7 @@ author: 'David Trudgian'
# A contact email address for questions
email
:
'
biohpc-help@utsouthwestern.edu'
# A more informative title for the workflow package
title
:
'
Example
Wordcount
Workflow'
title
:
'
Ignite
Example
Wordcount
Workflow'
# A summary of the workflow package in plain text
description
:
|
This is a minimal test workflow package that counts the occurences of words
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment