Skip to content
Snippets Groups Projects
Commit a44adde0 authored by Peng Lian's avatar Peng Lian
Browse files

Remove astrocyte_cli from the repo

parent 1200e1f8
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 11
− 81
View file @ a44adde0
......@@ -2,92 +2,22 @@
A skeleton repo acting as a host registry for various images used in Astrocyte.
## Container registry login configuration
Step 1. Join this project.
## R
The R images that are used by `Astrocyte CLI` and `Astrocyte` to build and run the R Shiny of the Vizapp.
Step 2. Generate your persional access token.
These images were built based on the Centos7 and the official release of different R versions. The default CRAN were set to https://cloud.r-project.org and https://cran.rstudio.com through `Rprofile.site`. R packages 'BiocManager', 'shiny', 'shinyFiles', and 'devtools' were installed by default as well. The users could customize the location of their libraries by modifying the environment variable R_LIBS. The UT Southwestern proxy settings are included as bellow. To run the containers outside the UTSW network, the environment variables need to be changed.
```
Any Gitlab page --> Edit profile --> Access Token --> Token Name --> select "api" -->
Copy and write down the strings shows in "Your new personal access token" box.
"Token Name" will be the docker username and "Your new personal access token" will be the docker password.
```
Step 3. For docker, besides the `docker login -u USERNAME` command, one can also config the `~/.docker/config.json` file for automatical login.
```
{
"proxies": {
"default": {
"httpProxy": "http://proxy.swmed.edu:3128",
"httpsProxy": "http://proxy.swmed.edu:3128",
"noProxy": "*.swmed.edu,*.swmed.org,*.utsouthwestern.edu,127.0.0.0/8"
}
},
"auths": {
"git.biohpc.swmed.edu:5050": {
"username": "YOUR_TOKEN_NAME",
"password": "YOUR_ACCESS_TOKEN",
"email": "YOUR_EMAIL",
"auth": "base64(username:password)"
}
},
}
```
Step 4. For Singularity, please use `--docker-login` option for interactive login, OR export `SINGULARITY_DOCKER_USERNAME` and `SINGULARITY_DOCKER_PASSWORD` variables for automatical login.
## Astrocyte_cli
The astrocyte_cli container includes Astrocyte 2.0.0 (under development), Nextflow 22.04.5, Singularity 3.9.8 and Python 3.7.13. It can be used as tool to check and test the Astrocyte workflows during development.
### Reqirements
It requires the usner namespace UID/GID mapping to use the interal Singularity.
https://docs.sylabs.io/guides/3.5/admin-guide/user_namespace.html
### Usage
* Load the Singularity module on BioHPC Cluster/Workstation
```
module load singularity/3.5.3
```
* Pull the Astrocyte CLI image
```
singularity pull --docker-login docker://git.biohpc.swmed.edu:5050/astrocyte/container/astrocyte_cli:2.0.0
```
* Run the image to see the help info. It is suggested to use the `run` command to run the image instead of the `exec` command
```
singularity run astrocyte_cli_2.0.0.sif -h
```
* Clone the Astrocyte word count example and checkout to the right version
```
git clone https://git.biohpc.swmed.edu/biohpc/astrocyte_example_wordcount.git
cd astrocyte_example_wordcount
git checkout test_2.0.0
http_proxy=http://proxy.swmed.edu:3128/
https_proxy=http://proxy.swmed.edu:3128/
no_proxy="*.swmed.edu,*.swmed.org,*.utsouthwestern.edu,localhost,127.0.0.1/8"
```
* Validate the configuration of the Astrocyte workflow
```
singularity run astrocyte_cli_2.0.0.sif check astrocyte_example_wordcount/
```
The Astrocyte workflow developers are suggested to use these containers to develop and test there R shiny vizapps. These containers are available through the the R/x.x.x-img modules on BioHPC as well.
* Test run the Astrocyte workflow on the local host
```
singularity run astrocyte_cli_2.0.0.sif test astrocyte_example_wordcount/
```
## Singularity
The inside container of Singularity. It was built with the `--without-suid` option to disable the user namespaces. See [here](https://docs.sylabs.io/guides/3.5/admin-guide/user_namespace.html) and [here](https://github.com/singularityhub/singularity-docker/tree/v3.9.8) for more details.
* Prepare for the shiny libraries
```
singularity run astrocyte_cli_2.0.0.sif shiny-prepare astrocyte_example_wordcount/ --with-container
```
* Run shiny on the local host
```
singularity run astrocyte_cli_2.0.0.sif shiny astrocyte_example_wordcount/ --with-container
```
## Nextflow
The container of nextflow.
astrocyte_cli/2.0.0/Dockerfile deleted 100644 → 0
+ 0
− 275
View file @ 1200e1f8
FROM golang:1.17.12-alpine3.16
LABEL CONTAINER_VERSION="0.0.1" \
AUTHOR="Peng Lian" \
EMAIL="biohpc-help@utsouthwestern.edu"
# build-time varialbes
ARG http_proxy="http://proxy.swmed.edu:3128/"
ARG https_proxy="http://proxy.swmed.edu:3128/"
ARG no_proxy="*.swmed.edu,*.swmed.org,*.utsouthwestern.edu,localhost,127.0.0.1/8"
ARG NXF_HOME="/.nextflow"
ENV LANG="C.UTF-8"
ENV http_proxy="${http_proxy}"
ENV https_proxy="${https_proxy}"
ENV no_proxy="${no_proxy}"
ENV SINGULARITY_VERSION="3.9.8"
ENV NEXTFLOW_VERSION="22.04.5"
ENV ASTROCYTE_CLI_VERSION="2.0.0"
# The following envs only useful for "singularity exec" but not "docker exec".
# See /.singularity.d/env/10-docker2singularity.sh for more info.
# Even though, the best way to use this images is to use the "run" command to call the entrypoint script
ENV TMP_UUID=$(uuidgen)
ENV ASTROCYTE_TMPDIR="/tmp/astrocyte_cli/\${TMP_UUID}"
ENV SINGULARITY_TMPDIR="${ASTROCYTE_TMPDIR}"
ENV SINGULARITY_CACHEDIR="${ASTROCYTE_TMPDIR}"
ENV NXF_HOME="${ASTROCYTE_TMPDIR}/nextflow"
# alpine image with the go tools
RUN env | sort && \
mkdir /astrocyte /apps /home1 /home2 /work /project /archive /cm /programs && \
apk update && \
apk add --no-cache \
bash \
git \
wget \
openssh \
util-linux \
squashfs-tools \
cryptsetup \
shadow-uidmap \
procps \
openjdk11 \
&& \
# install runtime dependencies
apk add --no-cache --virtual .build-deps \
build-base \
linux-headers \
libffi-dev \
sudo \
libtool \
gawk \
gcc \
openssl-dev \
util-linux-dev \
&& \
# install singularity
mkdir -p /usr/local/var/singularity/mnt && \
mkdir -p $GOPATH/src/github.com/sylabs && \
cd $GOPATH/src/github.com/sylabs && \
git clone --recursive https://github.com/sylabs/singularity.git && \
cd singularity && \
git checkout v${SINGULARITY_VERSION} && \
./mconfig --without-suid -p /usr/local && \
export http_proxy="http://proxy.swmed.edu:3128/" && \
export https_proxy="http://proxy.swmed.edu:3128/" && \
nproc="$(nproc)"&& \
make -j "$nproc" -C builddir && \
sudo make -C builddir install && \
\
# install nextflow
mkdir -p /.nextflow/src && \
wget -O /.nextflow/src/nextflow-${NEXTFLOW_VERSION}-all https://github.com/nextflow-io/nextflow/releases/download/v${NEXTFLOW_VERSION}/nextflow-${NEXTFLOW_VERSION}-all && \
chmod 755 /.nextflow/src/nextflow-${NEXTFLOW_VERSION}-all && \
ln -s /.nextflow/src/nextflow-${NEXTFLOW_VERSION}-all /usr/local/bin/nextflow && \
apk del --no-network .build-deps && \
nextflow info
# install python3
# runtime dependencies
RUN set -eux; \
apk add --no-cache \
ca-certificates \
tzdata \
;
ENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
ENV PYTHON_VERSION 3.7.13
RUN apk add --no-cache libc6-compat && \
ln -s /lib/libc.musl-x86_64.so.1 /usr/local/lib/libc.so.6 && \
ls -l /lib/ && \
ls -l /usr/local/lib/
RUN set -eux; \
\
apk add --no-cache --virtual .build-deps \
gnupg \
tar \
xz \
\
bluez-dev \
bzip2-dev \
dpkg-dev dpkg \
expat-dev \
findutils \
gcc \
gdbm-dev \
libc-dev \
libffi-dev \
libnsl-dev \
libtirpc-dev \
linux-headers \
make \
ncurses-dev \
openssl-dev \
pax-utils \
readline-dev \
sqlite-dev \
tcl-dev \
tk \
tk-dev \
util-linux-dev \
xz-dev \
zlib-dev \
; \
\
wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; \
wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; \
GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; \
#gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; \
#gpg --batch --verify python.tar.xz.asc python.tar.xz; \
#command -v gpgconf > /dev/null && gpgconf --kill all || :; \
rm -rf "$GNUPGHOME" python.tar.xz.asc; \
mkdir -p /usr/src/python; \
tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; \
rm python.tar.xz; \
\
cd /usr/src/python; \
gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; \
./configure \
--build="$gnuArch" \
--enable-loadable-sqlite-extensions \
--enable-optimizations \
--enable-option-checking=fatal \
--enable-shared \
--with-system-expat \
--without-ensurepip \
; \
nproc="$(nproc)"; \
make -j "$nproc" \
# set thread stack size to 1MB so we don't segfault before we hit sys.getrecursionlimit()
# https://github.com/alpinelinux/aports/commit/2026e1259422d4e0cf92391ca2d3844356c649d0
EXTRA_CFLAGS="-DTHREAD_STACK_SIZE=0x100000" \
LDFLAGS="-Wl,--strip-all" \
# setting PROFILE_TASK makes "--enable-optimizations" reasonable: https://bugs.python.org/issue36044 / https://github.com/docker-library/python/issues/160#issuecomment-509426916
PROFILE_TASK='-m test.regrtest --pgo \
test_array \
test_base64 \
test_binascii \
test_binhex \
test_binop \
test_bytes \
test_c_locale_coercion \
test_class \
test_cmath \
test_codecs \
test_compile \
test_complex \
test_csv \
test_decimal \
test_dict \
test_float \
test_fstring \
test_hashlib \
test_io \
test_iter \
test_json \
test_long \
test_math \
test_memoryview \
test_pickle \
test_re \
test_set \
test_slice \
test_struct \
test_threading \
test_time \
test_traceback \
test_unicode \
' \
; \
make install; \
\
cd /; \
rm -rf /usr/src/python; \
\
find /usr/local -depth \
\( \
\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
-o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name 'libpython*.a' \) \) \
-o \( -type f -a -name 'wininst-*.exe' \) \
\) -exec rm -rf '{}' + \
; \
\
find /usr/local -type f -executable -not \( -name '*tkinter*' \) -exec scanelf --needed --nobanner --format '%n#p' '{}' ';' \
| tr ',' '\n' \
| sort -u \
| awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' \
| xargs -rt apk add --no-network --virtual .python-rundeps \
; \
apk del --no-network .build-deps; \
\
python3 --version
# make some useful symlinks that are expected to exist ("/usr/local/bin/python" and friends)
RUN set -eux; \
for src in idle3 pydoc3 python3 python3-config; do \
dst="$(echo "$src" | tr -d 3)"; \
[ -s "/usr/local/bin/$src" ]; \
[ ! -e "/usr/local/bin/$dst" ]; \
ln -svT "$src" "/usr/local/bin/$dst"; \
done
# if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value '<VERSION>'"
ENV PYTHON_PIP_VERSION 22.0.4
# https://github.com/docker-library/python/issues/365
ENV PYTHON_SETUPTOOLS_VERSION 57.5.0
# https://github.com/pypa/get-pip
ENV PYTHON_GET_PIP_URL https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py
ENV PYTHON_GET_PIP_SHA256 ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d
RUN set -eux; \
\
wget -O get-pip.py "$PYTHON_GET_PIP_URL"; \
echo "$PYTHON_GET_PIP_SHA256 *get-pip.py" | sha256sum -c -; \
\
export PYTHONDONTWRITEBYTECODE=1; \
\
python get-pip.py \
--disable-pip-version-check \
--no-cache-dir \
--no-compile \
"pip==$PYTHON_PIP_VERSION" \
"setuptools==$PYTHON_SETUPTOOLS_VERSION" \
; \
rm -f get-pip.py; \
\
pip --version
# install astrocyte_cli
ADD astrocyte /astrocyte
RUN apk add --no-cache --virtual .build-deps \
build-base \
linux-headers \
libffi-dev \
sudo \
libtool \
gawk \
wget \
gcc \
openssl-dev \
util-linux-dev \
&& \
cd /astrocyte && \
git checkout -t origin/release/${ASTROCYTE_CLI_VERSION} && \
git status && \
ls -l . && \
pip install . && \
apk del --no-network .build-deps && \
astrocyte_cli --version
COPY entry.sh /usr/local/bin/entry.sh
ENTRYPOINT ["bash", "/usr/local/bin/entry.sh"]
\ No newline at end of file
astrocyte_cli/2.0.0/entry.sh deleted 100755 → 0
+ 0
− 17
View file @ 1200e1f8
#!/bin/bash
#
# Please use "docker run" or "singularity run" to use this script as a starter.
#
# Looks like $() will not be excuted in the ENV section of Dockerfile, so the best way is to
# run through this entrypoint script.
export TMP_UUID=$(uuidgen)
export ASTROCYTE_TMPDIR="/tmp/astrocyte_cli/${TMP_UUID}"
export SINGULARITY_TMPDIR="${ASTROCYTE_TMPDIR}"
export SINGULARITY_CACHEDIR="${ASTROCYTE_TMPDIR}"
export NXF_HOME="${ASTROCYTE_TMPDIR}/nextflow"
# wrap cli args with single quote to avoid wildcard expansion
cli=''; for x in "$@"; do cli+="'$x' "; done
exec bash -c "astrocyte_cli $cli"
\ No newline at end of file
astrocyte_cli/build_and_push.sh deleted 100755 → 0
+ 0
− 23
View file @ 1200e1f8
#!/bin/bash
#
# Please note that the build of astrocyte_cli container requires to clone and fetch the repo manualy!
#
PACKAGE="astrocyte_cli"
VERSION="2.0.0"
CONTAINER_TAG="${PACKAGE}:${VERSION}"
CONTAINER_NAME="git.biohpc.swmed.edu:5050/astrocyte/container/${CONTAINER_TAG}"
export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0
# build with proxy settings
docker build -f ./${VERSION}/Dockerfile -t ${CONTAINER_TAG} ./${VERSION}
# docker login
# Please create your person "Acess token" and config the "~/.docker/config.json" to login automatically
# Tag and push
docker tag ${CONTAINER_TAG} ${CONTAINER_NAME}
docker push ${CONTAINER_NAME}
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment