README.md 4.58 KB
Newer Older
Devin OKelly's avatar
Devin OKelly committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
This is an example minimal directory for configuring a Vagrant box, using Virtualbox
as the VM provider, which contains Singularity. This allows for building of Singularity
recipes locally to a BioHPC workstation, without relying on Cloud Builder or other remote
endpoints.

Author: Devin O'Kelly, Devin.OKelly@UTSouthwestern.edu
Last Edit: 2021-09-10

Tested on:
Vagrant (``vagrant --version``) 1.9.1
Virtualbox (``vboxmanage --version``) 5.1.38r122592
BioHPC thin client 


Usage instructions are as follows, with more technical detail below that. When possible, it will be noted in which context a command is meant to be executed.

The workstation will be referred to as "host", relative to the vagrant box, which is the "guest". The singularity
container, whether invoked with 'exec', 'run', or 'shell', will be referred to as "Singularity guest".  


0. (optional) Check to see if you have any other virtual machines running (in case of port collision) - ``vagrant global-status`` . You may want to manually check for virtualbox or VMware machines as well, if you suspect you'll have those. Vagrant can usually automatically handle port collisions for SSH.

1. Bring up the Vagrant box. Navigate to the folder containing the Vagrantfile (which should be where this README is), and run
``vagrant up`` from your terminal. This could take a while, so if you are running a remote session you may want to run ``vagrant up | tee log.txt`` instead,
just in case the box creation fails or your terminal drops for some reason.
  - If you want to run Singularity without having to type out the entire path to the binary at ``/usr/local/bin/singularity/``, add the following to your ``~/.bashrc``: 
  - ``alias sudo='sudo env PATH=$PATH'``
  - Notice that the file ``///provision/setup_env.sh`` creates and activates a default venv within the Vagrant box using packages specified in ``requirements.txt``.  

2. Once the Vagrant box is up, get into it with ``vagrant ssh``. This will drop you into an SSH shell running within the Vagrant box, as the ``vagrant`` user, which has sudo rights within the VM. Vagrant should also have automatically mounted the host ``"~/vagrant_singularity/"`` folder to the guest ``"/vagrant/"`` folder - if not, there is something wrong with your Virtualbox or Vagrant installs. 

3. Within Vagrant, if you navigate to /vagrant/, you should see the /vagrant/recipe/ folder which contains the ``example.def`` recipe file.
* ``sudo singularity build ./output.sif /vagrant/recipe/example.def`` will create a file called ``output.sif`` in the current folder. This is your Singularity guest image file.
* ``sudo singularity build --sandbox ./my_sandbox/ /vagrant/recipe/example.def`` will create a **folder** called ``my_sandbox`` in the current folder. This is a Singularity guest sandbox folder. If you navigate inside, you will see what looks like the root of a filesystem, with multiple directories like /bin/ and /var/. 

4. You can run either images or sandbox directories - the difference is that an image is meant to be immutable, and generally to function as a stable application+its environment. Conversely, a sandbox is meant to be mutable.

5. You can convert sandboxes to SIF files, and SIF files to sandboxes, via the build command. If you do this, you may limit reproducibility as Singularity does not track the changes you make within a sandbox in a reproducible manner. As such, it is best to follow the pattern
- Create base definition file
- Create SIF file
- Convert SIF to sandbox
- Within sandbox, figure out what changes need to be made to the image
- Modify or extend the base definition file.
- Create updated SIF file
- repeat.



==========

The Vagrant box functions purely to provide an encapsulated environment where you can have the root permissions needed to build SIF files.

It is expected that once you have a satisfactory Singularity image that you will be able to save that image elsewhere to be shared with others. In the context of having a stable environment, it should then be able to function as the 'engine' via which python and MATLAB scripts are executed.

Once you have developed an application which uses your stable base image, you can then build another container with that application included. This can be accomplished by appending the appropriate build instructions to the definition file, convert a writable sandbox into a static image, or using a From statement in another definition file which points to your base image. For reproducibility, it is generally best to modify a single definition file and rebuild each time, rather than building a sandbox. 




user@biohpcworkstation: 


vagrant@vagrant :


singularity>