Using Containers

This chapter describes how to use containers.

In the Data Analysis System, Apptainer, which is a container environment, can be used for work servers and LSF batch jobs.

Info

KEKCC's Singularity was replaced by Apptainer on 6/14/2023.

Note

Please refer to the following links for general usage of Apptainer.
Link to "Apptainer User Guide"

Preparing the container images creation environment

Administrator Privilege (root) is required for creating contaier images. Therefore, you need to create images on another system where Apptainer was installed and transfer images to an work server.

Note

See the link below for instructions on how to install Apptainer. It can be installed on Linux, Windows and Mac.
Link to "Installing Apptainer"

Creating container images

Create a container image on the system where Apptainer is installed. It is convenient to use a definition file to create images. The following is an example of a definition file (file name: centos7.def) that creates an image that can be used by the development environment and the "module" command based on the latest CentOS image downloaded from Docker Hub.

Note

See this page for the "module" command.

# cat centos7.def
Bootstrap: docker
From: centos:7

%post
    yum groupinstall -y 'Development Tools'
    yum -y install \
    environment-modules \
    which

Create a container image (file name: centos7.sif) with the following command.

# apptainer build centos7.sif centos7.def

The contents of SIF format image files cannot be changed. If you need to add files, etc. instead, use the following command to change to sandbox format.

# apptainer build --sandbox --fix-perms centos7 centos7.def

In this example, a directory called "catnos8" will be created in the current directory. Add files under that directory. You can also start a shell in the container with the following command.

# apptainer shell --writable centos7
Apptainer>

"Apptainer>" is the shell prompt started in the container. You can add packages with the yum command.

Note

The commands that can be used depend on the Linux distribution of the container.

After changing the content, exit from the container shell and convert the sandbox format to SIF format with the following command.

# apptainer build centos7.sif centos7

Transfer the created SIF file to the work server using the scp command, etc.

Compile a program to run in a container

The following is an example of compiling a program using the container image prepared in the above procedure. Here we compile the Intel MPI benchmark using the Intel compiler and Intel MPI.

Note

The source for the Intel MPI Benchmark is available at the link below.
Link to "Intel MPI Benchmarks"

Extract the obtained archive under the home directory of the work server.

$> tar xf IMB-v2019.6.tar.gz
$> cd mpi-benchmarks-IMB-v2019.6

Start the container shell with the following command.

$> apptainer shell --bind /opt,/opt/Modules/modulefiles /path/to/centos7.sif

In the container shell started by "apptainer shell", the following can be inherited and used as is.

  • user ID
  • home directory
  • current directory

You can also make the specified directory available to the container using the "--bind" option. In the above example, /opt is specified to use the Intel compiler and Intel MPI, and /opt/Modules/modulefiles is specified to use the module command.

Compile Intel MPI Benchmarks by executing the following command in the container shell.

Apptainer> source /usr/share/Modules/init/bash
Apptainer> module load intel/2020
Apptainer> export LANG=C
Apptainer> make IMB-MPI1
Apptainer> ls IMB-MPI1
IMB-MPI1
Apptainer> exit

Run LSF Jobs Using Containers

The following is an example of executing the program "IMB-MPI1" compiled above as an LSF job.

Note

See this page for more information on how to run LSF jobs.

The contents of the job to be executed are as follows.

  • job name: apptainer
  • batch queue: b2_a
  • number of slots (number of containers): 4
  • number of calculation servers: 2
  • MPI library: Intel MPI
  • interconnect between calculation servers: InfiniBand

Create the following three shell scripts.

  • submit.sh: script that executes bsub command
  • runjob.sh: script that launches apptainer command as an MPI job
  • imb.sh: script that executes IMB-MPI1 command in the container

The contents of each script are as follows.

$> cat submit.sh
#!/bin/sh

JOBNAME=apptainer
NP=4

bsub -q b2_a -n $NP -R span[ptile=2] -J $JOBNAME -o ${JOBNAME}.outfile ./runjob.sh $NP
  • -R span[ptile=2]: an option to use 2 slots on one computing server.
$> cat runjob.sh
#!/bin/sh

NP=$1
module load intel/2020

mpirun -n $NP apptainer exec \
--bind \
/usr/lib64:/usr/lib64/ofed,\
/etc/libibverbs.d,\
/opt \
centos7.sif ./imb.sh
  • module load intel/2020: module command for using Intel MPI.
  • mpirun -n $NP apptainer: run the apptainer command as an MPI job.
  • apptainer exec ... ./imb.sh: execute "imb.sh" in the container.
  • --bind: makes the Intel Compiler, Intel MPI, and InfiniBand libraries available to your container.
  • /usr/lib64:/usr/lib64/ofed: make the compute server /usr/lib64 visible in the container as /usr/lib64/ofed.

Warning

If "--bind /usr/lib64" is specified instead of "--bind /usr/lib64:/usr/lib64/ofed", original "/usr/lib64" in the container cannot be visible and commands will not work properly.

$> cat imb.sh
#!/bin/sh

export LANG=C
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:/usr/lib64/ofed:/opt/intel20/compilers_and_libraries_2020.1.217/linux/mpi/intel64/libfabric/lib
export I_MPI_FABRICS=shm:ofi
export I_MPI_DEBUG=1
./IMB-MPI1 pingpong
  • export LD_LIBRARY_PATH: specify to use Intel compiler, Intel MPI, and InfiniBand.

Warning

Specify "/usr/lib64" before "/usr/lib64/ofed" in the environment variable LD_LIBRARY_PATH so that the original library of the container is referenced first.

  • export I_MPI_FABRICS=shm:ofi: specify InfiniBand as the interconnect between the calculation servers.
  • export I_MPI_DEBUG=1: specify when checking the interconnect used by Intel MPI.
  • ./IMB-MPI1 pingpong: perform Intel MPI Benchmarks PingPong test.

Info

When the environment variable I_MPI_DEBUG is set to 1 or more, Intel MPI debug information is output to the job output file. If you are using InfiniBand, the following line will be output. "mlx" means InfiniBand.
[0] MPI startup(): libfabric provider: mlx

Make the scripts executable with the following command.

$> chmod +x submit.sh runjob.sh imb.sh

Submit the job with the following command.

$> ./submit.sh

After confirming that the job has finished with the bjobs command, confirm that the job result is output to "apptainer.outfile" in the current directory.