Containers

From ACCRE Wiki

Containers provide lightweight, virtualized operating environments that allow you to run an application and its dependencies in resource-isolated processes. They enable developers to package an application with all its dependencies into a standardized unit for software development. Unlike traditional virtual machines (VMs) that virtualize the entire hardware stack, containers virtualize at the OS level, with multiple containers running directly on top of the OS kernel. This makes them more portable, efficient, and scalable than VMs.

Apptainer/Singularity

Apptainer (previously known as Singularity) is a tool that allows containers to run within a shared high-performance computing environment. This empowers users to control the OS environment (including software libraries) in which their jobs execute. For instance, a user can run within an Ubuntu 16.04 environment even if the underlying OS on the ACCRE cluster is a different Linux distribution (e.g., Rocky Linux). Docker containers, for security reasons, cannot be directly run in shared environments like the ACCRE cluster. However, Docker container images can be easily used as Apptainer container images, which are safe for cluster execution.


Note: The terms "Singularity" and "Apptainer" are used interchangeably in this document.


When running within an Apptainer container, a user retains the same permissions and privileges they would have outside the container. An Apptainer image typically needs to be developed and built on a Linux machine where you have administrative access (e.g., a personal machine). If you lack administrative access to a Linux machine, you can create a virtual Linux machine using free tools like Vagrant or VirtualBox. Apptainer offers helpful documentation on this process [1].

A user's cluster storage is accessible from within the Apptainer container, but operations requiring root/sudo privileges (such as installing system software) are not permitted within the container's context when run on the cluster. Below are basic instructions for running Apptainer on the cluster. It is highly recommended to explore the comprehensive Apptainer documentation. Please also refer to the apptainer deep dive video provided by the CIQ.

Building an Apptainer Image

Once you have installed Apptainer on your own Linux machine or virtual machine, you are ready to create your image. First, create a spec file named Apptainer with the following content: 

BootStrap: docker
From: ubuntu:16.04

%runscript
	echo "This is what happens when you run the container..."

%post
	apt-get update
	apt-get -y install python3 python3-numpy python3-scipy
	# install any other software you need here...
	mkdir /nobackup /data /panfs

In this file:

  • BootStrap: docker and From: ubuntu:16.04 instruct Singularity to build an image based on the latest version of Ubuntu 16.04, pulling the base image from Docker Hub.
  • The %runscript section defines commands or tasks that will execute each time a container of this image is run.
  • The %post section contains one-time setup steps to be performed inside the image. Here, apt-get is used to install Python 3 with NumPy and SciPy. You can add commands to install any other necessary software.
  • Creating the directories /nobackup, /data, and /panfs within the %post section is crucial for accessing your cluster storage from within the container. Failure to do so will result in a warning message when running the container on the cluster. Specifically, creating /panfs ensures access to your entire /home space from within the container.

To build the image, execute the following command on your Linux machine with administrative privileges: 

# on your personal Linux machine
sudo apptainer build ubuntu16-accre.sif Apptainer

Building Containers Directly from Docker Hub or Singularity Hub

Often, creating your own image is unnecessary as others may have already created suitable ones. Docker Hub and Sylabs Cloud host registries of images contributed by various users. The example below, in fact, pulls its base image from Docker Hub. If you find an image you wish to use directly, you can either build it locally or run it directly.

Building the image locally is advisable if you plan to run it multiple times, as it avoids repeated downloads from the web, which can be slow. You can use the singularity pull subcommand to pull a container from the cloud and create a runnable container image. Since pull command does not rely on having a sudo access, you can run pull command directly on ACCRE. 

# From the cluster
apptainer pull ubuntu16-accre.sif docker://library/ubuntu:16.04

If you need to modify the container (e.g., adding access to ACCRE storage), it's recommended to create a spec file as described earlier.

Running an Apptainer Image on the Cluster

Note: You must explicitly set bind mount to make any directory available inside of the container. See Mounting Directories.

Once your image is successfully created, you can run the container using Apptainer commands. Apptainer is one of our core software and doesn’t have to be loaded through module load command. You can access the container’s shell using the following command on the cluster: 

# From the cluster
apptainer shell ubuntu16-accre.sif

The shell subcommand is useful for interactive work. Note that any new data created inside the container will persist outside its context. For batch processing (e.g., within a SLURM job), use the exec subcommand instead. For example: 

apptainer exec ubuntu16-accre.sif /path/to/my/script.sh

Here, script.sh contains the processing steps you want to execute within the batch job. You can also pass a generic Linux command to the exec subcommand. Pipes and redirected output are supported with exec. The following demonstrates the change in Linux distribution when entering a Apptainer container: 

[jill@vmps10 ~]$ cat /etc/*release
CentOS release 6.9 (Final)
...
[jill@vmps10 ~]$ apptainer shell ubuntu16-accre.sif
Apptainer> cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
...
Apptainer ubuntu-16.04.simg:~> exit
exit

Mounting Directories

In our old CentOS7 environment, certain commonly used paths such as /data, /panfs, and /home were automatically bind-mounted into Apptainer containers by default. However, in the R9 environment, this automatic bind-mounting setup is not configured by default. The rationale behind this decision is to minimize unintended side effects when users run containers. You must manually specify bind mounts to access files and directories on the host system from within the container. This can be done using either the APPTAINER_BINDPATH environment variable or the --bind/-Bflag when invoking the container. Using the APPTAINER_BINDPATH Environment Variable : 

export APPTAINER_BINDPATH="/panfs,/data,/nobackup"
apptainer shell my_python_container.sif

Using the --bind/-B Flag. 

apptainer shell --bind=/panfs:/panfs,/data:/data python_container.sif

If the destination path inside the container is omitted (e.g., --bind=/data), Apptainer will mount it to the same path inside the container.

Running GPU-Enabled Containers

Apptainer supports running on GPUs from within containers. The --nv option must be passed to the exec or shell subcommand. Here's an example of a spec file for creating a GPU-enabled Tensorflow/Keras image for use on ACCRE: 

BootStrap: docker
From: tensorflow/tensorflow:latest-gpu-py3

%help
	Apptainer image with TensorFlow 1.7 and Keras running on python 3.5 with GPU support.

%post
	export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH
	pip3 install -U keras
	mkdir /nobackup /data /panfs

Instantiating a container from such an image would look like this: 

$ apptainer exec --nv my-singularity-image.sif python -c "import tensorflow"

Alternatively, you can run a GPU-enabled image directly from Docker Hub. Here's an example SLURM script for running a GPU-enabled Tensorflow container on ACCRE GPUs: 

#!/bin/bash
#SBATCH --account=MY_GPU_ACCOUNT
#SBATCH --partition=pascal
#SBATCH --gres=gpu:1
#SBATCH --mem=10G
#SBATCH --time=12:00:00

# Clone tensorflow repo to run a tutorial
git clone https://github.com/tensorflow/models.git
apptainer exec --nv docker://tensorflow/tensorflow:latest-gpu \
	python ./models/tutorials/image/mnist/convolutional.py

Again, we recommend you to use apptainer pull to build an image locally from docker hub for repeated usage.

Using Docker to build an image

Since Apptainer supports running Docker images, many users prefer to build docker image locally on their computer, then either create an image archive to transfer to the cluster using tools like scp, or upload the image to Docker Hub so it can be pulled directly from the cluster. If you have a container image already available in Docker Hub or any other repo hosting the Docker image, follow this instruction on how to run docker container in Apptainer.

Writing a Dockerfile

A Dockerfile is a text document containing all the commands a user would execute on the command line to assemble an image. Here's an example: 

# 1. Use an official Python runtime as a parent image
FROM python:3.7-slim

# 2. Set the working directory in the container
WORKDIR /usr/src/app

# 3. Copy the current directory contents into the container at /usr/src/app
COPY . .

# 4. Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# 5. Make port 80 available to the world outside this container
EXPOSE 80

# 6. Define environment variable
ENV NAME World

# 7. Run app.py when the container launches
CMD ["python", "app.py"]

Creating and Running Docker Image

You can use docker build command to create a Docker image from the Dockerfile. Pass the directory on which you have the Dockerfile as an argument to docker build command to build the image. docker run command allows us to run that image as a container in local machine and check if the container is working as intended. 

# This creates a docker image named my-python-app using the Dockerfile from current directory
docker build -t my-python-app .
docker run -p 4000:80 my-python-app

Now that you have built a container image in your local machine, you can transfer that image to the cluster to be able to run it through Apptainer. Here are the two recommended process you could follow to do that :

Option 1 : Creating a Docker Hub Account and Uploading the Image to Docker Hub

To create a sharable Docker container images, you can upload them to Docker Hub. Create a Docker Hub account and follow these steps:

Authenticate your Docker client to Docker Hub: 

docker login

Tag your image to match your Docker Hub username: 

docker tag my-python-app username/my-python-app

Upload your image to Docker Hub: 

docker push username/my-python-app

Upon successfully publishing the docker image in Docker hub, you can follow this section to run the docker container.

Option 2 : Saving Docker Image and Running it with Apptainer

If you prefer not to use Docker Hub, you can save your Docker image as an archive, transfer it to your cluster, and run it using Apptainer. Follow these steps:

Save your Docker image as a tar archive 

docker save -o my-python-app.tar my-python-app

Transfer the image archive to your cluster using scp 

scp my-python-app.tar user@cluster-address:/path/to/destination

On the cluster, load and convert the Docker image using Apptainer 

apptainer build my-python-app.sif docker-archive://my-python-app.tar

Run the container using Apptainer 

apptainer run my-python-app.sif

Commonly Encountered Errors

Missing Libraries

Symptoms: Errors indicating that a library or dependency is not found when trying to run your application inside a container.

Possible Solutions:

  • Ensure all necessary libraries are listed in your requirements.txt (for Python) or the equivalent dependency file.
  • Double-check your Dockerfile to confirm the COPY command correctly transfers the dependency file and that the RUN command installs dependencies.
  • Verify that you are correctly binding paths using the -B flag:
apptainer run -B /accre,/cvmfs,/data/your_datapath:/workdir/data/ container.sif

Broken Links/Symlinks

Symptoms: The application fails because links point to files or directories that do not exist within the container's filesystem.

Possible Solutions:

  • Ensure that the path is mounted properly to the container.
  • Ensure that the paths you are mounting exist and are accessible on the host system
  • Confirm that the file or directory the link points to exists within the container.

Environment Variables Not Set

Symptoms: Application behavior suggests environment variables are missing or incorrect.

Possible Solutions:

  • Use the ENV instruction (Docker) or %environment section (Apptainer) to set variables.
  • Set environment variables at runtime:
env_var1=val1 env_var2=val2 apptainer run -e "VARIABLE_NAME=value" my-image

Multiple Environment Variables Overlapping

If you have Conda or pip installations in your profile folder, they might interfere with the container’s setup.

Solution: 

apptainer run --cleanenv ./lolcow.sif

ACCRE Apptainer GitHub Repository

ACCRE maintains an Apptainer GitHub repository with more examples. While we continuously work to update the content, due to the rapidly evolving nature of technology, some content in our GitHub Repository may be outdated. Therefore, we also encourage users to contribute their own examples to the repository!

Retrieved from "https://help.accre.vanderbilt.edu/index.php?title=Containers&oldid=1045"