Containers

From ACCRE Wiki

Containers are a form of lightweight, virtualized operating environments that allow you to run an application and its dependencies in resource-isolated processes. Containers enable developers to package an application with all of its dependencies into a standardized unit for software development.

Unlike traditional virtual machines (VMs) which virtualize the entire hardware stack, containers virtualize at the OS level, with multiple containers running atop the OS kernel directly. This means they are more portable, efficient, and scalable than VMs.

Docker

Writing a Dockerfile

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Here's an example of a Dockerfile: 

1. Use an official Python runtime as a parent image
FROM python:3.7-slim
 
2. Set the working directory in the container
WORKDIR /usr/src/app
 
3. Copy the current directory contents into the container at /usr/src/app
COPY . .
 
4. Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
5. Make port 80 available to the world outside this container
EXPOSE 80
 
6. Define environment variable
ENV NAME World
 
7. Run app.py when the container launches
CMD ["python", "app.py"]

Detailed information on Dockerfile instructions can be found at the Dockerfile reference.

Creating a Container Using Docker

To create a Docker container, you first need to build an image from your Dockerfile, and then run a container from this image: 

docker build -t my-python-app .
docker run -p 4000:80 my-python-app

These commands build and run your application in a Docker container.

Creating a Docker Hub Account and Uploading the Container to Docker Hub

To share your Docker container images, you need to upload them to Docker Hub. First, create a Docker Hub account. Then follow these steps: 

1. Authenticate your Docker client to Docker Hub:
  docker login

2. Tag your image to match your Docker Hub username:
  docker tag my-python-app username/my-python-app

3. Upload your image to Docker Hub:
  docker push username/my-python-app

Complete instructions can be found here.

Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Here's an example `docker-compose.yml` file for the application above: 

version: '3'
services:
  web:
    build: .
    ports:
    - "4000:80"

With this file in place, you run the following command to start your application: 

docker-compose up

And to stop your application, you would use: 

docker-compose down

More information on Docker Compose can be found here

Using Docker Containers in ACCRE With Singularity

ACCRE users can run their Docker containers using Singularity, which is a container platform focused on supporting "mobility of compute" — the ability to move compute environments and maintain reproducibility.

Since Singularity can import Docker images directly from Docker Hub, you can work on your local machine within a Docker container and then easily transfer and run it within ACCRE's infrastructure.

Here is how you would pull and run a Docker container inside ACCRE using Singularity: 

1. First, load Singularity module [Generally, this is only required if instruction 2 doesn't work]:
   ml GCC Singularity
2. Then pull the Docker image from Docker Hub:
   singularity pull image_name.sif docker://username/my-python-app
3. Finally, run the container:
   singularity exec my-python-app.sif python app.py

This enables easy and seamless transitioning from development to production in HPC environments like ACCRE.

Apptainer/Singularity

Apptainer(previously known as Singularity) is a tool that allows containers to be run within a shared high-performance computing environment. This enables users to control the OS environment (including software libraries) that their jobs run within. For example, if a user wishes to run within an Ubuntu 16.04 environment he or she may do so despite the fact that the OS on the ACCRE cluster is a completely different Linux distribution (i.e. CentOS)! Docker containers themselves cannot be run in a shared environment like the ACCRE cluster for security reasons. However, Docker container images can easily be run as Singularity container images, which are safe to run on the cluster. 

Note: The terms singularity and apptainer are used interchangeably in this document.

When running within a Singularity container, a user has the same permissions and privileges that he or she would have outside the container. A Singularity image generally must first be developed and built from a Linux machine where you have administrative access (i.e. a personal machine), although ACCRE makes standard images available to all cluster users at /accre/common/singularity/ . If you do not have administrative access to a Linux machine, you can create a virtual Linux machine using a free tool like Vagrant or VirtualBox. Singularity has some helpful documentation for how to do this here .

A user’s cluster storage may be accessed from within the Singularity container, but no operations (e.g. the installation of system software) that require root/sudo privileges are allowed within the context of the Singularity container when run from the cluster. Below are some basic instructions for running Singularity on the cluster. The Singularity documentation is very helpful, so we suggest you invest some time reading through it as well.

ACCRE also hosted a seminar on Singularity in Summer 2017. The recording can be found here. Some of the information in this presentation is out of date as new features have been added to Singularity since this recording, but the same principles apply. Please also refer to the | basic singularity commands video provided by the singularity team.

Building a Singularity Image

Once you have installed Singularity on your own Linux machine or virtual machine , you are ready to create your image. First, create a spec file called Singularity that looks like the following: 

BootStrap: docker
From: ubuntu:16.04

%runscript
    echo "This is what happens when you run the container..."

%post
    apt-get update
    apt-get -y install python3 python3-numpy python3-scipy
    # install any other software you need here... 
    mkdir /nobackup /data /panfs

In this file, we are telling Singularity we want to build an image based on the latest version of Ubuntu 16.04. The base image is pulled from Docker Hub (see below). The %runscript section is for defining commands or tasks that you want to run each time you run a container of this image. The %post section contains one-time setup steps that you want to be inside the image. This is where you install your custom software. In this case, we use apt-get to install Python 3 with NumPy and SciPy included. Finally, it’s important if you want to access all your cluster space from within the container to create the following directories from within the container: /nobackup, /data and /panfs. If you don’t make these directories within the container (inside the %post section), you will get a warning message when you run a container of this image on the cluster. In particular, creating /panfs will ensure you have access to all your /home space from within a container run on the cluster. To build the image, run the following command on your Linux machine where you have admin rights: 

# on your personal Linux machine
sudo singularity build ubuntu16-accre.simg Singularity

Building Containers Directly from Docker Hub or Singularity Hub

Often it is not necessary to create your own image because someone else has already created one for you! Docker Hub and Singularity Hub contain registries of images that have been contributed by various users. In fact, the first example above pulls its base image from Docker Hub. If you have found an image that you would like to run directly, you have the option of building it locally or running it directly.

Building the image locally is a good option if you intend to run it as a container multiple times. In this way you don’t need to re-download the image from the web repeatedly (which can be slow). Here is an example of how you might build a local version of an image from Docker Hub: 

sudo singularity build ubuntu16-accre.simg docker://library/ubuntu:16.04

Note you can also use the singularity pull subcommand to accomplish the same thing: 

singularity pull docker://library/ubuntu:16.04

If you need to modify the container in any way (e.g. adding access to ACCRE storage), we recommend creating a spec file as shown above.

Running a Singularity Image on the Cluster

Once you have successfully created your image, you can shell into it with the following command: 

# From the cluster
singularity shell ubuntu16-accre.simg

Note: In the past users had to load Singularity using an Lmod command; as of the upgrade of the cluster to CentOS 7, this is no longer needed.

The shell subcommand is useful for interactive work. Note that you can create new data inside the container that will persist outside the context of the container. For batch processing (e.g. within a SLURM job), you should use the exec subcommand instead. For example: 

singularity exec ubuntu16-accre.simg /path/to/my/script.sh

Where script.sh script contains the processing steps you want to run from within the batch job. You can also pass a generic Linux command to the exec subcommand. Pipes and redirected output are supported with the exec subcommand. Below is a quick demonstration showing the change in Linux distribution: 

[jill@vmps10 ~]$ cat /etc/*release
CentOS release 6.9 (Final)
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
CentOS release 6.9 (Final)
CentOS release 6.9 (Final)

[jill@vmps10 ~]$ singularity shell ubuntu16-accre.img 
Singularity: Invoking an interactive shell within container...
Singularity ubuntu-16.04.simg:~> cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Singularity ubuntu-16.04.simg:~> exit
exit
[jill@vmps10 ~]$

Running GPU-Enabled Containers

Singularity supports running on GPUs from within containers. The --nv option must be passed to the exec or shell subcommand. Here is an example of a spec file for generating a GPU-enabled Tensorflow/Keras image to be run on ACCRE: 

BootStrap: docker
From: tensorflow/tensorflow:latest-gpu-py3

%help
    Singularity image with TensorFlow 1.7 and Keras running on python 3.5 with GPU support. 

%post
    export LC_ALL=C
    export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH
    pip3 install -U keras
    mkdir /nobackup /data /panfs

The instantiation of a container from such an image would look like: 

$ ml GCC Singularity
$ singularity exec --nv my-singularity-image.simg python -c "import tensorflow"

Alternatively, you could run a GPU-enabled image directly from Docker Hub. Here is an example SLURM script for running a GPU-enabled Tensorflow container on ACCRE GPUs: 

#!/bin/bash
#SBATCH --account=MY_GPU_ACCOUNT
#SBATCH --partition=pascal
#SBATCH --gres=gpu:1
#SBATCH --mem=10G
#SBATCH --time=12:00:00

# Clone tensorflow repo to run a tutorial
git clone https://github.com/tensorflow/models.git
singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \
    python ./models/tutorials/image/mnist/convolutional.py

Commonly Encountered Errors

When working with containers and deploying applications, you may encounter a range of common errors. Below are a few typical issues and their probable solutions:

Missing Libraries

Symptoms: Errors stating that a library or dependency is not found when attempting to run your application inside a container.

Possible Solutions:

  • Ensure that all required libraries are listed in your `requirements.txt` (for Python) or an equivalent dependency file for your programming language.
  • Double-check the Dockerfile to ensure the `COPY` command is correctly copying the dependency file into the container and that the `RUN` command properly installs all dependencies.
  • Make sure that you are binding (the -B flag) the right paths when running the singularity container.
 singularity run -B /accre,/cvmfs,/data/your_datapath:/workdir/data/ container.sif
 # here the /workdir/data is the path that is used by services inside the container.

Broken Links/Symlinks

Symptoms: The application fails due to links pointing to files or directories that do not exist within the container's filesystem.

Possible Solutions:

  • Verify that the file or directory the link is pointing to exists within the container. You might need to adjust the Dockerfile to include the necessary files or directories using the `COPY` or `ADD` command.
  • For symlinks to paths outside the container (e.g., shared volumes), ensure that the corresponding volumes are correctly mapped and accessible when running the container with the `docker run -v` option.

Permission Issues

Symptoms: Errors relating to file or directory permissions, often indicating that the application does not have the necessary access rights.

Possible Solutions:

  • Consider setting the appropriate permissions on files and directories using the `chmod` and `chown` commands within the Dockerfile.
  • If running the container as a non-root user, ensure the user is correctly set up with the necessary permissions on the relevant files and directories.

Environment Variables Not Set

Symptoms: Application behavior indicates that environment variables expected by the application are not set or have incorrect values.

Possible Solutions:

  • Use the `ENV` instruction in the Dockerfile or ```%environment``` section in your singularity definition file to set environment variables explicitly.
  • When running the container, you can also set environment variables during runtime as well like so:
 env_var1=val1 env_var1=val2 singularity run -e "VARIABLE_NAME=value" my-image

Multiple Environment Variables Overlapping

If you have setup conda or pip installations in you profile folder, they get available in the container. This might conflict with the container setup which is unaware of programs installed in your home folder. To avoid such situations, you might need to run singularity with ```--cleanenv``` option i.e. 

singularity run -e ./lolcow.sif

ACCRE Singularity GitHub Repository

ACCRE has a Singularity GitHub repository if you would like to see more examples.

We are constantly working on updating the content, but because of the ever changing nature of technology, some of the contents in our Github Repository are outdated. Hence, we also encourage users to contribute their own examples to the repo!

Retrieved from "https://help.accre.vanderbilt.edu/index.php?title=Containers&oldid=727"