Python on the ACCRE cluster
More information Python Official Website
Python is an interpreted programming language that has become increasingly popular in high-performance computing environments because it’s available with an assortment of numerical and scientific computing libraries (numpy, scipy, pandas, etc.), relatively easy to learn, open source, and free.
On its own, the reference implementation of the Python language is poorly suited to scientific computing as it is not compiled to machine instructions and will perform large calculations slowly. However, with the help of several libraries one can perform very fast computations.
To manage these additional packages, Python includes standard support for easily installing additional packages from the internet with a tool called pip. Users can further maintain multiple “virtual environments” on a single machine with different packages installed using a module called venv.
Using Optimized Python from the Software Stack
Engineers at the Digital Research Alliance of Canada (alliancecan.ca) have compiled multiple versions of python with scientific libraries that are linked against highly optimized linear algebras like BLIS, OpenBLAS, and Intel’s MKL, and therefore will in general yield better performance (faster execution time) than the default system Python and packages downloaded from the PyPI index. To see a list of installed versions of Python on the cluster, use Lmod’s spider command:
[bob@gw01 ~]$ setup_accre_software_stack
[bob@gw01 ~]$ module spider python
--------------------------------------------------------------------------------------------------------------------
python:
--------------------------------------------------------------------------------------------------------------------
Description:
Python is a programming language that lets you work more quickly and integrate your systems more effectively.
Versions:
python/2.7.18
python/3.6.10
python/3.7.7
python/3.7.9
python/3.8.2
python/3.8.10
python/3.9.6
python/3.10.2
python/3.10.13
python/3.11.2
python/3.11.5
python/3.12.4
python/3.13.2
Other possible modules matches:
ipython-kernel python-build-bundle
--------------------------------------------------------------------------------------------------------------------
To find other possible module matches execute:
$ module -r spider '.*python.*'
--------------------------------------------------------------------------------------------------------------------
For detailed information about a specific "python" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider python/3.13.2
--------------------------------------------------------------------------------------------------------------------
Per Lmod’s instructions, we can get more information about an installed Python version:
[bob@gw01 ~]$ module spider python/3.12.4
----------------------------------------------------------------------------------------------------------------------------------
python: python/3.12.4
----------------------------------------------------------------------------------------------------------------------------------
Description:
Python is a programming language that lets you work more quickly and integrate your systems more effectively.
Properties:
Tools for development
You will need to load all module(s) on any one of the lines below before the "python/3.12.4" module is available to load.
StdEnv/2023
StdEnvACCRE/2023
This module provides the following extensions:
distlib/0.3.8 (E), filelock/3.13.4 (E), flit_core/3.9.0 (E), hatch_vcs/0.4.0 (E), hatchling/1.24.2 (E), packaging/24.0 (E), pathspec/0.12.1 (E), pip/24.0 (E), platformdirs/4.2.0 (E), pluggy/1.5.0 (E), setuptools-scm/8.0.4 (E), setuptools/70.0.0 (E), tomli/2.0.1 (E), trove-classifiers/2024.4.10 (E), typing_extensions/4.12.1 (E), virtualenv/20.26.2 (E), wheel/0.43.0 (E)
Help:
Description
===========
Python is a programming language that lets you work more quickly and integrate your systems
more effectively.
More information
================
- Homepage: https://python.org/
Included extensions
===================
distlib-0.3.8, filelock-3.13.4, flit_core-3.9.0, hatch_vcs-0.4.0,
hatchling-1.24.2, packaging-24.0, pathspec-0.12.1, pip-24.0,
platformdirs-4.2.0, pluggy-1.5.0, setuptools-70.0.0, setuptools-scm-8.0.4,
tomli-2.0.1, trove-classifiers-2024.4.10, typing_extensions-4.12.1,
virtualenv-20.26.2, wheel-0.43.0
Note that the StdEnvACCRE/2023 module is loaded by default, so the python/3.12.4 may be loaded without manually loading any other dependencies:
[bob@gw01 ~]$ ml python/3.12.4 [bob@gw01 ~]$ python --version Python 3.12.4 [bob@gw01 ~]$ which python /cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcccore/python/3.12.4/bin/python
In addition to Python, the software stack contains an optimized build of several commonly used python packages often described as the "SciPy Stack". These include numpy,
pandas, and matplotlib. These can be loaded into the environment and made available to the python interpreter via the scipy-stack module:
[bob@gw01 ~]$ ml scipy-stack/2025a [bob@gw01 ~]$ python Python 3.12.4 (main, Jun 7 2024, 23:47:47) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '2.2.2'
To use additional python libraries not included in the scipy-stack module we recommend using a virtual environment and installing these into your virtual environment using the pip command. The Research Alliance of Canada software stack provides a "wheelhouse" of optimized and tested python packages. For more information on using virtual environments and the wheelhouse see the sections below.
Examples of Python with the Software Stack
Running a Python script within a SLURM job is generally straightforward. Unless you are attempting to run one of Python’s multi-processing packages, you will want to request a single task, load the appropriate version of Python from your SLURM script, and then redirect your Python file to the Python interpreter.
Numpy Example
The following example runs Python 3.12.4 on a simple Python script demonstrating the utility of writing vectorized Python code with numpy:
[appelte1@gw01 run1]$ ls
python.slurm vectorization.py
[bob@gw01 run1]$ cat python.slurm
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:10:00
#SBATCH --mem=500M
#SBATCH --output=python_job_slurm.out
setup_accre_software_stack
module load python/3.12.4 scipy-stack/2025a
python vectorization.py
[bob@gw01 run1]$ cat vectorization.py
#!/usr/bin/env python
#
# Python script demonstrating vectorized execution
#
from textwrap import dedent
from timeit import timeit
import numpy as np
SETUP = """
import numpy as np
N = int(1e6)
t = np.linspace(-10, 10, N)
x1 = np.zeros(len(t))
x2 = np.zeros(len(t))
"""
NR = 10
def run_native():
"""native, naive, non-vectorized implementation"""
native = dedent("""
for i in range(N):
x1[i] = np.sin(t[i])
""")
result = timeit(native, setup=SETUP, number=NR)
print("native : {:6.3f}s".format(result))
def run_vectorized():
"""vectorized implementation"""
vectorized = dedent("""
x2 = np.sin(t)
""")
result = timeit(vectorized, setup=SETUP, number=NR)
print("vectorized: {:6.3f}s".format(result))
def test_equality():
"""Test equality of the methods, indepently of the speed test"""
N = 10000
t = np.linspace(-10, 10, N)
x1 = np.zeros(len(t))
x2 = np.zeros(len(t))
for i in range(N):
x1[i] = np.sin(t[i])
x2 = np.sin(t)
if (np.array_equal(x1,x2)):
print("arrays equal!")
if __name__ == '__main__':
run_native()
run_vectorized()
test_equality()
[bob@gw01 run1]$ sbatch python.slurm
Submitted batch job 1213121
After waiting a few minutes:
[bob@gw01 run1]$ cat python_job_slurm.out native : 11.076s vectorized: 0.173s arrays equal!
Image Manipulation Example
In this example, we will use the pillow image manipulation library that is included in the scipy-stack in order to convert a PNG image and create a blurred JPEG version with the interactive python interpreter.
[bob@gw01 run1]$ setup_accre_software_stack
[bob@gw01 run1]$ ml python/3.12.4 scipy-stack/2025a
[bob@gw01 run1]$ python
Python 3.12.4 (main, Jun 7 2024, 23:47:47) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image, ImageFilter
>>> original = Image.open('peewee.png')
>>> original.size
(522, 599)
>>> original.format
'PNG'
>>> converted = original.convert('RGB')
>>> blurred = converted.filter(ImageFilter.BLUR)
>>> blurred.save('peewee-blurred.jpg')
>>> quit()
- Results
-
Left: peewee.png
-
Right: peewee-blurred.jpg
Installing Additional Packages with Virtual Environments
When users need to use Python packages not included with the scipy-stack module in the Lmod software stack, our recommended option is to create a Python virtual environment and use pip to install additional packages into that environment, ideally from the Research Alliance of Canada wheelhouse.
A virtual environment is a self-contained and independent set of Python packages which can be easily created, modified, and cleanly removed by individual users as needed.
Managing Python Virtual Environments
Before creating or using a Python virtual environment, you should set up your Lmod modules that contain the compiled Python interpreter that will be used in your environment.
[bob@gw01 run1]$ setup_accre_software_stack [bob@gw01 run1]$ ml python/3.12.4 scipy-stack/2025a
You may wish to create a named module collection for your set of loaded modules so to make it easy to restore your Lmod environment in future sessions or for batch jobs.
A virtual environment can have an arbitrary name and be placed in any directory that you have access to. To create a virtual environment named myvenv, use the command:
[bob@gw01 run1]$ python -m venv myvenv
This will create a directory myvenv within your current working directory which contains all the files needed for your environment.
To utilize the virtual environment in your session, you must “activate” it with the following command:
[bob@gw01 run1]$ source myvenv/bin/activate (myvenv) [bob@gw01 run1]$
Notice that the prompt changes to show the active Python virtual environment in parenthesis. To exit your virtual environment, use the “deactivate” command:
(myvenv) [bob@gw01 run1]$ deactivate [bob@gw01 run1]$
ACCRE users may have as many virtual environments as they desire, limited only by their filesystem quotas. When you want to permanently remove a virtual environment, you can simply delete the directory:
[bob@gw01 run1]$ rm -r myvenv
Note that every time you start a new shell session or run a job you will need to set up the software stack, load the modules that you previously loaded when creating your virtual environment, and finally activate your virtual environment. In this example if we wanted to use the virtual environment in a slurm script then we would need to add the following setup commands after all #SBATCH directives and before calling our actual python application:
setup_accre_software_stack ml python/3.12.4 scipy-stack/2025a source /home/bob/run1/myvenv/bin/activate
Note that we have used the absolute path to the virtual environment when activating, which assures that you are activating the correct environment no matter what working directory you happen to be in.
Managing Packages in a Virtual Environment
After activating a virtual environment, no Python packages will be initially installed beyond the Python standard library and any Lmod Python libraries you have loaded. Gnereally speaking, to install additional packages into your virtual environment use the pip install PACKAGE command where PACKAGE is the name of your Python package in the public Python Package Index (PyPI). This will install the package into your virtual environment along with any required dependencies.
While you can use the public PyPI index to install any python package into your virtual environment, the Digital Research Alliance of Canada provides many optimized python packages in their "WheelHouse". When you have loaded a python module from the stack and created your virtual environment you will be able to use the WheelHouse to install packages rather than the public index. Installing python binary packages (a.k.a. "wheels") from the WheelHouse should generally be faster and more performant than using the public PyPI index.
As an example, we will consider installing a python molecular dynamics library MDAnalysis.
First, we will start a new session, setup the software stack, load required modules, and activate the virtual environment created in the section above:
[bob@gw02 run1]$ setup_accre_software_stack [bob@gw02 run1]$ ml python/3.12.4 scipy-stack/2025a [bob@gw02 run1]$ . myvenv/bin/activate
If you have not already done so, it is a good practice to update the pip command used in your virtual environment:
(myvenv) [bob@gw02 run1]$ pip install --no-index --upgrade pip
Note the --no-index flag passed to pip. Using this flag when calling pip will ensure that any packages are taken from the WheelHouse and not the public PyPI index. If you need to install a package that is not available in the WheelHouse, remove the --no-index flag when calling pip.
The Digital Research Alliance of Canada provides a tool to search avaiable wheels from their WheelHouse. This command is avail_wheels. In this example we will search for the MDAnalysis library:
(myvenv) [bob@gw02 run1]$ avail_wheels "MDAnalysis*" name version python arch --------------- --------- -------- ------- MDAnalysis 2.9.0 cp312 generic MDAnalysisTests 2.9.0 py3 generic
Then install the desired packages:
(myvenv) [bob@gw02 run1]$ pip install --no-index MDAnalysis MDAnalysisTests
To install a specific version of a package into your virtual environment, you can specify the requirement with ==, for example pip install --no-index MDAnalysis==2.9.0.
To uninstall a package, use the command pip uninstall PACKAGE. Note that this will not uninstall any dependencies that you installed along with that package.
You can get a list of all installed packages in your environment and their versions with the pip freeze command. This can be exported to a requirements file with pip freeze > requirements.txt. For example:
(myvenv) [bob@gw01 run1]$ pip freeze > mda-requirements.txt (myvenv) [bob@gw01 run1]$ cat mda-requirements.txt asttokens==3.0.0+computecanada attrs==25.3.0+computecanada comm==0.2.2+computecanada ...additional packages... wcwidth==0.2.13+computecanada widgetsnbextension==4.0.13+computecanada
For reproducibility, you can install a specific set of packages from a previous environment into a new one from an existing requirements.txt file with the command pip install -r requirements.txt.
Using A Virtual Environment
In this example, we will use some example code adapted from the MDAnalysis quickstart guide to submit a python job to produce a simple plot using test data.
First we will need a python analysis script to use MDAnalysis, numpy, and matplotlib, here is the file mda.py that will be submitted:
import pandas as pd
import matplotlib.pyplot as plt
import MDAnalysis as mda
from MDAnalysis.tests.datafiles import PSF, DCD, GRO, XTC
import warnings
# suppress some MDAnalysis warnings about PSF files
warnings.filterwarnings('ignore')
print("Using MDAnalysis version", mda.__version__)
u = mda.Universe(PSF, DCD)
print(u)
rgyr = []
time = []
protein = u.select_atoms("protein")
for ts in u.trajectory:
time.append(u.trajectory.time)
rgyr.append(protein.radius_of_gyration())
rgyr_df = pd.DataFrame(rgyr, columns=['Radius of gyration (A)'], index=time)
rgyr_df.index.name = 'Time (ps)'
rgyr_df.plot(title='Radius of gyration')
plt.savefig("rog.png")
The slurm submission script to run this code called mda.slurm is as follows:
#!/bin/bash #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --time=00:10:00 #SBATCH --mem=1G #SBATCH --output=python_job_mda.out setup_accre_software_stack module load python/3.12.4 scipy-stack/2025a source myvenv/bin/activate python mda.py
This can be submitted with the sbatch command:
(myvenv) [bob@gw01 run1]$ sbatch mda.slurm Submitted batch job 1229273
Then after a few minutes to let it run we can check the output:
(myvenv) [bob@gw01 run1]$ cat python_job_mda.out Using MDAnalysis version 2.9.0 <Universe with 3341 atoms> (myvenv) [bob@gw01 run1]$ ls -lh rog.png -rw-r----- 1 bob accre 26K Apr 2 19:00 rog.png
This has produced the small file rog.png with an analysis plot from the test data.
Using a Temporary Virtual Environment to Optimize Job Performance
The Software Stack maintained by the Research Alliance of Canada is provided on the ACCRE cluster nodes using a special shared filesystem called CVMFS. This system was developed at CERN specifically for distributing software in a scalable and performant manner. Jobs that use software that is provided via CVMFS or locally on a compute server's hard drive may greatly outperform those that rely on software installed on a general-purpose network filesystem such as the one used for /home, /data, or /nobackup at ACCRE.
For this reason, jobs that rely on packages installed in a virtual environment stored on /home, /data, or /nobackup may perform poorly due to waiting on network file operations. If the python packages used by a job are present in CVMFS or on the local disk then they can be much more easily cached in system memory.
One way to improve job performance is to create a temporary virtual environment for your job in the /tmp directory of a compute node which resides on the local disk. This may seem counterintuitive as it may introduce a few minutes of overhead for each job and will waste disk space, but as long as the job cleans up its usage of /tmp at the end of the job, the excess disk usage should not be a problem. For a job running several hours, the few minutes of initial overhead will be reclaimed when the job does not need to wait on network file operations to read the python packages installed in the virtual environment.
ACCRE provides a simple bash script setup_accre_runtime_dir which can be used via the command source
setup_accre_runtime_dir in your slurm job. This script will create a secure directory inside /tmp that is owned by your user and will set up a signal trap so that the directory will be removed upon job completion even if your code fails. The location of the temporary directory is then stored in the environment variable $ACCRE_RUNTIME_DIR.
In the MDAnalysis example above, after we created a virtual environment we used pip freeze to create a file mda-requirements.txt listing the packages and versions used in that environment. This file can be used to create identical virtual environments in /tmp for individual submitted jobs.
Using these tools, we can make an improved submission script that creates a temporary virtual environment on the local disk of the compute node for better performance:
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:10:00
#SBATCH --mem=1G
#SBATCH --output=python_job_mda2.out
setup_accre_software_stack
module load python/3.12.4 scipy-stack/2025a
source setup_accre_runtime_dir
echo "Using temporary directory ${ACCRE_RUNTIME_DIR}"
python -m venv ${ACCRE_RUNTIME_DIR}/venv
source ${ACCRE_RUNTIME_DIR}/venv/bin/activate
pip install --no-index --upgrade pip
pip install --no-index -r mda-requirements.txt
python mda.py
Jupyter Notebooks
Jupyter notebooks (formerly iPython notebooks) enable a user to interactively code in Python from a web browser with support for inline plotting, equation editing, among many other things. Historically, cluster environments have been used for batch processing rather than interactive processing, however advances in web-based cluster interfaces have made these environments also suitable for interactive coding with Jupyter.
On the ACCRE cluster, the preferred method of using a Jupyter notebook is through the ACCRE Visualization Portal. Please refer to the Portal documentation for instructions on starting a notebook server. Jupyter notebook servers run on the ACCRE compute nodes as scheduled SLURM jobs and so users can request whatever resources are needed for their interactive work.
For computationally intensive or otherwise long running tasks, we recommend that the notebook be used only for code development and testing on smaller samples, and that the bulk of the computation be performed in Python scripts submitted as non-interactive batch jobs if possible.
Using Python on GPU Nodes
Python may be used on ACCRE GPU nodes just as it is on normal compute nodes, but additional Lmod packages compiled with CUDA support are available on these nodes. To explore available packages, set up your environment, and test code, it is recommended to use the salloc command to run a short interactive job on a GPU node and develop from the command line interface on that node, for example:
[bob@gw01 test]$ salloc --account=accre_guests_acc --partition=batch_gpu --gres=gpu:nvidia_geforce_rtx_2080_ti:1 --time=1:00:00
salloc: Pending job allocation 1229824
salloc: job 1229824 queued and waiting for resources
salloc: job 1229824 has been allocated resources
salloc: Granted job allocation 1229824
salloc: Waiting for resource configuration
salloc: Nodes gpu0037 are ready for job
[bob@gpu0037 test]$ setup_accre_software_stack
[bob@gpu0037 test]$ ml python/3.12.4 scipy-stack/2025a cuda/12.6
[bob@gpu0037 test]$ avail_wheels "tensorflow*"
name version python arch
----------------------------- --------- -------- -------
tensorflow 2.17.0 cp312 generic
tensorflow_datasets 4.9.4 py3 generic
tensorflow_estimator 2.15.0 py2,py3 generic
tensorflow_federated 0.57.0 py3 generic
tensorflow_gan 2.0.0 py2,py3 generic
tensorflow_hub 0.14.0 py2,py3 generic
tensorflow_metadata 1.15.0 py3 generic
tensorflow_model_analysis 0.29.0 py3 generic
tensorflow_model_optimization 0.8.0 py2,py3 generic
tensorflow_privacy 0.8.9 py3 generic
tensorflow_probability 0.25.0 py2,py3 generic
tensorflow_tensorboard 1.5.1 py3 generic
[bob@gpu0037 test]$ python -m venv tf-venv
[bob@gpu0037 test]$ source tf-venv/bin/activate
(tf-venv) [appelte1@gpu0037 test]$ pip install --no-index --upgrade pip
...installation output skipped...
(tf-venv) [appelte1@gpu0037 test]$ pip install --no-index tensorflow
...installation output skipped...
(tf-venv) [bob@gpu0037 test]$ python
Python 3.12.4 (main, Jun 7 2024, 23:47:47) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
...warning messages skipped...
>>> tf.config.experimental.list_physical_devices('GPU')
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>> quit()
(tf-venv) [bob@gpu0037 test]$ deactivate
[bob@gpu0037 test]$ rm -r tf-venv
[bob@gpu0037 test]$ exit
exit
salloc: Relinquishing job allocation 1229824
salloc: Job allocation 1229824 has been revoked.
[bob@gw01 test]$
Python kernels on Jupyter Notebooks with CUDA support are also available through the ACCRE Visualization Portal. In addition, you can create a GPU desktop on the Visualization Portal to use for code development and testing.
ACCRE Policy Regarding Anaconda
Anaconda provides an easy to use, extended distribution of Python that is widely used in many domains on personal computers. However, Anaconda has been found to be poorly suited to clustered environments such as ACCRE. The use of Anaconda is no longer supported at ACCRE and we are generally not able to help debug issues relating to or resulting from the use of Anaconda on the cluster. If at all possible, we ask that you please do not install Anaconda on the ACCRE cluster.
This policy is similar to that of the Research Alliance of Canada which provides our current Software Stack. Their Anaconda Policy Documentation page provides more detailed reasoning for this policy along with some helpful tips to transition away from Anaconda.
Please note that this request to not install Anaconda does not apply to software installed within an Apptainer (formerly Singularity) container, but note that our ability to support arbitrary container environments is limited.
Using manylinux Wheels Outside of the CC WheelHouse
The use of manylinux wheels that are available on the public PyPI index that are not compiled by the Digital Research Alliance of Canada (CC) and available in the WheelHouse is discouraged and is not supported. Please do not open an ACCRE helpdesk ticket regarding the use of manylinux wheels that are not in the WheelHouse in conjunction with the CC software stack. If you need to use such packages, we recommend that you use your own python interpreter ideally within an Apptainer container and do not initialize the software stack at all for your workflow.
However, we understand that in some cases an analysis may be able to make use of one or more simple packages not available in the wheelhouse and that CC has intentionally disabled the ability to use manylinux wheels outside of the WheelHouse. It may be possible to enable this functionality using the procedure below, but these instructions are provided as an example only, and may not work in the future.
The code intentionally disabling manylinux wheels is referenced by the PYTHONPATH variable set by the CC software stack shell environment setup:
$ echo $PYTHONPATH /cvmfs/soft.computecanada.ca/easybuild/python/site-packages:/cvmfs/soft.computecanada.ca/custom/python/site-packages $ ls -lh /cvmfs/soft.computecanada.ca/custom/python/site-packages total 1.5K -rw-rw-r-- 1 cvmfs cvmfs 220 Oct 6 2022 _manylinux.py -rw-r--r-- 1 cvmfs cvmfs 470 Oct 21 2022 _manylinux.pyc -rw-rw-r-- 1 cvmfs cvmfs 199 Jan 14 2019 _manylinux.pyo
To allow the installation of external manylinux wheels into a virtual environment, simply remove the custom directory from the PYTHONPATH environment variable:
$ export PYTHONPATH=/cvmfs/soft.computecanada.ca/easybuild/python/site-packages
