Software Stack

From ACCRE Wiki

The R9 cluster now supports software provided by the Digital Research Alliance of Canada (the Alliance). These software packages are available through CVMFS file system mount. Compute Canada(CC) or Digital Research Alliance of Canada maintains wide range of research software to allow users to seamlessly access a large, centrally managed catalog of scientific software without the need for local installations or manual updates. For the software modules not provided by the Alliance, we will maintain and manage our own software list to complement the Alliance's offerings.

Key Notes

  • Software Stack Not Loaded by Default : The software stack is not automatically loaded into your environment. To load it, use the command:
setup_accre_software_stack

Refer to the Using the New Software Stack section for more details.

  • Choice Between Software Stacks:
    • Users can choose between the Alliance-based stack (new) or may choose not to load a software stack and use their own tools on top of the standard Rocky 9 Linux environment.
    • In the future, additional software stacks from other providers may be made available
    • The use of the legacy CentOS 7 software stack may be accessed through a provided container, see legacy software stack section below.
  • Standard Environments (StdEnvACCRE/2023, StdEnv/2020):

StdEnvACCRE/2023 is the default environment that is loaded by Lmod during the startup. It is recommended that you use the versions of software under StdEnvACCRE/2023 or StdEnv/2023. StdEnv/2020 is an old environment having older versions of softwares. While modules under StdEnv/2020 should generally be usable, it has not been fully tested and may not function correctly in all scenarios.

Using the New Software Stack

Setting Up the Software Stack

The setup_accre_software_stack command configures the environment based on the ACCRE_SOFTWARE_STACK environment variable. accre_cc is the default and currently the only supported option for ACCRE_SOFTWARE_STACK variable. The variable will allow for us to make multiple software stacks easily available.

Autoloading the stack at login

Depending on the shell you use, to automatically load the software stack at login, you can add the setup command to your rc file. The .bashrc file is a hidden configuration script in your home directory. It runs automatically whenever you log in using the Bash shell.

Example .bashrc snippet: 

# .bashrc  
# Source global definitions  
if [ -f /etc/bashrc ]; then  
    . /etc/bashrc  
fi 
 
...

# accre_cc is the default; other option is accre_legacy
export ACCRE_SOFTWARE_STACK=accre_cc

# Load ACCRE software stack  
setup_accre_software_stack

Loading Software

Lmod is a powerful tool for managing software modules in shared high-performance computing environments. The process for loading software from the new stack remains unchanged from the old stack. For detailed instructions on using Lmod commands, please visit the existing Software Stack page.

Notes

  • Python, R, and Perl Packages: We no longer accept requests for installing individual Python, R, or Perl packages. Users are responsible to manage their own installations. Refer to the relevant section for guidance:
  • Singularity is Now Apptainer : Containers, including Docker containers, can now be run using the apptainer command.
  • OS-Related Packages : Many OS-related packages (e.g., Git) are included in the StdEnv environment. These packages are available to use without having to use module load command.

Legacy Software Stack

We will no longer maintain our old CentOS 7-based software stack. All new software requests will be installed exclusively on the new stack. If you notice software missing in the new stack, submit a software installation request from PI Dashboard. We'll review and try to include it in the stack promptly. In the meantime, you have two options for accessing legacy software:

Running Legacy Software in a Container

We provide an Apptainer container with a CentOS 7 environment for running legacy software. The container can be accessed from /panfs/accrepfs.vampire/accre/common/singularity/lmod_c7.sif directory. Make sure you bind /cvmfs and /accre directory to the container. As running apptainer run will pass the environment variables in the session to the container, make sure you have a fresh session and have not loaded our software stack with setup_accre_software_stack command. 

# Example apptainer command.
apptainer run --bind /cvmfs:/cvmfs:ro,/accre:/accre:ro /panfs/accrepfs.vampire/accre/common/singularity/lmod_c7.sif /bin/bash

Additionally, for convenience, we have added a legacy_load module to the new stack. This module creates an alias in your environment, forwarding arguments directly to the Apptainer container. 

[~]$ module spider legacy_load

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  legacy_load: legacy_load/1.0
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load legacy_load/1.0

    Help:
      legacy_load (Run software from legacy CentOS 7 stack in containerized environment)

[~]$ module load legacy_load
------------------------------------------------------------
This helper alias command assists with running software from our legacy CentOS 7 stack under an apptainer container.
For greater flexibility, you can use the apptainer commands yourself with the container located at /panfs/accrepfs.vampire/accre/common/singularity/lmod_c7.sif.
Usage : legacy_load "<commands to pass to the container>"
Example : legacy_load "module load ffmpeg; ffmpeg -i input.mp4 -q:a 0 -map a output.mp3"
------------------------------------------------------------

[~]$ legacy_load "module load GCCcore/.11.3.0 FFmpeg;ffmpeg"
ffmpeg version 5.0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.3.0 (GCC)
  configuration: --prefix=/accre/arch/easybuild/software/Compiler/GCCcore/11.3.0/FFmpeg/5.0.1 --enable-pic --enable-shared --enable-gpl --enable-version3 --enable-nonfree --cc=gcc --cxx=g++ --enable-libx264 --enable-libx265 --enable-libmp3lame --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-sdl2
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
  libpostproc    56.  3.100 / 56.  3.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Using the Legacy Stack on R9

Set the ACCRE_SOFTWARE_STACK environment variable to accre_legacy to use the legacy software stack from old cluster. Since those software were compiled under CentOS7 environment, there are no guarantees that the software will work properly with Rocky 9 Linux. Therefore, we strongly advise against using this unless using the container option is not feasible for your work. 

export ACCRE_SOFTWARE_STACK=accre_legacy  
setup_accre_software_stack

Python

Although you will get a Python interpreter from our base OS package without loading the software stack, it is generally not the one you should use, especially if you need to install external Python packages. You should load the appropriate version of Python through the module command. When in doubt, use the latest available version of Python in the software stack.

For convenience, in addition to the base Python module, the SciPy package is also available as a module. The scipy-stack module includes:

  • NumPy
  • SciPy
  • Matplotlib
  • IPython
  • pandas
  • SymPy
  • Nose

The pip command can install packages from a variety of sources, including PyPI and prebuilt distribution packages called Python wheels. Alliance provides Python wheels for a number of packages in their CVMFS repository. You can run the following command to get the list of available wheels: 

avail_wheels

Using wheels provided by Alliance can substantially reduce installation time. You just have to add the --no-index option to pip to prevent installation from PyPI and install only from Alliance's wheelhouse. If you want to ignore prebuilt packages entirely, you can use the --no-binary option in pip. 

pip install numpy --no-index

R

To install packages from CRAN in an interactive R session, use the install.packages function. We recommend loading a GCC module before installing any R packages since most R packages have dependencies on it. 

module load gcc/9.3.0 r/4.0.2

Perl

Loading the perl Interpreter

The Perl language is made available as a module like any other.

To see which versions are installed use: 

setup_accre_software_stack
module spider perl

Then to load a particular version use: 

setup_accre_software_stack
module load perl/5.36.1

Installing Packages

Perl packages can be installed using the Comprehensive Perl Archive Network (CPAN). You can use the cpan tool in the software stack to install packages in your home directory. We recommend loading a GCC module before installing any Perl packages through CPAN: 

module load gcc/12.3

To install a Perl package, enter the CPAN shell and use the install command: 

$ cpan
$ cpan[1]> install Chess

Apptainer/Singularity

In the R9 environment, automatic binding is not enabled yet. Users must manually specify bind mounts using the APPTAINER_BINDPATH environment variable or the --bind flag when launching a container. This page will be updated when automatic binding is enabled.

1. Using the APPTAINER_BINDPATH Environment Variable

You can predefine bind mounts by setting the APPTAINER_BINDPATH variable before running the container: 

export APPTAINER_BINDPATH="/panfs,/data"

This method ensures that the specified directories are consistently mounted for all subsequent container executions within the session.

2. Using the --bind Flag

You can also specify bind mounts directly when starting an Apptainer container: 

apptainer shell --bind=/panfs:/panfs,/data:/data container.sif

This method is useful when you want to define bind mounts on a per-command basis.


Compute Nodes Types

Compute nodes on the ACCRE cluster are heterogeneous in terms of CPU architecture (and also RAM and local disk space). Some compute nodes contain processors that are 4-5 years old, while others use processors that are less than a year old. The ACCRE jobs will be assigned to these nodes by slurm in random way. Usually we use teh CPU arch name to label the compute node types.

Currently in ACCRE we have the following types of compute nodes, from oldest to newest:

  1. westmere;
  2. sandy bridge;
  3. haswell;
  4. broadwell;
  5. skylake;
  6. icelake

The above CPU arch are all Intel CPUs. For AMD CPU, currently we have zen structure (AMD epyc cpus).

The mixture of different CPU arch presents some challenges when it comes to building software, as newer processors can use instructions that are unsupported by older processors. As a result, programs that are built from source code on a newer compute node may not run successfully on a compute node or gateway with an older processor. “Illegal Instruction” error messages are likely to occur in this scenario. This is a typical error when running the local built R libraries.

If the software is compiled on the older CPU arch, but the compiled binary is used on newer CPUs, it may cause an error. This is usually happens with R. For example, sometimes the R user may have the job error like below: 

All output looks like this:
*** caught illegal operation ***
address 0x2b02bd5762bc, cause 'illegal operand'

Sometimes, you may need to restrict job submissions to specific node type. For this, use the --constraint directive in you batch script: 

#SBATCH --constraint=haswell|skylake

This means the job will be assigned to the compute node either with haswell or skylake CPUs. This will exclude the jobs running on sandy bridge and westmere CPU in the cluster.

LMOD

For loading the software from the stack ACCRE provides the user of Lmod. Lmod is a tool for managing software modules within a shared high-performance computing environment (e.g. the ACCRE cluster). Lmod is designed to intelligently manage, negotiate, and enforce the complex dependencies between software and libraries in a HPC environment. This should lead to better usability of administrator-installed software modules and prevent conflicts between lower-level libraries that many modules depend on.

For instructions on basic Lmod usage, please start with the excellent and concise User Guide for Lmod.


Searching for available Modules, Normal modules and Hidden Modules

In Lmod you can use module spider command to search the modules. You can provide part of the module name, and the search is case insensitive: 



[bob@gw02 ~]$ module spider xtensor

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  xtensor:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

     Versions:
        xtensor/0.24.2
        xtensor/0.25.0

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "xtensor" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider xtensor/0.25.0
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

If you can directly used module spider command to search the module names, these modules are normal modules. You can directly search them, load them to use. However there are some modules we choose to hide them during the installation, because they mostly serves as dependent libraries for the normal modules. For example, the GCCcore module is used by the GCC compiler module: 


[bob@gw02 ~]$ module --show-hidden spider gcccore

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  gcccore:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).

     Versions:
        gcccore/.8.4.0
        gcccore/.9.3.0
        gcccore/.10.2.0
        gcccore/.10.3.0
        gcccore/.11.3.0
        gcccore/.12.3
        gcccore/.13.3
     Other possible modules matches:
        gcccorecuda  gcccoreflexiblas  gcccoremkl

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*gcccore.*'

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "gcccore" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider gcccore/.13.3
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

These hidden modules have a . sign in front of their version number to indicate there is a hidden module. To search for a hidden module, you need to append option --show-hidden in the command like: 

module --show-hidden spider

; otherwise it will issue an error: 

[bob@gw01 ~]$ module spider gcccore
Lmod has detected the following error:  Unable to find: "gcccore".

List Modules, Unload Modules

Use module list command to see what modules are currently loaded in your environment: 

[bob@gw01 ~]$ setup_accre_software_stack
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            5) hwloc/2.9.1        9) ucc/1.2.0               13) StdEnvACCRE/2023 (S)
  2) gentoo/2023   (S)   6) ucx/1.14.1        10) openmpi/4.1.5   (m)     14) mii/1.1.2
  3) gcccore/.12.3 (H)   7) libfabric/1.18.0  11) flexiblas/3.3.1
  4) gcc/12.3      (t)   8) pmix/4.2.4        12) openblas/0.3.24 (math)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module

[bob@gw01 ~]$ module load python
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            5) hwloc/2.9.1        9) ucc/1.2.0               13) StdEnvACCRE/2023 (S)
  2) gentoo/2023   (S)   6) ucx/1.14.1        10) openmpi/4.1.5   (m)     14) mii/1.1.2
  3) gcccore/.12.3 (H)   7) libfabric/1.18.0  11) flexiblas/3.3.1         15) python/3.11.5    (t)
  4) gcc/12.3      (t)   8) pmix/4.2.4        12) openblas/0.3.24 (math)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module

The StdEnvACCRE/2023 module is automatically loaded when you run the setup_accre_software_stack command. Note, the StdEnvACCRE/2023 module is a minor variation of the StdEnv/2023 module normally provided by Canada Research Alliance. The reason for this variation is for compatibility with ACCRE hardware virtualization. For most purposes, this module should behave identically to StdEnv/2023.

Use module unload command to remove a module from your environment. For example: 

[bob@gw01 ~]$ module unload python
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            5) hwloc/2.9.1        9) ucc/1.2.0               13) StdEnvACCRE/2023 (S)
  2) gentoo/2023   (S)   6) ucx/1.14.1        10) openmpi/4.1.5   (m)     14) mii/1.1.2
  3) gcccore/.12.3 (H)   7) libfabric/1.18.0  11) flexiblas/3.3.1
  4) gcc/12.3      (t)   8) pmix/4.2.4        12) openblas/0.3.24 (math)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module

With the exception of compilers, a module’s dependencies will not be removed from your environment after unloading the module via module unload. However, Lmod will remove those dependencies if they cause conflicts in a subsequent module load command.

Use module purge command to remove all non-sticky modules from your environment. Modules that have the sticky property and their dependencies will not be removed with a normal purge. The StdEnvACCRE/2023 is sticky as it is designed as the base environment used for the entire stack. Use module --force purge. For example: 

[bob@gw01 ~]$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) CCconfig        4) gcc/12.3      7) libfabric/1.18.0   10) openmpi/4.1.5     13) StdEnvACCRE/2023
  2) gentoo/2023     5) hwloc/2.9.1   8) pmix/4.2.4         11) flexiblas/3.3.1
  3) gcccore/.12.3   6) ucx/1.14.1    9) ucc/1.2.0          12) openblas/0.3.24
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            5) hwloc/2.9.1        9) ucc/1.2.0               13) StdEnvACCRE/2023 (S)
  2) gentoo/2023   (S)   6) ucx/1.14.1        10) openmpi/4.1.5   (m)
  3) gcccore/.12.3 (H)   7) libfabric/1.18.0  11) flexiblas/3.3.1
  4) gcc/12.3      (t)   8) pmix/4.2.4        12) openblas/0.3.24 (math)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module



[bob@gw01 ~]$ module --force purge
[bob@gw01 ~]$ module list
No modules loaded

Modules collections

Lmod provides a simple way to store the currently loaded modules and restore them later through named collections. This method is especially useful to ensure that the correct environment is set when using a user build software and represents a better alternative to your shell initialization file (e.g. ~/.bashrc).

To create a new collection, use module save followed by the name you want to assign to the collection as shown in the following example: 


[bob@gw01 ~]$ setup_accre_software_stack
[bob@gw01 ~]$ module load python/3.11.5
[bob@gw01 ~]$ module load scipy-stack/2025a
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            6) ucx/1.14.1            11) flexiblas/3.3.1          16) ipykernel/2025a
  2) gentoo/2023   (S)   7) libfabric/1.18.0      12) openblas/0.3.24  (math)  17) scipy-stack/2025a (math)
  3) gcccore/.12.3 (H)   8) pmix/4.2.4            13) StdEnvACCRE/2023 (S)
  4) gcc/12.3      (t)   9) ucc/1.2.0             14) mii/1.1.2
  5) hwloc/2.9.1        10) openmpi/4.1.5    (m)  15) python/3.11.5    (t)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module



[bob@gw01 ~]$ module save testpy
Saved current collection of modules to: "testpy"

This will create a new file ~/.lmod.d/testnc that contains the list of the loaded modules now associated with the collection.

IMPORTANT: If you have loaded the default module for a software (i.e. without specifying the version), the named collection will contain the default module, not the specific version’s module assigned as default at the time of the creation of the collection. If a new module is set as default, Lmod will show an error when restoring the collection and you will have to re-create it by following the displayed instructions.

To get the list of the saved collections, use the module savelist command: 

[bob@gw01 ~]$ module savelist
Named collection list :
  1) testpy
[appelte1@gw01 ~]$ module describe testpy
Collection "testpy" contains:
   1) CCconfig         5) hwloc/2.9.1         9)  ucc/1.2.0    13) StdEnvACCRE        17) scipy-stack
   2) gentoo/2023      6) ucx/1.14.1          10) openmpi      14) mii
   3) gcccore/.12.3    7) libfabric/1.18.0    11) flexiblas    15) python
   4) gcc              8) pmix/4.2.4          12) openblas     16) ipykernel/2025a

The modules saved in a named collection can be restored by issuing the module restore command: 

[bob@gw01 ~]$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) CCconfig        4) gcc/12.3      7) libfabric/1.18.0   10) openmpi/4.1.5     13) StdEnvACCRE/2023
  2) gentoo/2023     5) hwloc/2.9.1   8) pmix/4.2.4         11) flexiblas/3.3.1
  3) gcccore/.12.3   6) ucx/1.14.1    9) ucc/1.2.0          12) openblas/0.3.24
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            5) hwloc/2.9.1        9) ucc/1.2.0               13) StdEnvACCRE/2023 (S)
  2) gentoo/2023   (S)   6) ucx/1.14.1        10) openmpi/4.1.5   (m)
  3) gcccore/.12.3 (H)   7) libfabric/1.18.0  11) flexiblas/3.3.1
  4) gcc/12.3      (t)   8) pmix/4.2.4        12) openblas/0.3.24 (math)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module



[bob@gw01 ~]$ module restore testpy
Restoring modules from user's testpy
[bob@gw01 ~]$ module list

Currently Loaded Modules:
  1) CCconfig            6) ucx/1.14.1            11) flexiblas/3.3.1          16) ipykernel/2025a
  2) gentoo/2023   (S)   7) libfabric/1.18.0      12) openblas/0.3.24  (math)  17) scipy-stack/2025a (math)
  3) gcccore/.12.3 (H)   8) pmix/4.2.4            13) StdEnvACCRE/2023 (S)
  4) gcc/12.3      (t)   9) ucc/1.2.0             14) mii/1.1.2
  5) hwloc/2.9.1        10) openmpi/4.1.5    (m)  15) python/3.11.5    (t)

  Where:
   S:     Module is Sticky, requires --force to unload or purge
   m:     MPI implementations
   math:  Mathematical libraries
   t:     Tools for development
   H:                Hidden Module

To delete a named collection simply remove the corresponding file in ~/lmod.d

Retrieved from "https://help.accre.vanderbilt.edu/index.php?title=Software_Stack&oldid=1207"