CINES software stack
Introduction
Before going into more technical subjects, know that CINES uses Spack to provide precompiled softwares which we shall name Spack products. We recommend that you first search for an already CINES provided software before trying to install it yourself. To look for such preinstalled software, search the modules using module spider or browse the catalog.
A note on dependencies
Spack tries to solve the problem of satisfying the somewhat esoteric and hpc specific dependencies and does a reasonably good job at it. It is good enough that it is considered viable and used by many sites. That said, note that dependencies are a problem and that even if Spack mitigates this to a certain extent, the resolution of a dependency is still a problem. You should probably not add new dependencies lightly. To that, we should add:
Are all your dependencies built and tested in CI on all target platform ? Probably not, so you should be even more careful with dependencies?
It is almost always costlier to fix something, than to start from scratch.
Basic tools and libraries
CINES offers a basic software suite primarily deployed using the Spack package manager and GCC toolchain. This suite includes commonly used tools and libraries with no MPI dependencies (with a few exceptions), ensuring there are no risks of ABI (Application Binary Interface) incompatibilities or symbol conflicts during linking.
You can safely use these modules without risk of conflicts with other modules, either to link your applications with the available libraries or to use the underlying tools.
Running module available could give you:
--------------------------------------------------------- /opt/software/gaia/prod/latest/Base ---------------------------------------------------------
anaconda3/2023.09-0 gdal/3.8.5 libsndfile/1.0.28 ncview/2.1.9-mpi r/4.4.0
blitz/1.0.2 gdb/14.1 libtirpc/1.3.3 ninja/1.11.1 rust/1.78.0
bzip2/1.0.8 gnuplot/6.0.0 libxml2/2.10.3 nsimd/3.0.1 sox/14.4.2
cfitsio/4.3.0 gsl/2.7.1 libxsmm/1.17 papi/7.1.0 swig/4.1.1
cmake/3.27.9 hpctoolkit/2023.08.1-mpi metis/5.1.0-int64 paraview/5.13.0-osmesa zlib-ng/2.1.4
easybuild/4.7.0 hpcviewer/2024.02 metis/5.1.0 (D) paraview/5.13.0 (D) zlib/1.3.1
eigen/3.4.0 hwloc/2.9.1-rocm-gpu mmg/5.7.1 poppler/23.04.0
ffmpeg/6.1.1 jasper/3.0.3 mpifileutils/0.11.1-mpi python/3.12.1
firefox/134.0.2 julia/1.10.0 nco/5.1.9-mpi qt/5.15.11
Extensible software stacks
CINES offers extensible software stacks in the form of modules named according to the format <compiler>-[GPU|CPU]-<version>. Each extensible software stack groups together a set of software packages targeting a specific architecture and built using a specific toolchain. CINES will provide software stacks adapted to each Adastra hardware partition. Each software stack version is based on a specific version of the Cray programming environment (cpe/XX.YY).
The <compiler>-[GPU|CPU]-<version> modules can be considered as architecture- and compiler-specific environments. The <version> number represents the production version of the software stack. CINES will endeavor to respect semantic versioning concepts.
Adastra partition |
Extensible software stacks |
Toolchain used |
|---|---|---|
MI250 (GPU) |
CCE-GPU-<version> |
CCE
|
GCC-GPU-<version> |
GCC
|
|
GENOA (CPU) |
CCE-CPU-<version> |
CCE
|
GCC-CPU-<version> |
GCC
|
|
|
TBD |
TBD |
In addition to these extensible software stacks mapped to the partitions of Adastra, we provide what we call Spack user or Spack pre-configured environment (PreConf). It enables the user to install their own softwares by reusing the CINES’ Spack configuration and binary cache. Spack PreConf provides Spack support for a given partition and a set of compiler. The syntax is: spack-user-<version>. More information on how you can reuse the CINES’ Spack configuration is given in the Using CINES’ pre-configured (PreConf) Spack installation section.
Running module available could give you:
--------------------------------- /opt/software/gaia/prod/latest/Core ---------------------------------
CCE-CPU-3.2.0 CCE-GPU-3.2.0 GCC-CPU-3.2.0 GCC-GPU-3.2.0 develop spack-user-3.2.0
The develop extensible module exposes the software stacks being currently developed (release candidate). Running module load develop and then module av could give you:
--------------------------------- /opt/software/gaia/dev/latest/Core ----------------------------------
CCE-CPU-4.0.0 CCE-GPU-4.0.0 GCC-CPU-4.0.0 GCC-GPU-4.0.0 spack-user-4.0.0
Warning
This category includes software versions under development and testing before their deployment in production. Use at your own risk. Package stability is not guaranteed, and packages may be removed or modified when redeploying develop packages/modules.
A production release will take place every 3 to 6 months. Each such release may introduce new versions for the components of the stack (say a CCE or GCC version bump). As the machine evolves, more extensible software stacks (based on a CrayPE release) and their respective products will be added, some will be deprecated. We do not guarantee an extensible software stacks will be produced for each CrayPE release. CINES is likely to release a new extensible software stacks at the start of each DARI call. We guarantee that the CINES’ software stack will be available for at least 2 CrayPE releases before being removed from the system. This should give the user something like 9 months to 1 year of support per release (at least).
When the software stack is put into production, the old modules are moved to a space called deprecated and are still available, as shown below using module available:
------------------------------------ /opt/software/gaia/deprecated ------------------------------------
CCE-CPU-3.1.0 CCE-GPU-3.1.0 GCC-CPU-3.1.0 GCC-GPU-3.1.0 archive
Finally, deprecated modules are moved to the .archive directory and become available only after the module load archive command is executed. These archives are kept for a long time, our only support limit would be the operating system itself not being able to run the binary. In this case, ask svp@cines.fr for a reinstallation of the product old product version.
Using the CINES software stack
To see the softwares made available by an environment, you can simply load the extensible module corresponding to an environment. That said the recommended way is the following; supposing you want to use the GROMACS software, you would do:
$ module spider gromacs
----------------------------------------------------------------------------
gromacs:
----------------------------------------------------------------------------
Versions:
gromacs/2021.4-mpi-omp-plumed-ocl-python3
gromacs/2022.3-mpi-omp-plumed-python3
gromacs/2022.5-mpi-omp-plumed-python3
gromacs/2023-mpi-omp-plumed-ocl-python3
gromacs/2023.3-cp2k-omp-plumed-mpi
gromacs/2023.3-mpi-omp-plumed-python3
gromacs/2024.3-cp2k-omp-mpi
Here, module shows multiple GROMACS versions (2021.4, 2022.3, 2022.5, 2023, …) but with different variants for the 2023 version. Lets use gromacs/2023-mpi-omp-plumed-ocl-python3. To find which programming environment provides the software package we do:
$ module spider gromacs/2023-mpi-omp-plumed-ocl-python3
----------------------------------------------------------------------------
gromacs: gromacs/2023-mpi-omp-plumed-ocl-python3
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "gromacs/2023-mpi-omp-plumed-ocl-python3" module is available to load.
CCE-GPU-2.0.0
CPE-23.02-cce-15.0.1-GPU-softs
This tells us that gromacs/2023-mpi-omp-plumed-ocl-python3 is available, under the CCE-GPU-2.0.0 and CPE-23.02-cce-15.0.1-GPU-softs extensible module. Latest stable version of the extensible module/environment (i.e. production) should be preferably used.
In your scripts, the procedure above reduces to the following commands:
$ module purge
$ module load CCE-GPU-2.0.0
$ module load gromacs/2023-mpi-omp-plumed-ocl-python3
Advanced module mixing
As an example, let’s suppose you’ve located (using the module spider) a library from the CINES software stack. Let’s assume that this library has been built in an environment (extensible module of the form <compiler>-[GPU|CPU]-<version>) using the Cray compiler, but you want to use the GCC toolchain (PrgEnv-gnu). Conventional wisdom says you shouldn’t mix the two.
In practice, if the interfaces are properly defined, as it is in the library used in this example, and if the library does not have toolchain dependencies such as an OpenMP runtime (which you absolutely do not want to mix), then you should be able to have interoperability of the library across toolchains.
Note
Check the dependencies on shared libraries using readelf -d or ldd. It is trickier on static libraries.
Using CINES’ pre-configured (PreConf) Spack installation
Note
You may also find Spack PreConf designated by the Spack user name.
Adastra provides a Spack module that allows you to load a pre-configured instance of Spack (adapted to Adastra’s architecture). This Spack instance will allow you to compile software while reusing the Cray programming environment toolchain and architecture. The software will be installed in a location determined by the ${SPACK_USER_PREFIX} environment variable. The variable defaults to ${HOME} and it is advisable to set it to an other value, especially if you’re installing on different Adastra partitions. While we reuse CINES’ support team build cache, we do not have the Spack PreConf using the CINES’ software stacks as upstream.
Pro and cons : You will be restricted to using a fixed version of Spack, allowing you to benefit from installations performed by the CINES teams, including access to the mirror and binary cache, resulting in inherently shorter installation times.
Getting started
To install software with Spack, perform the following steps:
Initialize Spack.
We provide the following Spack PreConf module that you can use to introduce CINES’ Spack configuration to your environment.
Targeted partition |
Spack PreConf module |
Configured toolchains |
|---|---|---|
All |
|
CCE, GCC, AMD LLVM |
Note
We provide multiple compilers per Spack PreConf. This differs from the environment mentioned in the previous section.
Note
An extensible software stack module (or environment) or Spack PreConf will implicitly load modules; a behavior you can observe using module show spack-user-<version>. Running module purge before loading an extensible software stack module is recommended.
$ module purge
$ export SPACK_USER_PREFIX="${WORKDIR}/spack-install-<version>"
$ module load spack-user-<version>
mkdir: created directory '<path_to_spack_configuration>/spack-install-4.0.0'
For more information on how spack user works on Adastra, please consult documentation https://dci.dci-gitlab.cines.fr/webextranet/user_support/index.html#installing-software-using-spack-on-adastra
SPACK_USER_PREFIX=<path_to_spack_configuration>/spack-install-4.0.0
SPACK_USER_CONFIG_PATH=<path_to_spack_configuration>/spack-install-4.0.0/configuration
Note
We recommend that you set ${SPACK_USER_PREFIX} in an environment script or in your .bash_profile to avoid having to set it every time you want to use Spack.
Warning
Ensure that the ${SPACK_USER_PREFIX} variable is not too long (ambiguous) or you may get Spack errors of this kind: Error: Failed to install XXX due to SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (YYY) exceeds limit (127).
When loading the Spack PreConf module, a fully preconfigured Spack instance will be placed where the ${SPACK_USER_PREFIX} points to.
Consult the information that Spack has on the product you wish to install and in particular on the configuration options that Spack calls variant:
$ spack info kokkos
Let’s assume that we intend to setup this product using the ROCm toolchain. Upon reviewing the product information, it follows that the installation command could be:
$ spack install kokkos@4.2.00%rocmcc@5.7.1~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic+rocm+serial+shared~sycl~tests~threads~tuning~wrapper amdgpu_target==gfx90a build_system=cmake build_type=Release cxxstd=17 generator=ninja patches=145619e arch=linux-rhel8-zen3
The +rocm flag enables AMD GPU support. Additionally, we need to specify the GPU type: amdgpu_target==gfx90a (note the double equal signs, which have the special meaning of propagating the GPU target to all dependencies). While we specify the rocmcc@5.7.1 compiler, it’s hipcc of the ROCm toolchain that will be used for this build. The arch=linux-rhel8-zen3 tells Spack to target the Zen3 CPU architecture (say, enable AVX2).
Warning
You must ensure that the arch= is correctly specified. If you want to target the MI250X nodes, you should use arch=linux-rhel8-zen3, else you should use arch=linux-rhel8-zen4.
It is advisable to review the dependencies that Spack intends to install:
$ spack spec kokkos%rocmcc amdgpu_target=gfx90a arch=linuInput spec
--------------------------------
- kokkos%rocmcc amdgpu_target=gfx90a arch=linux-rhel8-zen3
Concretized
--------------------------------
- kokkos@4.2.00%rocmcc@5.7.1~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_docm+serial+shared~sycl~tests~threads~tuning~wrapper amdgpu_target=gfx90a build_system=cmake build_type=Release c
- ^cmake@3.27.7%rocmcc@5.7.1~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel8
- ^curl@8.4.0%rocmcc@5.7.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools
- ^mbedtls@3.3.0%rocmcc@5.7.1+pic build_system=makefile build_type=Release libs=static arch=linux
- ^nghttp2@1.57.0%rocmcc@5.7.1 build_system=autotools arch=linux-rhel8-zen3
- ^pkgconf@1.9.5%rocmcc@5.7.1 build_system=autotools arch=linux-rhel8-zen3
- ^ncurses@6.4%rocmcc@5.7.1~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-zen3
- ^zlib-ng@2.1.4%rocmcc@5.7.1+compat+opt build_system=autotools arch=linux-rhel8-zen3
- ^gmake@4.4.1%rocmcc@5.7.1~guile build_system=generic arch=linux-rhel8-zen3
[e] ^hip@5.7.1%rocmcc@5.7.1~cuda+rocm build_system=cmake build_type=Release generator=make patches=3f783ae,
[e] ^hsa-rocr-dev@5.7.1%rocmcc@5.7.1+image+shared build_system=cmake build_type=Release generator=make arch
[e] ^llvm-amdgpu@5.7.1%rocmcc@5.7.1~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake
In your Spack instance, packages you installed will be denoted by a [+] in the first column, while packages already provided by Spack and cached from previous builds will display [^]. A - indicates that Spack did not find the package and will proceed to build it. A [e] indicates that the product is external to Spack, typical a system product, in our case, it is looking for ROCm libraries.
Once you’ve reviewed Spack’s plan and are satisfied with it, proceed to install the packages.
Warning
We are experiencing some signature inconsistency, so to workaround issues such as Error: Failed to install XXX due to NoVerifyException, specify --no-check-signature in the spack install command. Alternatively, you can use --no-cache to bypass the CINES’ build cache but you will experience longer build time.
$ spack install kokkos@4.2.00%rocmcc@5.7.1~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic+rocm+serial+shared~sycl~tests~threads~tuning~wrapper amdgpu_target==gfx90a build_system=cmake build_type=Release cxxstd=17 generator=ninja patches=145619e arch=linux-rhel8-zen3
...
Stage: 0.18s. Cmake: 4.19s. Build: 6.30s. Install: 3.03s. Post-install: 1.03s. Total: 14.99s
[+] <path_to_spack_configuration>/spack-install-4.0.0/linux-rhel8-zen3/rocmcc-5.7.1/kokkos-4.2.00-52g6
The last line indicates the location of the software installation on the disk.
After successful installation of the product, you may want to ask Spack to forcefully generate modules.
$ spack module tcl refresh --delete-tree --yes-to-all
Then, you can find your product in your module environment:
$ module available kokkos/4.2.00
---- <path_to_spack_configuration>/spack-install-4.0.0/modules/tcl/linux-rhel8-zen3 ----
rocmcc/5.7.1/zen3/kokkos/4.2.00
If you wish to utilize the modules generated by this Spack instance without loading the associated modules (spack-[GENOA|MI250]-<version>), you can simply add the path of your modules to the ${MODULEPATH} (they should be under ${SPACK_USER_PREFIX}/modules/tcl/linux-rhel8-[zen3|zen4]).
$ # For MI250:
$ module use "${SPACK_USER_PREFIX}/modules/tcl/linux-rhel8-zen3"
$ # For GENOA:
$ module use "${SPACK_USER_PREFIX}/modules/tcl/linux-rhel8-zen4"
If a Spack install fails
If you encounter a failed Spack installation, consider examining the error message displayed. Additionally, Spack may direct you to an installation log for the specific product, which can be found in the same directory. The complete build directory is also available in /tmp. Examining the configure or cmake output logs may sometimes yield fruitful results. Finally, the spack -d install -v command may prove useful.
Here are some tips if the installation fails:
Modify the software version;
Modify the compiler : If the compiler is not specified, it takes by default the cce compiler (the recommended compiler for installing our software on Cray systems). If the installation fails, try with another compiler;
Disable variants that seem to cause problems;
Modify the dependencies used to build the target product (also called package);
Edit
package.py: take a look at thepackage.pyof the package that crashes:spack edit package_name. Unfortunately the process is not always simple since the package repository is situated in a read-only location. In these situations, it is necessary to clone your own Spack instance and configure it using our configuration files.
Assistance is available: you may refer to the official Spack documentation, open a ticket at the CINES help desk, or seek help from the Spack community via the Spack Slack.
Limitations
Spack evolves quickly, the tagged Spack version (i.e.: 0.22) are always lacking compared to the develop Spack branch;
you may have to specify
--no-check-signature.Spack is installed on login nodes that do not have access to all the internet, only certain package versions are available to users. These versions are those previously downloaded by the CINES support team, they can be found in the local repository of archive packages, called mirror (see mirrors documentation for more information). The CINES mirror is located in this path:
/opt/software/gaia/local_repo(you can search in it if the package you want to install is already there). To circumvent this limitation you can download yourself the archive packages (tarballs) then copy it in a folder created for this purpose, overload the configuration filemirrors.yamlby adding a mirror pointing to your archive packages folder. Here is an example of this protocol:
Download the tarballs from your workstation (spack expect a file with a tar.gz extension files), copy it to Adastra and connect to Adastra :
$ scp packages.tar.gz <login>@adastra.cines.fr:/path/to/your/home
$ ssh <login>@adastra.cines.fr
After initializing spack, copy your tarballs into your archive package folder, for example :
$ export SPACK_USER_PREFIX="${HOME}/spack-install-<version>"
$ module load spack-user-<version>
$ mkdir -p -- "${SPACK_USER_PREFIX}/mirror/<package_name>" # change <package_name> to its real name, e.g. “cmake”.
$ mv "${HOME}/packages.tar.gz" "${SPACK_USER_PREFIX}/mirror/<package_name>/."
Then overload the defaults configuration file mirrors.yaml by adding a mirror pointing to your archive packages folder :
Next, update the default mirrors.yaml configuration file by adding a mirror that points to your archive packages folder using the spack mirror add command. Ensure that the path to the mirror is hardcoded, as Spack will not automatically resolve variables in mirrors.yaml. By using the spack mirror add command, Spack evaluates the provided path directly, effectively avoiding this issue.
$ spack mirror add local_filesystem_user $SPACK_USER_PREFIX/mirror
$ grep -v ^" #" ${SPACK_USER_PREFIX}/configuration/mirrors.yaml
mirrors:
local_filesystem_user: /lus/home/path/to/your/login/hardcoded_spack_user_prefix/mirror
local_filesystem:
url: file:///opt/software/gaia/local_repo
binary: true
source: true
signed: false # disable signing and verification
spack-public: https://mirror.spack.io
Recommendations on correct usage
Below, we give an example script that you can use as is (you’ll probably want to change the VERSION environment variable as time evolves).
#!/bin/bash
SCRIPT_DIRECTORY="$(dirname -- "$(readlink -f -- "${BASH_SOURCE:-${0}}")")"
# VERSION="spack-user-3.2.0"
VERSION="spack-user-4.0.0"
export SPACK_USER_PREFIX="${SCRIPT_DIRECTORY}/Configuration.${VERSION}"
export SPACK_USER_CACHE_PATH="${SPACK_USER_PREFIX}/cache"
module purge
module load develop
module load "${VERSION}"
module list
spack config --scope user update --yes-to-all packages
spack config --scope user update --yes-to-all config
spack config --scope user add 'packages:all:permissions:read:world'
spack config --scope user add 'packages:all:permissions:write:group'
spack config --scope user add 'concretizer:unify:true'
spack config --scope user add 'modules:default:tcl:hash_length:4'
spack debug report
We recommend that you write this script into your WORKDIR, you can source it and install an application like below, taking the example of Charmpp:
$ spack install charmpp@7.0.0%gcc@13.2.1.generic~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target='charm++' build_system=generic pmi=none arch=linux-rhel8-zen3
For some software, the source is not publicly available. In such case, you should place it in a directory, go to this directory and execute spack install from it. If you have more troubles, contact svp@cines.fr