CINES software stack
Introduction
Before going into more technical subjects, know that CINES uses Spack to provide precompiled softwares which we shall name Spack products. We recommend that you first search for an already CINES provided software before trying to install it yourself. To look for such preinstalled software, search the modules using module spider or browse the catalog.
A note on dependencies
Spack tries to solve the problem of satisfying the somewhat esoteric and hpc specific dependencies and does a reasonably good job at it. It is good enough that it is considered viable and used by many sites. That said, note that dependencies are a problem and that even if Spack mitigates this to a certain extent, the resolution of a dependency is still a problem. You should probably not add new dependencies lightly. To that, we should add:
Are all your dependencies built and tested in CI on all target platform ? Probably not, so you should be even more careful with dependencies?
It is almost always costlier to fix something, than to start from scratch.
Basic tools and libraries
CINES offers a basic software suite primarily deployed using the Spack package manager and GCC toolchain. This suite includes commonly used tools and libraries with no MPI dependencies (with a few exceptions), ensuring there are no risks of ABI (Application Binary Interface) incompatibilities or symbol conflicts during linking.
You can safely use these modules without risk of conflicts with other modules, either to link your applications with the available libraries or to use the underlying tools.
Running module available could give you:
--------------------------------------------------------- /opt/software/gaia/prod/latest/Base ---------------------------------------------------------
anaconda3/2023.09-0 gdal/3.8.5 libsndfile/1.0.28 ncview/2.1.9-mpi r/4.4.0
blitz/1.0.2 gdb/14.1 libtirpc/1.3.3 ninja/1.11.1 rust/1.78.0
bzip2/1.0.8 gnuplot/6.0.0 libxml2/2.10.3 nsimd/3.0.1 sox/14.4.2
cfitsio/4.3.0 gsl/2.7.1 libxsmm/1.17 papi/7.1.0 swig/4.1.1
cmake/3.27.9 hpctoolkit/2023.08.1-mpi metis/5.1.0-int64 paraview/5.13.0-osmesa zlib-ng/2.1.4
easybuild/4.7.0 hpcviewer/2024.02 metis/5.1.0 (D) paraview/5.13.0 (D) zlib/1.3.1
eigen/3.4.0 hwloc/2.9.1-rocm-gpu mmg/5.7.1 poppler/23.04.0
ffmpeg/6.1.1 jasper/3.0.3 mpifileutils/0.11.1-mpi python/3.12.1
firefox/134.0.2 julia/1.10.0 nco/5.1.9-mpi qt/5.15.11
Extensible software stacks
CINES offers extensible software stacks in the form of modules named according to the format <compiler>-[GPU|CPU]-<version>. Each extensible software stack groups together a set of software packages targeting a specific architecture and built using a specific toolchain. CINES will provide software stacks adapted to each Adastra hardware partition. Each software stack version is based on a specific version of the Cray programming environment (cpe/XX.YY).
The <compiler>-[GPU|CPU]-<version> modules can be considered as architecture- and compiler-specific environments. The <version> number represents the production version of the software stack. CINES will endeavor to respect semantic versioning concepts.
Adastra partition |
Extensible software stacks |
Toolchain used |
|---|---|---|
MI250 (GPU) |
CCE-GPU-<version> |
CCE
|
GCC-GPU-<version> |
GCC
|
|
GENOA (CPU) |
CCE-CPU-<version> |
CCE
|
GCC-CPU-<version> |
GCC
|
|
|
TBD |
TBD |
In addition to these extensible software stacks mapped to the partitions of Adastra, we provide what we call Spack user or Spack pre-configured environment (PreConf). It enables the user to install their own softwares by reusing the CINES’ Spack configuration and binary cache. Spack PreConf provides Spack support for a given partition and a set of compiler. The syntax is: spack-user-<version>. More information on how you can reuse the CINES’ Spack configuration is given in the Spack-user : Using pre-configured Spack installation on Adastra section.
Running module available could give you:
--------------------------------- /opt/software/gaia/prod/latest/Core ---------------------------------
CCE-CPU-3.2.0 CCE-GPU-3.2.0 GCC-CPU-3.2.0 GCC-GPU-3.2.0 develop spack-user-3.2.0
The develop extensible module exposes the software stacks being currently developed (release candidate). Running module load develop and then module av could give you:
--------------------------------- /opt/software/gaia/dev/latest/Core ----------------------------------
CCE-CPU-4.0.0 CCE-GPU-4.0.0 GCC-CPU-4.0.0 GCC-GPU-4.0.0 spack-user-4.0.0
Warning
This category includes software versions under development and testing before their deployment in production. Use at your own risk. Package stability is not guaranteed, and packages may be removed or modified when redeploying develop packages/modules.
A production release will take place every 3 to 6 months. Each such release may introduce new versions for the components of the stack (say a CCE or GCC version bump). As the machine evolves, more extensible software stacks (based on a CrayPE release) and their respective products will be added, some will be deprecated. We do not guarantee an extensible software stacks will be produced for each CrayPE release. CINES is likely to release a new extensible software stacks at the start of each DARI call. We guarantee that the CINES’ software stack will be available for at least 2 CrayPE releases before being removed from the system. This should give the user something like 9 months to 1 year of support per release (at least).
When the software stack is put into production, the old modules are moved to a space called deprecated and are still available, as shown below using module available:
------------------------------------ /opt/software/gaia/deprecated ------------------------------------
CCE-CPU-3.1.0 CCE-GPU-3.1.0 GCC-CPU-3.1.0 GCC-GPU-3.1.0 archive
Finally, deprecated modules are moved to the .archive directory and become available only after the module load archive command is executed. These archives are kept for a long time, our only support limit would be the operating system itself not being able to run the binary. In this case, ask svp@cines.fr for a reinstallation of the product old product version.
Using the CINES software stack
To see the softwares made available by an environment, you can simply load the extensible module corresponding to an environment. That said the recommended way is the following; supposing you want to use the GROMACS software, you would do:
$ module spider gromacs
----------------------------------------------------------------------------
gromacs:
----------------------------------------------------------------------------
Versions:
gromacs/2021.4-mpi-omp-plumed-ocl-python3
gromacs/2022.3-mpi-omp-plumed-python3
gromacs/2022.5-mpi-omp-plumed-python3
gromacs/2023-mpi-omp-plumed-ocl-python3
gromacs/2023.3-cp2k-omp-plumed-mpi
gromacs/2023.3-mpi-omp-plumed-python3
gromacs/2024.3-cp2k-omp-mpi
Here, module shows multiple GROMACS versions (2021.4, 2022.3, 2022.5, 2023, …) but with different variants for the 2023 version. Lets use gromacs/2023-mpi-omp-plumed-ocl-python3. To find which programming environment provides the software package we do:
$ module spider gromacs/2023-mpi-omp-plumed-ocl-python3
----------------------------------------------------------------------------
gromacs: gromacs/2023-mpi-omp-plumed-ocl-python3
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "gromacs/2023-mpi-omp-plumed-ocl-python3" module is available to load.
CCE-GPU-2.0.0
CPE-23.02-cce-15.0.1-GPU-softs
This tells us that gromacs/2023-mpi-omp-plumed-ocl-python3 is available, under the CCE-GPU-2.0.0 and CPE-23.02-cce-15.0.1-GPU-softs extensible module. Latest stable version of the extensible module/environment (i.e. production) should be preferably used.
In your scripts, the procedure above reduces to the following commands:
$ module purge
$ module load CCE-GPU-2.0.0
$ module load gromacs/2023-mpi-omp-plumed-ocl-python3
Advanced module mixing
As an example, let’s suppose you’ve located (using the module spider) a library from the CINES software stack. Let’s assume that this library has been built in an environment (extensible module of the form <compiler>-[GPU|CPU]-<version>) using the Cray compiler, but you want to use the GCC toolchain (PrgEnv-gnu). Conventional wisdom says you shouldn’t mix the two.
In practice, if the interfaces are properly defined, as it is in the library used in this example, and if the library does not have toolchain dependencies such as an OpenMP runtime (which you absolutely do not want to mix), then you should be able to have interoperability of the library across toolchains.
Note
Check the dependencies on shared libraries using readelf -d or ldd. It is trickier on static libraries.