RAJA Build Configurations¶

RAJA is tested with a wide range of compilers in a variety of build configurations. Automated continuous integration (CI) testing employed by the project is described in Continuous Integration (CI) Testing. Building and testing manually is also important for day-to-day development. We maintain two ways to build and test configurations in a reproducible manner:

Build scripts. The RAJA source repository contains a collection of simple shell scripts that we use to generate build configurations for a variety of platforms, such as Livermore Computing (LC) systems, MacOS, and Linux environments.

Generated host-config files. The RAJA repository includes a mechanism to generate host-config files (i.e., CMake cache files) that are identical to those generated by Spack for use in our GitLab CI testing.

The configurations specify compiler versions and options, build targets (Release, Debug, etc.), RAJA features to enable (OpenMP, CUDA, HIP, etc.), and paths to required tool chains, such as CUDA, ROCm, etc. We describe them briefly in the following sections.

RAJA Build Scripts¶

The build scripts mentioned above reside in subdirectories of the RAJA/scripts directory. These include:

RAJA/scripts/lc-builds for configurations on LC machines

RAJA/scripts/macos-builds for MacOS configurations

RAJA/scripts/ubuntu-builds for other Linux configurations

To create a RAJA build space using one of the scripts, execute the script from the top-level RAJA directory, passing required arguments such as compiler version, target compute architecture for GPU builds, etc. For example,

$ ./scripts/lc-builds/toss3_clang.sh 10.0.1

will create a build space for the clang 10.0.1 compiler on a TOSS3 LC machine.

$ ./scripts/lc-builds/blueos_nvcc_gcc.sh 10.2.89 70 8.3.1

will create a build space for a blueos LC machine using GNU 8.3.1 as the host compiler and nvcc 10.2.89 as the GPU device compiler targeting CUDA compute architecture ‘sm_70’.

Note

Each RAJA build script for an LC machine does the following:

Create a build space directory named for the specified build configuration in the top-level RAJA directory

Load a CMake module version appropriate for the configuration

Invoke CMake with appropriate arguments to create the specified build configuration. These arguments typically contain full paths to the compiler and any toolchains needed, and a host-config file that contains compiler flags.

Host-config files, also known and CMake cache files that are passed to CMake are maintained in subdirectories of the RAJA/host-configs directory.

After running a build script, one then goes into that directory and runs ‘make’ to build RAJA and its tests, example codes, etc. For example,

$ ./scripts/lc-builds/blueos_nvcc_gcc.sh 10.2.89 sm_70 8.3.1
$ cd build_lc_blueos-nvcc10.2.89-70-gcc8.3.1
$ make -j
$ make test

Spack-Generated Host-Config Files¶

As described in GitLab CI Testing Dependencies (specific to LC CZ), the RAJA repository contains two submodules Uberenv and RADIUSS Spack Configs that work together with Spack to generate host-config files. Uberenv and RADIUSS Spack Configs are projects in the GitHub LLNL organization and contain utilities shared and maintained by various LLNL open-source projects. Uberenv contains a Python script that is used to drive Spack to generate a host-config file. The generated file can be passed to CMake using the ‘-C’ option to create a build configuration. Spack specs defining compiler and toolchain configurations, and Spack packages for RAJA and other projects are maintained in the RADIUSS Spack Configs project.

Additional information about this process is available in the RADIUSS Uberenv Guide.

Important

The process of generating a host-config file and using it to test a build of RAJA is very useful since the same procedure is done in our GitLab CI testing. See Continuous Integration (CI) Testing for more information about our GitLab testing process.

Generating a RAJA host-config file¶

This section describes the host-config file generation process for RAJA.

Platform configurations¶

Compiler configurations for Livermore machines are contained in sub-directories of the RAJA/scripts/radiuss-spack-configs submodule directory:

$ ls -c1 ./scripts/radiuss-spack-configs
toss_4_x86_64_ib_cray
toss_4_x86_64_ib
toss_3_x86_64_ib
packages
modules.yaml
darwin
blueos_3_ppc64le_ib
config.yaml
blueos_3_ppc64le_ib_p9
...

Each subdirectory named for a system type contains a compilers.yaml file and packages.yaml file with compiler and toolchain information associated with the system.

Generating a host-config file¶

To generate a host-config file for a desired configuration, run the uberenv.py python script from the top-level RAJA directory.

Important

Do not run the uberenv.py script, which invokes Spack in your LC home directory. Running Spack in your home directory may cause you to exceed your LC disk space quota.

For example,

$ python3 ./scripts/uberenv/uberenv.py --spec="%gcc@8.1.0"

will generate a basic configuration for building RAJA with the GNU 8.1.0 compiler, while

$ python3 ./scripts/uberenv/uberenv.py --spec="+openmp +tests +desul %clang@10.0.1 cxxflags==--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags==--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1"

will generate a configuration for building with the clang 10.0.1 compiler using the GNU 8.3.1 toolchain (for C++ standard library headers, etc.) and with RAJA OpenMP back-end, tests, and desul atomics enabled.

Running a command like one of these will generate a host-config file in the top-level RAJA directory. The name of the file will contain the name of the machine and OS to which it applies, as well as the compiler and version. For example, running the first example command above on the LC machine quartz will generate a host-config file named something like this:

hc-quartz-toss_3_x86_64_ib-gcc@8.1.0-fjcjwd6ec3uen5rh6msdqujydsj74ubf.cmake

Building RAJA with a generated host-config file¶

To build RAJA with a generated host-config file, create a build directory and run CMake in it by passing a host-config file to CMake using the ‘-C’ option. Then, run ‘make’ to build RAJA. To ensure the build was successful, you may want to run the RAJA tests. For example,

$ pwd
path-to-raja/RAJA
$ mkdir <build dir name> && cd <build dir name>
$ cmake -C <path to>/<host-config>.cmake ..
$ make -j
$ ctest --output-on-failure -T test

You may also run the RAJA tests with the command

$ make test

as an alternative to the ‘ctest’ command used above.

It is also possible to use the configuration with the RAJA GitLab CI script outside of the GitLab environment:

$ HOST_CONFIG=<path_to>/<host-config>.cmake ./scripts/gitlab/build_and_test.sh

Important

Before adding a new configuration to RAJA GitLab CI, it is prudent to manually perform the process of generating the corresponding host-config file and testing it by running it though our GitLab testing script as described here.

MacOS¶

In RAJA, the Spack configuration for MacOS contains the default compiler corresponding to the OS version in the compilers.yaml file in the RAJA/scripts/radiuss-spack-configs/darwin/ directory, and a commented section to illustrate how to add CMake as an external package in the packages.yaml in the same directory. You may also install CMake with Homebrew, for example, and follow the process outlined above after it is installed.

Reproducing Docker Builds Locally¶

RAJA uses Docker container images that it shares with other LLNL GitHub projects for Azure CI testing (see Azure Pipelines CI for more information). We use Azure Pipelines for Linux, Windows, and MacOS builds. Typically, we do RAJA testing for Linux on Azure with compilers, or compiler versions, that are not available on LLNL LC machines.

You can reproduce Azure builds locally for testing with the following steps if you have Docker installed.

Run the command to build a local Docker image:
$ DOCKER_BUILDKIT=1 docker build --target <compiler>  --no-cache
Here, <compiler> is one of the names following AS in the RAJA Dockerfile.
To get dropped into a terminal in the Docker image, run the following:
$ docker run -it ghcr.io/rse-ops/<os image>:<compiler> /bin/bash
Here, <os image>:<compiler> is the OS image and compiler image you want (see the aforementioned Dockerfile).

Then, you can build, run tests, edit files, etc. in the Docker image. Note that the docker command has a -v argument that you can use to mount a local directory in the image. For example

& docker -v pwd:/opt/RAJA

will mount your current local directory as /opt/RAJA in the image.