RAJA Build Configurations¶
RAJA is tested with a wide range of compilers in a variety of build configurations. Automated continuous integration (CI) testing employed by the project is described in Continuous Integration (CI) Testing. Building and testing manually is also important for day-to-day development. We maintain two ways to build and test configurations in a reproducible manner:
- Build scripts. The RAJA source repository contains a collection of simple shell scripts that we use to generate build configurations for a variety of platforms, such as Livermore Computing (LC) systems, MacOS, and Linux environments.
- Generated host-config files. The RAJA repository includes a mechanism to generate host-config files (i.e., CMake cache files) that are identical to those generated by Spack for use in our GitLab CI testing.
The configurations specify compiler versions and options, build targets (Release, Debug, etc.), RAJA features to enable (OpenMP, CUDA, HIP, etc.), and paths to required tool chains, such as CUDA, ROCm, etc. We describe them briefly in the following sections.
RAJA Build Scripts¶
The build scripts mentioned above reside in subdirectories of the RAJA/scripts directory. These include:
RAJA/scripts/lc-buildsfor configurations on LC machines
RAJA/scripts/macos-buildsfor MacOS configurations
RAJA/scripts/ubuntu-buildsfor other Linux configurations
To create a RAJA build space using one of the scripts, execute the script from the top-level RAJA directory, passing required arguments such as compiler version, target compute architecture for GPU builds, etc. For example,
$ ./scripts/lc-builds/toss3_clang.sh 10.0.1
will create a build space for the clang 10.0.1 compiler on a TOSS3 LC machine.
$ ./scripts/lc-builds/blueos_nvcc_gcc.sh 10.2.89 70 8.3.1
will create a build space for a blueos LC machine using GNU 8.3.1 as the host compiler and nvcc 10.2.89 as the GPU device compiler targeting CUDA compute architecture ‘sm_70’.
Each RAJA build script for an LC machine does the following:
- Create a build space directory named for the specified build configuration in the top-level RAJA directory
- Load a CMake module version appropriate for the configuration
- Invoke CMake with appropriate arguments to create the specified build configuration. These arguments typically contain full paths to the compiler and any toolchains needed, and a host-config file that contains compiler flags.
Host-config files, also known and CMake cache files that are passed to CMake are maintained in subdirectories of the RAJA/host-configs directory.
After running a build script, one then goes into that directory and runs ‘make’ to build RAJA and its tests, example codes, etc. For example,
$ ./scripts/lc-builds/blueos_nvcc_gcc.sh 10.2.89 sm_70 8.3.1 $ cd build_lc_blueos-nvcc10.2.89-70-gcc8.3.1 $ make -j $ make test
Spack-Generated Host-Config Files¶
As described in GitLab CI Testing Dependencies (specific to LC CZ), the RAJA repository contains two submodules Uberenv and RADIUSS Spack Configs that work together with Spack to generate host-config files. Uberenv and RADIUSS Spack Configs are projects in the GitHub LLNL organization and contain utilities shared and maintained by various LLNL open-source projects. Uberenv contains a Python script that is used to drive Spack to generate a host-config file. The generated file can be passed to CMake using the ‘-C’ option to create a build configuration. Spack specs defining compiler and toolchain configurations, and Spack packages for RAJA and other projects are maintained in the RADIUSS Spack Configs project.
Additional information about this process is available in the RADIUSS Uberenv Guide.
The process of generating a host-config file and using it to test a build of RAJA is very useful since the same procedure is done in our GitLab CI testing. See Continuous Integration (CI) Testing for more information about our GitLab testing process.
Generating a RAJA host-config file¶
This section describes the host-config file generation process for RAJA.
Compiler configurations for Livermore machines are contained
in sub-directories of the
$ ls -c1 ./scripts/radiuss-spack-configs toss_4_x86_64_ib_cray toss_4_x86_64_ib toss_3_x86_64_ib packages modules.yaml darwin blueos_3_ppc64le_ib config.yaml blueos_3_ppc64le_ib_p9 ...
Each subdirectory named for a system type contains a
packages.yaml file with compiler and toolchain information associated
with the system.
Generating a host-config file¶
To generate a host-config file for a desired configuration, run the
uberenv.py python script from the top-level RAJA directory. For example,
$ python3 ./scripts/uberenv/uberenv.py --spec="%firstname.lastname@example.org"
will generate a basic configuration for building RAJA with the GNU 8.1.0 compiler, while
$ python3 ./scripts/uberenv/uberenv.py --spec="+openmp +tests +desul %email@example.com cxxflags==--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1 cflags==--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1"
will generate a configuration for building with the clang 10.0.1 compiler using the GNU 8.3.1 toolchain (for C++ standard library headers, etc.) and with RAJA OpenMP back-end, tests, and desul atomics enabled.
Running a command like one of these will generate a host-config file in the top-level RAJA directory. The name of the file will contain the name of the machine and OS to which it applies, as well as the compiler and version. For example, running the first example command above on the LC machine quartz will generate a host-config file named something like this:
Building RAJA with a generated host-config file¶
To build RAJA with a generated host-config file, create a build directory and run CMake in it by passing a host-config file to CMake using the ‘-C’ option. Then, run ‘make’ to build RAJA. To ensure the build was successful, you may want to run the RAJA tests. For example,
$ pwd path-to-raja/RAJA $ mkdir <build dir name> && cd <build dir name> $ cmake -C <path to>/<host-config>.cmake .. $ make -j $ ctest --output-on-failure -T test
You may also run the RAJA tests with the command
$ make test
as an alternative to the ‘ctest’ command used above.
It is also possible to use the configuration with the RAJA GitLab CI script outside of the GitLab environment:
$ HOST_CONFIG=<path_to>/<host-config>.cmake ./scripts/gitlab/build_and_test.sh
Before adding a new configuration to RAJA GitLab CI, it is prudent to manually perform the process of generating the corresponding host-config file and testing it by running it though our GitLab testing script as described here.
In RAJA, the Spack configuration for MacOS contains the default compiler
corresponding to the OS version in the
compilers.yaml file in the
RAJA/scripts/radiuss-spack-configs/darwin/ directory, and a commented
section to illustrate how to add CMake as an external package in the
packages.yaml in the same directory. You may also install CMake
with Homebrew, for example, and follow the process
outlined above after it is installed.
Reproducing Docker Builds Locally¶
RAJA uses Docker container images that it shares with other LLNL GitHub projects for Azure CI testing (see Azure Pipelines CI for more information). We use Azure Pipelines for Linux, Windows, and MacOS builds. Typically, we do RAJA testing for Linux on Azure with compilers, or compiler versions, that are not available on LLNL LC machines.
You can reproduce Azure builds locally for testing with the following steps if you have Docker installed.
Run the command to build a local Docker image:$ DOCKER_BUILDKIT=1 docker build --target <compiler> --no-cache
<compiler>is one of the names following
ASin the RAJA Dockerfile.
To get dropped into a terminal in the Docker image, run the following:$ docker run -it ghcr.io/rse-ops/<os image>:<compiler> /bin/bash
<os image>:<compiler>is the OS image and compiler image you want (see the aforementioned Dockerfile).
Then, you can build, run tests, edit files, etc. in the Docker image. Note that
the docker command has a
-v argument that you can use to mount a local
directory in the image. For example
& docker -v pwd:/opt/RAJA
will mount your current local directory as
/opt/RAJA in the image.