Continuous Integration (CI) Testing¶
Important
- All CI test checks must pass before a pull request can be merged.
- The status (pass/fail and run) for all checks can be viewed by clicking the appropriate link in the checks section of a GitHub pull request.
The CI tools used by the RAJA project, and which integrate with GitHub are:
- Azure Pipelines runs builds and tests for Linux, Windows, and MacOS environments using recent versions of various compilers. While we do GPU builds for CUDA, HIP, and SYCL on Azure, RAJA tests are only run for CPU-only pipelines. See the RAJA Azure DevOps project to learn more about our testing there.
- Gitlab instances in the Livermore Computing (LC) Center runs builds and tests in LC resource and compiler environments important to many RAJA user applications. Execution of RAJA CI pipelines on LC Gitlab resources has restrictions described below. If you have access to LC resources, you can access additional information about LC GitLab CI
The tools automatically run RAJA builds and tests when a PR is created and when changes are pushed to a PR branch.
The following sections describe basic elements of the operation of the CI tools.
Gitlab CI¶
The Gitlab CI instance used by the RAJA project lives in the Livermore Computing (LC) Collaboration Zone (CZ). It runs builds and tests in LC resource and compiler environments important to RAJA user applications at LLNL.
Constraints¶
Running Gitlab CI on Livermore Computing (LC) resources is constrained by LC security policies. The policies require that all members of a GitHub project be members of the LLNL GitHub organization and have two-factor authentication enabled on their GitHub accounts to automatically mirror a GitHub repo and trigger Gitlab CI functionality from GitHub. For compliant LLNL GitHub projects, auto-mirroring of the GitHub repo on LC Gitlab is done when changes are pushed to PRs for branches in the RAJA repo, but not for PRs for a branch on a fork of the repo. An alternative procedure we use to handle this is described in Contributing to RAJA. If you have access to LC resources, you can learn more about LC Gitlab mirroring.
Gitlab CI (LC) Testing Workflow¶
The figure below shows the high-level steps in the RAJA Gitlab CI testing process. The main steps, which we will discuss in more detail later, are:
- A mirror of the RAJA GitHub repo in the RAJA Gitlab project is updated whenever the RAJA
develop
ormain
branches are changed as well as when any PR branch in the RAJA GitHub project is changed.- Gitlab launches CI test pipelines. While running, the execution and pass/fail status may be viewed and monitored in the Gitlab CI GUI.
- For each resource and compiler combination, Spack is used to generate a build configuration in the form of a CMake cache file, or host-config file.
- A host-config file is passed to CMake, which configures a RAJA build space. Then, RAJA and its tests are compiled.
- Next, the RAJA tests are run.
- When a test pipeline completes, final results are reported in Gitlab.
In the next section, we will describe the roles that specific files in the RAJA repo play in defining these steps.

The main steps in the RAJA Gitlab CI testing workflow are shown in the figure. This process is triggered when a developer makes a PR on the GitHub project or whenever changes are pushed to the source branch of a PR.
Gitlab CI (LC) Testing Files¶
The following figure shows directories and files in the RAJA repo that support LC Gitlab CI testing. Files with names in blue are specific to RAJA and are maintained by the RAJA team. Directories and files with names in red are in Git submodules, shared and maintained with other projects.

The figure shows directories and files in the RAJA repo that support Gitlab CI testing. Files in blue are specific to RAJA and owned by the RAJA team. Red directories and files are part of Git submodules shared with other projects.
In the following sections, we discuss how these files are used in the steps in the RAJA Gitlab CI testing process summarized above.
Launching CI pipelines (step 2)¶
In step 2 of the diagram above, Gitlab launches RAJA test pipelines. The RAJA/.gitlab-ci.yml file contains high-level testing information, such as stages (resource allocation, build-and-test, and resource deallocation) and locations of files that define which jobs will run in each pipeline. For example, these items appear in the file as:
stages:
- r_allocate_resources
- r_build_and_test
- r_release_resources
- l_build_and_test
- c_build_and_test
- multi_project
and:
include:
- local: .gitlab/ruby-templates.yml
- local: .gitlab/ruby-jobs.yml
- local: .gitlab/lassen-templates.yml
- local: .gitlab/lassen-jobs.yml
- local: .gitlab/corona-templates.yml
- local: .gitlab/corona-jobs.yml
In the stages
section above, prefixes ‘r_’, ‘l_’, and ‘c_’ refer to
resources in the LC on which tests are run. Specifically, the machines ‘ruby’,
‘lassen’, and ‘corona’, respectively. Jobs that will run in pipeline(s) on each
resource are defined in the files listed in the include
section above.
Note that the stage labels above appear on each Gitlab CI run web page as the
title of a column containing other information about what is run in that stage,
such as build and test jobs.
The RAJA/.gitlab
directory contains a templates and jobs file for each LC resource on which
test pipelines will be run. The <resource>-templates.yml
files contain
information that is common across jobs that run on the corresponding resource,
such as commands and scripts that are run for stages identified in the
RAJA/.gitlab-ci.yml
file. For example, the
RAJA/.gitlab/ruby-templates.yml
file contains a section:
allocate_resources (on ruby):
variables:
GIT_STRATEGY: none
extends: .on_ruby
stage: r_allocate_resources
script:
- salloc -N 1 -p pdebug -t 45 --no-shell --job-name=${ALLOC_NAME}
which contains the resource allocation command associated with the
r_allocate_resources
stage identifier on ‘ruby’. Analogous stages are
defined similarly in other RAJA/.gitlab/<resource>-templates.yml
files.
The <resource>-jobs.yml
files are described in the following sections.
Running a CI build/test pipeline (steps 3, 4, 5, 6)¶
The RAJA/scripts/gitlab/build_and_test.sh file defines the steps executed
for each build and test run as well as information that will appear in the
log output for each step. First, the script invokes the
RAJA/scripts/uberenv/uberenv.py
Python script located in the
uberenv submodule:
...
python3 scripts/uberenv/uberenv.py --spec="${spec}" ${prefix_opt}
...
Project specific settings related to which Spack version to use, where Spack packages live, etc. are located in the RAJA/.uberenv_config.json file.
The uberenv python script invokes Spack to generate a CMake host-config
file containing a RAJA build specification (step 3). To generate
a host-config file, Spack uses the
RAJA Spack package, plus Spack spec information.
The RAJA/.gitlab/<resource>-jobs.yml
file defines a build specification
(Spack spec) for each job that will be run on the corresponding resource.
For example, in the lassen-jobs.yml
file, you will see an entry such as:
gcc_8_3_1_cuda_10_1_168:
variables:
SPEC: "+cuda %gcc@8.3.1 cuda_arch=70 ^cuda@10.1.168"
extends: .build_and_test_on_lassen
This defines the Spack spec for the test job in which CUDA device code will
be built with the nvcc 10.1.168 compiler and non-device code will be compiled
with the GNU 8.3.1 compiler. In the Gitlab CI GUI, this pipeline will be
labeled gcc_8_3_1_cuda_10_1_168
. Details for compilers, such as file
system paths, target architecture, etc. are located in the
RAJA/scripts/radiuss-spack-configs/<sys-type>/compilers.yaml
file for the
system type associated with the resource. Analogous information for packages
like CUDA and ROCm (HIP) are located in the corresponding
RAJA/scripts/radiuss-spack-configs/<sys-type>/packages.yaml
file.
Note
Please see Spack-Generated Host-Config Files for more information about Spack-generated host-config files and how to use them for local debugging.
After the host-config file is generated, the
scripts/gitlab/build_and_test.sh
script creates a build space directory
and runs CMake in it, passing the host-config (cache) file. Then, it builds
the RAJA code and tests (step 4):
...
build_dir="${build_root}/build_${hostconfig//.cmake/}"
...
date
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo "~ Host-config: ${hostconfig_path}"
echo "~ Build Dir: ${build_dir}"
echo "~ Project Dir: ${project_dir}"
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo ""
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo "~~~~~ Building RAJA"
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
rm -rf ${build_dir} 2>/dev/null
mkdir -p ${build_dir} && cd ${build_dir}
...
cmake \
-C ${hostconfig_path} \
${project_dir}
cmake --build . -j ${core_counts[$truehostname]}
Next, it runs the tests (step 5):
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo "~~~~~ Testing RAJA"
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
...
cd ${build_dir}
...
ctest --output-on-failure -T test 2>&1 | tee tests_output.txt
Lastly, the script packages the test results into a JUnit XML file that Gitlab uses for reporting the results in its GUI (step 6):
echo "Copying Testing xml reports for export"
tree Testing
xsltproc -o junit.xml ${project_dir}/blt/tests/ctest-to-junit.xsl Testing/*/Test.xml
mv junit.xml ${project_dir}/junit.xml
The commands shown here intermingle with other commands that emit messages, timing information for various operations, etc. which appear in a log file that can be viewed in the Gitlab GUI.
Azure Pipelines CI¶
The Azure Pipelines tool builds and tests for Linux, Windows, and MacOS environments. While we do builds for CUDA, HIP, and SYCL RAJA back-ends in the Azure Linux environment, RAJA tests are only run for CPU-only pipelines.
Azure Pipelines Testing Workflow¶
The Azure Pipelines testing workflow for RAJA is much simpler than the Gitlab testing process described above.
The test jobs we run for each OS environment are specified in the
RAJA/azure-pipelines.yml file. This file defines the job steps, commands,
compilers, etc. for each OS environment in the associated - job:
section.
A summary of the configurations we build are:
Windows. The
- job: Windows
Windows section contains information for the Windows test builds. For example, we build and test RAJA as a static and shared library. This is indicated in the Windowsstrategy
section:strategy: matrix: shared: ... static: ...We use the Windows/compiler image provided by the Azure application indicated the
pool
section; for example:pool: vmImage: 'windows-2019'MacOS. The
- job: Mac
section contains information for Mac test builds. For example, we build RAJA using the the MacOS/compiler image provided by the Azure application indicated in thepool
section; for example:pool: vmImage: 'macOS-latest'Linux. The
- job: Docker
section contains information for Linux test builds. We build and test RAJA using Docker container images generated with recent versions of various compilers. The RAJA project shares these images with other open-source LLNL RADIUSS projects and they are maintained in the RES-ops Docker project on GitHub. The builds we do at any point in time are located in thestrategy
block:strategy: matrix: gccX: docker_target: ... ... clangY: docker_target: ... ... nvccZ: docker_target: ... ...The Linux OS the docker images are run on is indicated in the
pool
section; for example:pool: vmImage: 'ubuntu-latest'
Docker Builds¶
For each Linux/Docker pipeline, the base container images, CMake, build, and test commands are located in RAJA/Dockerfile.
The base container images are built and maintained through the RSE-Ops RADIUSS project. A table of the most up to date containers can be found here. These images are rebuilt regularly ensuring that we have the most up to date builds of each container / compiler.
Note
Please see Reproducing Docker Builds Locally for more information about reproducing Docker builds locally for debugging purposes.