Getting Started With RAJA¶
This section should help get you up and running with RAJA quickly.
Requirements¶
The primary requirement for using RAJA is a C++14 standard compliant compiler. Certain features, such as programming model back-ends like CUDA or HIP, must be supported by the compiler you chose to use them. Available RAJA configuration options and how to enable or disable features are described in Build Configuration Options.
To build RAJA and use its most basic features, you will need:
C++ compiler with C++14 support
CMake version 3.23 or greater when building the HIP back-end, and version 3.20 or greater otherwise.
Get the Code¶
The RAJA project is hosted on GitHub: GitHub RAJA project. To get the code, clone the repository into a local working space using the command:
$ git clone --recursive https://github.com/LLNL/RAJA.git
The --recursive
option above is used to pull RAJA Git submodules, on
which RAJA depends, into your local copy of the RAJA repository.
After running the clone command, a copy of the RAJA repository will reside in
the RAJA
subdirectory where you ran the clone command. You will be on the
develop
branch, which is the default RAJA branch.
If you do not pass the --recursive
argument to the git clone
command, you can also type the following commands after cloning:
$ cd RAJA
$ git submodule update --init --recursive
Either way, the end result is the same and you should be good to configure the code and build it.
Note
If you switch branches in a RAJA repo (e.g., you are on a branch, with everything up-to-date, and you run the command
git checkout <different branch name>
, you may need to run the commandgit submodule update
to set the Git submodule versions to what is used by the new branch.If the set of submodules in a new branch is different than the previous branch you were on, you may need to run the command
git submodule update --init --recursive
to pull in the correct set of submodule and versions.
Dependencies¶
RAJA has several dependencies that are required based on how you want to build and use it. The RAJA Git repository has submodules that contain most of these dependencies.
RAJA includes other submodule dependencies, which are used to support our Gitlab CI testing. These are described in the RAJA Developer Guide.
Dependencies that are required to build the RAJA code are:
A C++ 14 standard compliant compiler
CMake version 3.23 or greater when building the HIP back-end, and version 3.20 or greater otherwise.
Other dependencies that users should be aware of that support certain features are:
CUB CUDA utilities library, which is required for using the RAJA CUDA back-end.
rocPRIM HIP parallel primitives library, which is required for using the RAJA HIP back-end.
Desul, which is required if you want to use Desul atomics in RAJA instead of our current default atomics. Note that we plan to switch over to Desul atomics exclusively at some point.
Note
You may want or need to use external versions of camp, CUB, or rocPRIM instead of the RAJA submodules. This is usually the case when you are using RAJA along with some other library that also needs one of these. To do so, you need to use CMake variables to pass a path to a valid installation of each library. Specifically:
External camp:
cmake \ ... \ -Dcamp_DIR=path/to/camp/install \ ...External CUB:
cmake \ ... \ -DRAJA_ENABLE_EXTERNAL_CUB=On \ -DCUB_DIR=path/to/cub \ ...External rocPRIM:
cmake \ ... \ -DRAJA_ENABLE_EXTERNAL_ROCPRIM=On \ -DROCPRIM_DIR=path/to/rocPRIM \ ...
More information about configuring GPU builds with CUDA or HIP is provided in Additional RAJA Back-end Build Information
Additional discussion of these dependencies, with respect to building RAJA, is provided in Build and Install. Other than that, you probably don’t need to know much about them. If you are curious and want to know more, please click on the link to the library you want to know about in the above list.
Build and Install¶
The complexity of building and installing RAJA depends on which features you want to use and how easy it is to do this on your system.
Note
RAJA builds must be out-of-source. In particular, RAJA does not allow building in its source directory. You must create a build directory and run CMake in it.
RAJA uses CMake to configure a build. To create a “bare bones” configuration, build, and install it, you can do the following:
$ mkdir build-dir && cd build-dir
$ cmake -DCMAKE_INSTALL_PREFIX=/path/to/install ../
$ make (or make -j <N> for a parallel build)
$ make install
Running cmake
generates the RAJA build configuration. Running make
compiles the code. Running make install
copies RAJA header files
to an include
directory and installs the RAJA library in a lib
directory, both in the directory location specified with the
-DCMAKE_INSTALL_PREFIX
CMake option.
Other build configurations are accomplished by passing other options to CMake.
For example, if you want to use a C++ compiler other than the default on
your system, you would pass a path to the compiler using the standard
CMake option -DCMAKE_CXX_COMPILER=path/to/compiler
.
When you run CMake, it will generate output about the build configuration
(compiler and version, options, etc.), which is helpful to make sure CMake
is doing what you want. For a summary of RAJA configuration
options, please see Build Configuration Options.
Note
RAJA is configured to build its tests, examples, and tutorial exercises by default. If you do not disable them with the appropriate CMake option (see Build Configuration Options), you can run them after the build completes to check if everything is built properly.
The easiest way to run the full set of RAJA tests is to type:
$ make test
in the build directory after the build completes.
You can also run individual tests by invoking the corresponding
test executables directly. They will be located in the test
subdirectory in your build space. RAJA tests use the
Google Test framework,
so you can also run and filter tests via Google Test commands.
The source files for RAJA examples and exercises are located in
the RAJA/examples
and RAJA/exercises
directories,
respectively. When built, the executables for the examples and
exercises will be located in the bin
subdirectory in your build
space.
Additional RAJA Back-end Build Information¶
Configuring a RAJA build to support a GPU back-end, such as CUDA, HIP, or OpenMP target offload, typically requires additional CMake options, which we describe next.
CUDA¶
To run RAJA code on NVIDIA GPUs, one typically must have a CUDA compiler
installed on the system, in addition to a host code compiler. You may need
to specify both when you run CMake. The host compiler is specified using the
CMAKE_CXX_COMPILER
CMake variable as described earlier. The CUDA software
stack and compiler are specified using the following CMake options:
-DCUDA_TOOLKIT_ROOT_DIR=path/to/cuda/toolkit
-DCMAKE_CUDA_COMPILER=path/to/nvcc
When using the NVIDIA nvcc compiler for RAJA CUDA functionality, the variables:
CMAKE_CUDA_FLAGS_RELEASE
CMAKE_CUDA_FLAGS_DEBUG
CMAKE_CUDA_FLAGS_RELWITHDEBINFO
correspond to the standard CMake build types and are used to pass additional compiler options to nvcc.
Note
Often, nvcc must pass options to the host compiler, the arguments
can be included using the CMAKE_CUDA_FLAGS...
CMake variables
listed above. Host compiler options must be prepended with the
-Xcompiler
directive to properly propagate.
To set the CUDA compute architecture, which should be chosen based on the
NVIDIA GPU hardware you are using, you can use the CMAKE_CUDA_ARCHITECTURES
CMake variable. For example, the CMake option
-DCMAKE_CUDA_ARCHITECTURES=70
will tell the
compiler to use the sm_70 SASS architecture in its second stage of
compilation. The compiler will pick the PTX architecture to use in the first
stage of compilation that is suitable for the SASS architecture you specify.
Alternatively, you may specify the PTX and SASS architectures, using
appropriate nvcc options in the CMAKE_CUDA_FLAGS_*
variables.
Note
RAJA requires a minimum CUDA architecture level of `sm_35` to use all supported CUDA features. Mostly, the architecture level affects which RAJA CUDA atomic operations are available and how they are implemented inside RAJA. This is described in Atomic Operations.
If you do not specify a value for
CMAKE_CUDA_ARCHITECTURES
, it will be set to 35 by default and CMake will emit a status message indicating this choice was made.If you give a
CMAKE_CUDA_ARCHITECTURES
value less than 35 (e.g., 30), CMake will report this as an error and stop processing.
Also, RAJA relies on the CUB CUDA utilities library, mentioned earlier, for some CUDA back-end functionality. The CUB version included in the CUDA toolkit installation is used by default when available. This is the case for CUDA version 11 and later. RAJA includes a CUB submodule that is used by default with older versions of CUDA. To use an external CUB installation, provide the following options to CMake:
cmake \
... \
-DRAJA_ENABLE_EXTERNAL_CUB=On \
-DCUB_DIR=<path/to/cub> \
...
Note
The CUDA toolkit version of CUB is required for compatibility with the CUDA toolkit version of thrust starting with CUDA version 11.0.0. So, if you build RAJA with CUDA version 11 or higher, you should use the version of CUB contained in the CUDA toolkit version you are using to use Thrust and to be compatible with libraries that use Thrust.
Note
The version of Googletest that is used in RAJA version v0.11.0 or newer requires CUDA version 9.2.x or newer when compiling with nvcc. Thus, if you build RAJA with CUDA enabled and want to also enable RAJA tests, you must use CUDA version 9.2.x or newer.
HIP¶
To run RAJA code on AMD GPUs, one typically uses a ROCm compiler and tool chain (which can also be used to compile code for NVIDIA GPUs, which is not covered in detail in RAJA user documentation).
Note
RAJA requires version 3.5 or newer of the ROCm software stack to use the RAJA HIP back-end.
Unlike CUDA, you do not specify a host compiler and a device compiler when using the AMD ROCm software stack. Typical CMake options to use when building with a ROCm stack are:
-DROCM_ROOT_DIR=path/to/rocm
-DHIP_ROOT_DIR=path/to/hip
-DHIP_PATH=path/to/hip/binaries
-DCMAKE_CXX_COMPILER=path/to/rocm/compiler
Additionally, you use the CMake variable CMAKE_HIP_ARCHITECTURES
to set
the target compute architecture. For example:
-DCMAKE_HIP_ARCHITECTURES=gfx908
RAJA relies on the rocPRIM HIP utilities library for some HIP functionality. The rocPRIM included in the ROCm install is used by default if available. RAJA includes a rocPRIM submodule that is used if it is not available. To use an external rocPRIM install provide the following options to CMake:
cmake \
... \
-DRAJA_ENABLE_EXTERNAL_ROCPRIM=On \
-DROCPRIM_DIR=<pat/to/rocPRIM> \
...
Note
When using HIP and targeting NVIDIA GPUs, RAJA uses CUB instead of rocPRIM. In this case, you must configure with an external CUB install using the CMake variables described in the CUDA section above.
OpenMP¶
To use OpenMP target offload GPU execution, additional options may need to be passed to the compiler. BLT variables are used for this. Option syntax follows the CMake list pattern. For example, to specify OpenMP target options for NVIDIA GPUs using a clang-based compiler, one may do something like:
cmake \
... \
-DBLT_OPENMP_COMPILE_FLAGS="-fopenmp;-fopenmp-targets=nvptx64-nvidia-cuda" \
-DBLT_OPENMP_LINK_FLAGS="-fopenmp;-fopenmp-targets=nvptx64-nvidia-cuda" \
...
Compiler flags are passed to other compilers similarly, using flags specific to the compiler. Typically, the compile and link flags are the same as shown here.
RAJA Example Build Configuration Files¶
The RAJA repository has subdirectories RAJA/scripts/*-builds
that contain
a variety of build scripts we use to build and test RAJA on various platforms
with various compilers. These scripts pass files (CMake cache files)
located in the RAJA/host-configs
directory to CMake using the ‘-C’ option.
These files serve as useful examples of how to configure RAJA prior to
compilation.
Learning to Use RAJA¶
The RAJA repository contains a variety of example source codes that you are encouraged to view and run to learn about how to use RAJA:
The
RAJA/examples
directory contains various examples that illustrate algorithm patterns.The
RAJA/exercises
directory contains exercises for users to work through along with complete solutions. These are described in detail in the RAJA Tutorial and Examples section.Other examples can also be found in the
RAJA/test
directories.
We mentioned earlier that RAJA examples, exercises, and tests are built by
default when RAJA is compiled. So, unless you explicitly disable them when
you run CMake to configure a RAJA build, you can run them after compiling RAJA.
Executables for the examples and exercises will be located in the
<build-dir>/bin
directory in your build space. Test executables will
be located in the <build-dir>/test
directory.
For an overview of all the main RAJA features, see RAJA Features. A full tutorial with a variety of examples showing how to use RAJA features can be found in RAJA Tutorial and Examples.