RAJA Tests¶

As noted in Continuous Integration (CI) Testing, all RAJA test checks must pass before any PR contribution will be merged. Additionally, we recommend that contributors include new tests in their code contributions when adding new features and bug fixes.

Note

If RAJA team members think adequate testing is not included in a PR branch, they will ask for additional testing to be added during the review process.

Test Organization¶

The goals of the RAJA test organization are to:

Make it easy to see what is tested and where tests live. We want developers and users to be able to find tests easily and know where to put new tests when they add them.

Parameterize tests as much as reasonable to ensure that features work with all supported RAJA back-ends and we are testing them consistently. We want the source files for each test case to allow testing of each RAJA back-end. Specifically, tests for each back-end are generated by instantiating the same source routines with different type information.

Have test source code generated for compilation by CMake when the code is configured. This significantly reduces code redundancy and enables our test parameterization goals.

All RAJA tests reside in the RAJA/test directory. The test directory structure looks like this:

RAJA/test/functional/forall
                     kernel
                     scan
                     ...
          include/...
          integration/...
          unit/algorithm
               atomic
               index
               ...

RAJA tests are partitioned into three main categories:

Unit tests exercise basic interfaces and features of individual RAJA classes and methods in standalone fashion; i.e., integrated with other parts of RAJA as minimally as is reasonable. RAJA unit tests reside in sub-directories of the RAJA/test/unit directory.

Functional tests integrate multiple RAJA features in common ways to test how RAJA is used in practice. RAJA functional tests reside in sub-directories of the RAJA/test/functional directory.

Integration tests exercise features that integrate RAJA with other libraries, such as Kokkos performance tools as plug-ins. RAJA integration tests reside in sub-directories of the RAJA/test/integration directory.

The RAJA/test/include directory contains header files that define types and other items that are commonly used in various tests.

Important

Please follow the existing sub-directory structure and code implementation patterns for RAJA tests when adding or modifying tests.

Anatomy Of A Test Case¶

This section discusses in some detail the structure of files for a single RAJA test case and how the work together. In particular, we describe the set of basic tests that exercise RAJA::forall execution with various RAJA segment types.

Note

The implementation pattern described in the following sections is similarly used by all other RAJA tests.

Since these tests integrate multiple RAJA features, it is considered a functional test. The files for this test are located in the RAJA/test/functional/forall/segment directory. The contents of the directory are:

$ ls -c1 -R ./test/functional/forall/segment
./test/functional/forall/segment:
tests
test-forall-segment.cpp.in
CMakeLists.txt

./test/functional/forall/segment/tests:
test-forall-RangeStrideSegment.hpp
test-forall-RangeSegment.hpp
test-forall-ListSegment.hpp

Next, we describe these and their relationships.

Test Source File¶

The test-forall-segment.cpp.in file is the parameterized test source file. It contains header file include statements:

//
// test/include headers
//
#include "RAJA_test-base.hpp"
#include "RAJA_test-camp.hpp"
#include "RAJA_test-index-types.hpp"

#include "RAJA_test-forall-data.hpp"
#include "RAJA_test-forall-execpol.hpp"

//
// Header for tests in ./tests directory
//
// Note: CMake adds ./tests as an include dir for these tests.
//
#include "test-forall-@SEGTYPE@.hpp"

The first set of header files live in the RAJA/test/include directory mentioned earlier. The headers are centrally located since their contents are shared with other test files. The last include statement pulls in the header file containing the parameterized tests for the corresponding RAJA segment type.

Next, a camp::cartesian_product type is defined to assemble sets of types used in the parameterized tests:

//
// Cartesian product of types used in parameterized tests
//
using @BACKEND@ForallSegmentTypes =
  Test< camp::cartesian_product<StrongIdxTypeList,
                                @BACKEND@ResourceList,
                                @BACKEND@ForallExecPols>>::Types;

The first template argument defining the camp::cartesian_product object type refers to a list of segment index types defined in the RAJA_test-index-types.hpp header file. The second argument refers to a list of RAJA/camp resource types appropriate for the RAJA execution back-end defined in the RAJA_test-camp.hpp header file (see Test Header files for where this is used). The third argument refers to a list of RAJA execution policy types defined in the RAJA_test-forall-execpol.hpp header file. This results in the generation of a combinatorial collection of typed tests being run. Each test is defined by a unique tuple of types, described in Test Header files.

Lastly, the parameterized set of tests is instantiated:

//
// Instantiate parameterized test
//
INSTANTIATE_TYPED_TEST_SUITE_P(@BACKEND@,
                               Forall@SEGTYPE@Test,
                               @BACKEND@ForallSegmentTypes);

INSTANTIATE_TYPED_TEST_SUITE_P is a GoogleTest macro. The first argument is a label noting the RAJA back-end used for the generated tests. This can be used to filter the tests when they are manually run. The second argument is a label identifying the test set, and the third argument matches the CMake generated name for the camp::cartesian_product type described above.

Important

The second argument passed to the INSTANTIATE_TYPED_TEST_SUITE_P macro must match the name of the test suite class discussed in Test Header files.

CMakeLists.txt File¶

The concrete version of each of the items described above is generated by CMake when a RAJA build is configured. CMake fills in the segment type and back-end identifiers, @SEGTYPE@ and @BACKEND@, respectively. These identifiers and the test file and executable generation process is defined in the CMakeLists.txt file in the test directory. If you look in the file, you will see nested loops over RAJA back-ends and segment types which process the test source file test-forall-segment.cpp.in multiple times to create a uniquely named source file for each back-end/segment type combination in the RAJA build space. Each source file will be compiled into a similarly named, unique test executable when the code is compiled.

Test Header files¶

Recall the line in the test source file:

#include "test-forall-@SEGTYPE@.hpp"

This identifies the header file containing the actual test code used to generate the tests. The test header files are located in the RAJA/test/functional/forall/segment/tests directory. The main elements of each test header file are described next. We use the test-forall-RangeSegment.hpp file to illustrate the essential test implementation elements.

The file contains the following important items:

test implementation method

typed test suite class

typed test invocation

type test suite registration

The test implementation is contained in a parameterized template method:

template <typename INDEX_TYPE, typename WORKING_RES, typename EXEC_POLICY>
void ForallRangeSegmentTestImpl(INDEX_TYPE first, INDEX_TYPE last)
{
   ...
}

Here, the template parameters identify the index type of the RAJA segment INDEX_TYPE, the resource type for allocating test memory in the proper execution environment WORKING_RES, and the execution policy EXEC_POLICY for the RAJA::forall method used to run the tests.

The test suite class plugs into the GoogleTest framework:

TYPED_TEST_SUITE_P(ForallRangeSegmentTest);
template <typename T>
class ForallRangeSegmentTest : public ::testing::Test
{
};

using the TYPED_TEST_SUITE_P GoogleTest macro.

Important

The name of the test class must be identical to the label passed to the GoogleTest TYPED_TEST_SUITE_P macro.

The specific tests that are run are defined by calls to the test implementation template method ForallRangeSegmentTestImpl described above:

TYPED_TEST_P(ForallRangeSegmentTest, RangeSegmentForall)
{
  using INDEX_TYPE  = typename camp::at<TypeParam, camp::num<0>>::type;
  using WORKING_RES = typename camp::at<TypeParam, camp::num<1>>::type;
  using EXEC_POLICY = typename camp::at<TypeParam, camp::num<2>>::type;

  // test zero-length range segment
  ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(3), INDEX_TYPE(3));

  ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(0), INDEX_TYPE(27));
  ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(1), INDEX_TYPE(2047));
  ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(1), INDEX_TYPE(32000));

  runNegativeTests<INDEX_TYPE, WORKING_RES, EXEC_POLICY>();
}

Here, TYPED_TEST_P is a GoogleTest macro defining the method for executing the tests. Note that the first three lines in the method extract the template parameter types from the camp::tuple produced by the camp::cartesian_product described earlier in Test Source File. If you look in the file, you will see an example of how we use C++ SFINAE to exclude running tests with negative index values for index types that are unsigned.

Important

The label passed as the first argument to the GoogleTest TYPED_TEST_P macro must match the name of the test suite class. The second argument is discussed below.
It is critical to use the same type ordering when extracting the types that was used when the camp::cartesian_product type was defined in the test source file, described in Test Source File.

Lastly, the test suite is registered with GoogleTest using the REGISTER_TYPED_TEST_SUITE_P macro:

REGISTER_TYPED_TEST_SUITE_P(ForallRangeSegmentTest,
                            RangeSegmentForall);

Important

The label passed as the first argument to the GoogleTest REGISTER_TYPED_TEST_SUITE_P macro must match the name of the test suite class.
The label passed as the second argument to the GoogleTest REGISTER_TYPED_TEST_SUITE_P macro must match the label passed as the second argument to the TYPED_TEST_P macro.