.. ##
.. ## Copyright (c) Lawrence Livermore National Security, LLC and other
.. ## RAJA Project Developers. See top-level LICENSE and COPYRIGHT
.. ## files for dates and other details. No copyright assignment is required
.. ## to contribute to RAJA.
.. ##
.. ## SPDX-License-Identifier: (BSD-3-Clause)
.. ##

.. _tests-label:

***************************
RAJA Tests
***************************

As noted in :ref:`ci-label`, all RAJA test checks must pass before any PR 
contribution will be merged. Additionally, we recommend that contributors
include new tests in their code contributions when adding new features
and bug fixes.

.. note:: If RAJA team members think adequate testing is not included in a 
          PR branch, they will ask for additional testing to be added during
          the review process.

.. _tests_organization-label:

=========================
Test Organization
=========================

The goals of the RAJA test organization are to:

  * Make it easy to see what is tested and where tests live. We want
    developers and users to be able to find tests easily and know where
    to put new tests when they add them.
  * Parameterize tests as much as reasonable to ensure that features work with 
    all supported RAJA back-ends and we are testing them consistently. We want
    the source files for each test case to allow testing of each RAJA back-end.
    Specifically, tests for each back-end are generated by instantiating the 
    same source routines with different type information.
  * Have test source code generated for compilation by CMake when the code is 
    configured. This significantly reduces code redundancy and enables our
    test parameterization goals.

.. important: RAJA uses the `GoogleTest <https://github.com/google/googletest>`_ framework, which is included in the `BLT <https://github.com/LLNL/blt>`_ build system that RAJA uses. 

All RAJA tests reside in the 
`RAJA/test <https://github.com/LLNL/RAJA/tree/develop/test>`_ directory.
The test directory structure looks like this::

  RAJA/test/functional/forall
                       kernel
                       scan
                       ...
            include/...
            integration/...
            unit/algorithm
                 atomic
                 index
                 ...

RAJA tests are partitioned into three main categories:

  * **Unit tests** exercise basic interfaces and features of individual RAJA
    classes and methods in standalone fashion; i.e., integrated with other 
    parts of RAJA as minimally as is reasonable. RAJA unit tests reside
    in sub-directories of the `RAJA/test/unit <https://github.com/LLNL/RAJA/tree/develop/test/unit>`_ directory.
  * **Functional tests** integrate multiple RAJA features in common ways to 
    test how RAJA is used in practice. RAJA functional tests reside
    in sub-directories of the `RAJA/test/functional <https://github.com/LLNL/RAJA/tree/develop/test/functional>`_ directory.
  * **Integration tests** exercise features that integrate RAJA with other 
    libraries, such as Kokkos performance tools as plug-ins. RAJA integration 
    tests reside in sub-directories of the `RAJA/test/integration <https://github.com/LLNL/RAJA/tree/develop/test/integration>`_ directory.

The `RAJA/test/include <https://github.com/LLNL/RAJA/tree/develop/test/include>`_ directory contains header files that define types and other items that are 
commonly used in various tests.

.. important:: Please follow the existing sub-directory structure and code 
               implementation patterns for RAJA tests when adding or modifying 
               tests. 

.. _tests_anatomy-label:

=========================
Anatomy Of A Test Case
=========================

This section discusses in some detail the structure of files for a single
RAJA test case and how the work together. In particular, we describe the set
of basic tests that exercise ``RAJA::forall`` execution with various RAJA
segment types. 

.. note:: The implementation pattern described in the following sections is 
          similarly used by all other RAJA tests. 

Since these tests integrate multiple RAJA features, it is considered a 
*functional* test. The files for this test are located in the
`RAJA/test/functional/forall/segment <https://github.com/LLNL/RAJA/tree/develop/test/functional/forall/segment>`_ directory. The contents of the directory are::

  $ ls -c1 -R ./test/functional/forall/segment 
  ./test/functional/forall/segment:
  tests
  test-forall-segment.cpp.in
  CMakeLists.txt

  ./test/functional/forall/segment/tests:
  test-forall-RangeStrideSegment.hpp
  test-forall-RangeSegment.hpp
  test-forall-ListSegment.hpp

Next, we describe these and their relationships.

.. _tests_source-label:

Test Source File
-----------------

The `test-forall-segment.cpp.in <https://github.com/LLNL/RAJA/blob/develop/test/functional/forall/segment/test-forall-segment.cpp.in>`_ file is the 
parameterized test source file. It contains header file include statements::

  //
  // test/include headers
  //
  #include "RAJA_test-base.hpp"
  #include "RAJA_test-camp.hpp"
  #include "RAJA_test-index-types.hpp"

  #include "RAJA_test-forall-data.hpp"
  #include "RAJA_test-forall-execpol.hpp"

  //
  // Header for tests in ./tests directory
  //
  // Note: CMake adds ./tests as an include dir for these tests.
  //
  #include "test-forall-@SEGTYPE@.hpp"

The first set of header files live in the ``RAJA/test/include`` directory
mentioned earlier. The headers are centrally located since their contents 
are shared with other test files. The last include statement pulls in the
header file containing the parameterized tests for the corresponding RAJA 
segment type.

Next, a ``camp::cartesian_product`` type is defined to assemble sets of types 
used in the parameterized tests::

  //
  // Cartesian product of types used in parameterized tests
  //
  using @BACKEND@ForallSegmentTypes =
    Test< camp::cartesian_product<StrongIdxTypeList,
                                  @BACKEND@ResourceList,
                                  @BACKEND@ForallExecPols>>::Types;

The first template argument defining the ``camp::cartesian_product object``
type refers to a list of segment index types defined in the 
`RAJA_test-index-types.hpp <https://github.com/LLNL/RAJA/blob/develop/test/include/RAJA_test-index-types.hpp>`_ header file. 
The second argument refers to  a list
of RAJA/camp resource types appropriate for the RAJA execution back-end defined
in the `RAJA_test-camp.hpp <https://github.com/LLNL/RAJA/blob/develop/test/include/RAJA_test-camp.hpp>`_ header file (see :ref:`tests_header-label` for 
where this is used). The third argument refers to a list of RAJA
execution policy types defined in the 
`RAJA_test-forall-execpol.hpp <https://github.com/LLNL/RAJA/blob/develop/test/include/RAJA_test-forall-execpol.hpp>`_
header file. This results in the generation of a combinatorial collection of 
typed tests being run. Each test is defined by a unique tuple of types, 
described in :ref:`tests_header-label`.

Lastly, the parameterized set of tests is instantiated::

  //
  // Instantiate parameterized test
  //
  INSTANTIATE_TYPED_TEST_SUITE_P(@BACKEND@,
                                 Forall@SEGTYPE@Test,
                                 @BACKEND@ForallSegmentTypes);

``INSTANTIATE_TYPED_TEST_SUITE_P`` is a GoogleTest macro. The first 
argument is a label noting the RAJA back-end used for the generated tests.
This can be used to filter the tests when they are manually run.
The second argument is a label identifying the test set, and the
third argument matches the CMake generated name for the 
``camp::cartesian_product`` type described above.

.. important:: The second argument passed to the 
              ``INSTANTIATE_TYPED_TEST_SUITE_P`` macro must match the name of
              the test suite class discussed in :ref:`tests_header-label`.

.. _tests_cmakelists-label:

CMakeLists.txt File
--------------------

The concrete version of each of the items described above is generated by 
CMake when a RAJA build is configured. CMake fills in the segment type and 
back-end identifiers, ``@SEGTYPE@`` and ``@BACKEND@``, respectively. These 
identifiers and the test file and executable generation process is defined in 
the 
`CMakeLists.txt <https://github.com/LLNL/RAJA/blob/develop/test/functional/forall/segment/CMakeLists.txt>`_ file in the test directory. If you look in the file,
you will see nested loops over RAJA back-ends and segment types which
process the test source file ``test-forall-segment.cpp.in`` multiple times 
to create a uniquely named source file for each back-end/segment type 
combination in the RAJA build space. Each source file will be compiled into 
a similarly named, unique test executable when the code is compiled.

.. _tests_header-label:

Test Header files
--------------------

Recall the line in the test source file::

  #include "test-forall-@SEGTYPE@.hpp"

This identifies the header file containing the actual test code used to 
generate the tests. The test header files are located in the 
`RAJA/test/functional/forall/segment/tests <https://github.com/LLNL/RAJA/tree/develop/test/functional/forall/segment/tests>`_ directory. The main elements of 
each test header file are described next. We use the 
`test-forall-RangeSegment.hpp <https://github.com/LLNL/RAJA/tree/develop/test/functional/forall/segment/tests/test-forall-RangeSegment.hpp>`_ file to 
illustrate the essential test implementation elements.

The file contains the following important items:

  * test implementation method
  * typed test suite class
  * typed test invocation
  * type test suite registration
 
The test implementation is contained in a parameterized template method::

  template <typename INDEX_TYPE, typename WORKING_RES, typename EXEC_POLICY>
  void ForallRangeSegmentTestImpl(INDEX_TYPE first, INDEX_TYPE last)
  {
     ...
  }

Here, the template parameters identify the index type of the RAJA
segment ``INDEX_TYPE``, the resource type for allocating test memory in the
proper execution environment ``WORKING_RES``, and the execution policy
``EXEC_POLICY`` for the ``RAJA::forall`` method used to run the tests.

The test suite class plugs into the GoogleTest framework::

  TYPED_TEST_SUITE_P(ForallRangeSegmentTest);
  template <typename T>
  class ForallRangeSegmentTest : public ::testing::Test
  {
  };

using the ``TYPED_TEST_SUITE_P`` GoogleTest macro.

.. important:: The name of the test class must be identical to the label passed
               to the GoogleTest ``TYPED_TEST_SUITE_P`` macro.

The specific tests that are run are defined by calls to the test implementation
template method ``ForallRangeSegmentTestImpl`` described above::

  TYPED_TEST_P(ForallRangeSegmentTest, RangeSegmentForall)
  {
    using INDEX_TYPE  = typename camp::at<TypeParam, camp::num<0>>::type;
    using WORKING_RES = typename camp::at<TypeParam, camp::num<1>>::type;
    using EXEC_POLICY = typename camp::at<TypeParam, camp::num<2>>::type;

    // test zero-length range segment
    ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(3), INDEX_TYPE(3));

    ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(0), INDEX_TYPE(27));
    ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(1), INDEX_TYPE(2047));
    ForallRangeSegmentTestImpl<INDEX_TYPE, WORKING_RES, EXEC_POLICY>(INDEX_TYPE(1), INDEX_TYPE(32000));

    runNegativeTests<INDEX_TYPE, WORKING_RES, EXEC_POLICY>();
  } 

Here, ``TYPED_TEST_P`` is a GoogleTest macro defining the method for 
executing the tests. Note that the first three lines 
in the method extract the template parameter types from the ``camp::tuple`` 
produced by the ``camp::cartesian_product`` described earlier in
:ref:`tests_source-label`. If you look in the file, you will see an example of
how we use C++ SFINAE to exclude running tests with negative index values 
for index types that are unsigned.

.. important:: * The label passed as the first argument to the GoogleTest
                 ``TYPED_TEST_P`` macro must match the name of the test suite 
                 class. The second argument is discussed below.
               * It is critical to use the same type ordering when extracting
                 the types that was used when the ``camp::cartesian_product`` 
                 type was defined in the test source file, described in
                 :ref:`tests_source-label`.

Lastly, the test suite is registered with GoogleTest using the
``REGISTER_TYPED_TEST_SUITE_P`` macro::

  REGISTER_TYPED_TEST_SUITE_P(ForallRangeSegmentTest,
                              RangeSegmentForall);

.. important:: * The label passed as the first argument to the GoogleTest
                 ``REGISTER_TYPED_TEST_SUITE_P`` macro must match the name of 
                 the test suite class. 
               * The label passed as the second argument to the GoogleTest
                 ``REGISTER_TYPED_TEST_SUITE_P`` macro must match the label
                 passed as the second argument to the ``TYPED_TEST_P`` macro.