1==============
2Testing libc++
3==============
4
5.. contents::
6  :local:
7
8Getting Started
9===============
10
11libc++ uses LIT to configure and run its tests.
12
13The primary way to run the libc++ tests is by using ``make check-cxx``.
14
15However since libc++ can be used in any number of possible
16configurations it is important to customize the way LIT builds and runs
17the tests. This guide provides information on how to use LIT directly to
18test libc++.
19
20Please see the `Lit Command Guide`_ for more information about LIT.
21
22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
23
24Usage
25-----
26
27After building libc++, you can run parts of the libc++ test suite by simply
28running ``llvm-lit`` on a specified test or directory. If you're unsure
29whether the required libraries have been built, you can use the
30``cxx-test-depends`` target. For example:
31
32.. code-block:: bash
33
34  $ cd <monorepo-root>
35  $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt
36  $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
37  $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
38  $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
39
40.. note::
41  If you used the Bootstrapping build instead of the default runtimes build, the
42  ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and
43  you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the
44  paths of all tests.
45
46In the default configuration, the tests are built against headers that form a
47fake installation root of libc++. This installation root has to be updated when
48changes are made to the headers, so you should re-run the ``cxx-test-depends``
49target before running the tests manually with ``lit`` when you make any sort of
50change, including to the headers.
51
52Sometimes you'll want to change the way LIT is running the tests. Custom options
53can be specified using the ``--param <name>=<val>`` flag. The most common option
54you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the
55test suite will select the newest C++ dialect supported by the compiler and use
56that. However, you can manually specify the option like so if you want:
57
58.. code-block:: bash
59
60  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
61  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
62
63Other parameters are supported by the test suite. Those are defined in ``libcxx/utils/libcxx/test/params.py``.
64If you want to customize how to run the libc++ test suite beyond what is available
65in ``params.py``, you most likely want to use a custom site configuration instead.
66
67The libc++ test suite works by loading a site configuration that defines various
68"base" parameters (via Lit substitutions). These base parameters represent things
69like the compiler to use for running the tests, which default compiler and linker
70flags to use, and how to run an executable. This system is meant to be easily
71extended for custom needs, in particular when porting the libc++ test suite to
72new platforms.
73
74Using a Custom Site Configuration
75---------------------------------
76
77By default, the libc++ test suite will use a site configuration that matches
78the current CMake configuration. It does so by generating a ``lit.site.cfg``
79file in the build directory from one of the configuration file templates in
80``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around
81``llvm/utils/lit/lit.py``) to that file. So when you're running
82``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded
83instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site
84configuration, simply point the CMake build to it using
85``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration
86will be used instead. That file can use CMake variables inside it to make
87configuration easier.
88
89   .. code-block:: bash
90
91     $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
92     $ make -C <build> cxx-test-depends
93     $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file
94
95Writing Tests
96-------------
97
98When writing tests for the libc++ test suite, you should follow a few guidelines.
99This will ensure that your tests can run on a wide variety of hardware and under
100a wide variety of configurations. We have several unusual configurations such as
101building the tests on one host but running them on a different host, which add a
102few requirements to the test suite. Here's some stuff you should know:
103
104- All tests are run in a temporary directory that is unique to that test and
105  cleaned up after the test is done.
106- When a test needs data files as inputs, these data files can be saved in the
107  repository (when reasonable) and referenced by the test as
108  ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
109  directories will be made available to the test in the temporary directory
110  where it is run.
111- You should never hardcode a path from the build-host in a test, because that
112  path will not necessarily be available on the host where the tests are run.
113- You should try to reduce the runtime dependencies of each test to the minimum.
114  For example, requiring Python to run a test is bad, since Python is not
115  necessarily available on all devices we may want to run the tests on (even
116  though supporting Python is probably trivial for the build-host).
117
118Benchmarks
119==========
120
121Libc++ contains benchmark tests separately from the test of the test suite.
122The benchmarks are written using the `Google Benchmark`_ library, a copy of which
123is stored in the libc++ repository.
124
125For more information about using the Google Benchmark library see the
126`official documentation <https://github.com/google/benchmark>`_.
127
128.. _`Google Benchmark`: https://github.com/google/benchmark
129
130Building Benchmarks
131-------------------
132
133The benchmark tests are not built by default. The benchmarks can be built using
134the ``cxx-benchmarks`` target.
135
136An example build would look like:
137
138.. code-block:: bash
139
140  $ cd build
141  $ ninja cxx-benchmarks
142
143This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
144built against the just-built libc++. The compiled tests are output into
145``build/projects/libcxx/benchmarks``.
146
147The benchmarks can also be built against the platforms native standard library
148using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
149is useful for comparing the performance of libc++ to other standard libraries.
150The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
151``<test>.native.out`` otherwise.
152
153Also See:
154
155  * :ref:`Building Libc++ <build instructions>`
156  * :ref:`CMake Options`
157
158Running Benchmarks
159------------------
160
161The benchmarks must be run manually by the user. Currently there is no way
162to run them as part of the build.
163
164For example:
165
166.. code-block:: bash
167
168  $ cd build/projects/libcxx/benchmarks
169  $ ./algorithms.libcxx.out # Runs all the benchmarks
170  $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
171
172For more information about running benchmarks see `Google Benchmark`_.
173