1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. 12 13The primary way to run the libc++ tests is by using `make check-cxx`. 14 15However since libc++ can be used in any number of possible 16configurations it is important to customize the way LIT builds and runs 17the tests. This guide provides information on how to use LIT directly to 18test libc++. 19 20Please see the `Lit Command Guide`_ for more information about LIT. 21 22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html 23 24Setting up the Environment 25-------------------------- 26 27After building libc++ you must setup your environment to test libc++ using 28LIT. 29 30#. Create a shortcut to the actual lit executable so that you can invoke it 31 easily from the command line. 32 33 .. code-block:: bash 34 35 $ alias lit='python path/to/llvm/utils/lit/lit.py' 36 37#. Tell LIT where to find your build configuration. 38 39 .. code-block:: bash 40 41 $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg 42 43Example Usage 44------------- 45 46Once you have your environment set up and you have built libc++ you can run 47parts of the libc++ test suite by simply running `lit` on a specified test or 48directory. For example: 49 50.. code-block:: bash 51 52 $ cd path/to/src/libcxx 53 $ lit -sv test/std/re # Run all of the std::regex tests 54 $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 55 $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic 56 57Sometimes you'll want to change the way LIT is running the tests. Custom options 58can be specified using the `--param=<name>=<val>` flag. The most common option 59you'll want to change is the standard dialect (ie -std=c++XX). By default the 60test suite will select the newest C++ dialect supported by the compiler and use 61that. However if you want to manually specify the option like so: 62 63.. code-block:: bash 64 65 $ lit -sv test/std/containers # Run the tests with the newest -std 66 $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 67 68Occasionally you'll want to add extra compile or link flags when testing. 69You can do this as follows: 70 71.. code-block:: bash 72 73 $ lit -sv --param=compile_flags='-Wcustom-warning' 74 $ lit -sv --param=link_flags='-L/custom/library/path' 75 76Some other common examples include: 77 78.. code-block:: bash 79 80 # Specify a custom compiler. 81 $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std 82 83 # Enable warnings in the test suite 84 $ lit -sv --param=enable_warnings=true test/std 85 86 # Use UBSAN when running the tests. 87 $ lit -sv --param=use_sanitizer=Undefined 88 89 90LIT Options 91=========== 92 93:program:`lit` [*options*...] [*filenames*...] 94 95Command Line Options 96-------------------- 97 98To use these options you pass them on the LIT command line as --param NAME or 99--param NAME=VALUE. Some options have default values specified during CMake's 100configuration. Passing the option on the command line will override the default. 101 102.. program:: lit 103 104.. option:: cxx_under_test=<path/to/compiler> 105 106 Specify the compiler used to build the tests. 107 108.. option:: cxx_stdlib_under_test=<stdlib name> 109 110 **Values**: libc++, libstdc++ 111 112 Specify the C++ standard library being tested. Unless otherwise specified 113 libc++ is used. This option is intended to allow running the libc++ test 114 suite against other standard library implementations. 115 116.. option:: std=<standard version> 117 118 **Values**: c++98, c++03, c++11, c++14, c++17, c++2a 119 120 Change the standard version used when building the tests. 121 122.. option:: libcxx_site_config=<path/to/lit.site.cfg> 123 124 Specify the site configuration to use when running the tests. This option 125 overrides the environment variable LIBCXX_SITE_CONFIG. 126 127.. option:: cxx_headers=<path/to/headers> 128 129 Specify the c++ standard library headers that are tested. By default the 130 headers in the source tree are used. 131 132.. option:: cxx_library_root=<path/to/lib/> 133 134 Specify the directory of the libc++ library to be tested. By default the 135 library folder of the build directory is used. This option cannot be used 136 when use_system_cxx_lib is provided. 137 138 139.. option:: cxx_runtime_root=<path/to/lib/> 140 141 Specify the directory of the libc++ library to use at runtime. This directory 142 is not added to the linkers search path. This can be used to compile tests 143 against one version of libc++ and run them using another. The default value 144 for this option is `cxx_library_root`. 145 146.. option:: use_system_cxx_lib=<bool> 147 148 **Default**: False 149 150 Enable or disable testing against the installed version of libc++ library. 151 Note: This does not use the installed headers. 152 153.. option:: use_lit_shell=<bool> 154 155 Enable or disable the use of LIT's internal shell in ShTests. If the 156 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as 157 the default value. Otherwise the default value is True on Windows and False 158 on every other platform. 159 160.. option:: compile_flags="<list-of-args>" 161 162 Specify additional compile flags as a space delimited string. 163 Note: This options should not be used to change the standard version used. 164 165.. option:: link_flags="<list-of-args>" 166 167 Specify additional link flags as a space delimited string. 168 169.. option:: debug_level=<level> 170 171 **Values**: 0, 1 172 173 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 174 assertions and debugging of iterator misuse. 175 176.. option:: use_sanitizer=<sanitizer name> 177 178 **Values**: Memory, MemoryWithOrigins, Address, Undefined 179 180 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 181 building libc++ then that sanitizer will be used by default. 182 183.. option:: color_diagnostics 184 185 Enable the use of colorized compile diagnostics. If the color_diagnostics 186 option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is 187 present then color diagnostics will be enabled. 188 189.. option:: llvm_unwinder 190 191 Enable the use of LLVM unwinder instead of libgcc. 192 193.. option:: builtins_library 194 195 Path to the builtins library to use instead of libgcc. 196 197 198Environment Variables 199--------------------- 200 201.. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> 202 203 Specify the site configuration to use when running the tests. 204 Also see `libcxx_site_config`. 205 206.. envvar:: LIBCXX_COLOR_DIAGNOSTICS 207 208 If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt 209 to use color diagnostic outputs from the compiler. 210 Also see `color_diagnostics`. 211 212Writing Tests 213------------- 214 215When writing tests for the libc++ test suite, you should follow a few guidelines. 216This will ensure that your tests can run on a wide variety of hardware and under 217a wide variety of configurations. We have several unusual configurations such as 218building the tests on one host but running them on a different host, which add a 219few requirements to the test suite. Here's some stuff you should know: 220 221- All tests are run in a temporary directory that is unique to that test and 222 cleaned up after the test is done. 223- When a test needs data files as inputs, these data files can be saved in the 224 repository (when reasonable) and referrenced by the test as 225 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or 226 directories will be made available to the test in the temporary directory 227 where it is run. 228- You should never hardcode a path from the build-host in a test, because that 229 path will not necessarily be available on the host where the tests are run. 230- You should try to reduce the runtime dependencies of each test to the minimum. 231 For example, requiring Python to run a test is bad, since Python is not 232 necessarily available on all devices we may want to run the tests on (even 233 though supporting Python is probably trivial for the build-host). 234 235Benchmarks 236========== 237 238Libc++ contains benchmark tests separately from the test of the test suite. 239The benchmarks are written using the `Google Benchmark`_ library, a copy of which 240is stored in the libc++ repository. 241 242For more information about using the Google Benchmark library see the 243`official documentation <https://github.com/google/benchmark>`_. 244 245.. _`Google Benchmark`: https://github.com/google/benchmark 246 247Building Benchmarks 248------------------- 249 250The benchmark tests are not built by default. The benchmarks can be built using 251the ``cxx-benchmarks`` target. 252 253An example build would look like: 254 255.. code-block:: bash 256 257 $ cd build 258 $ cmake [options] <path to libcxx sources> 259 $ make cxx-benchmarks 260 261This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 262built against the just-built libc++. The compiled tests are output into 263``build/benchmarks``. 264 265The benchmarks can also be built against the platforms native standard library 266using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 267is useful for comparing the performance of libc++ to other standard libraries. 268The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 269``<test>.native.out`` otherwise. 270 271Also See: 272 273 * :ref:`Building Libc++ <build instructions>` 274 * :ref:`CMake Options` 275 276Running Benchmarks 277------------------ 278 279The benchmarks must be run manually by the user. Currently there is no way 280to run them as part of the build. 281 282For example: 283 284.. code-block:: bash 285 286 $ cd build/benchmarks 287 $ make cxx-benchmarks 288 $ ./algorithms.libcxx.out # Runs all the benchmarks 289 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 290 291For more information about running benchmarks see `Google Benchmark`_. 292