1==============
2Testing libc++
3==============
4
5.. contents::
6  :local:
7
8Getting Started
9===============
10
11libc++ uses LIT to configure and run its tests. The primary way to run the
12libc++ tests is by using make check-libcxx. However since libc++ can be used
13in any number of possible configurations it is important to customize the way
14LIT builds and runs the tests. This guide provides information on how to use
15LIT directly to test libc++.
16
17Please see the `Lit Command Guide`_ for more information about LIT.
18
19.. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html
20
21Setting up the Environment
22--------------------------
23
24After building libc++ you must setup your environment to test libc++ using
25LIT.
26
27#. Create a shortcut to the actual lit executable so that you can invoke it
28   easily from the command line.
29
30   .. code-block:: bash
31
32     $ alias lit='python path/to/llvm/utils/lit/lit.py'
33
34#. Tell LIT where to find your build configuration.
35
36   .. code-block:: bash
37
38     $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
39
40Example Usage
41-------------
42
43Once you have your environment set up and you have built libc++ you can run
44parts of the libc++ test suite by simply running `lit` on a specified test or
45directory. For example:
46
47.. code-block:: bash
48
49  $ cd path/to/src/libcxx
50  $ lit -sv test/std/re # Run all of the std::regex tests
51  $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
52  $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
53
54Sometimes you'll want to change the way LIT is running the tests. Custom options
55can be specified using the `--param=<name>=<val>` flag. The most common option
56you'll want to change is the standard dialect (ie -std=c++XX). By default the
57test suite will select the newest C++ dialect supported by the compiler and use
58that. However if you want to manually specify the option like so:
59
60.. code-block:: bash
61
62  $ lit -sv test/std/containers # Run the tests with the newest -std
63  $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
64
65Occasionally you'll want to add extra compile or link flags when testing.
66You can do this as follows:
67
68.. code-block:: bash
69
70  $ lit -sv --param=compile_flags='-Wcustom-warning'
71  $ lit -sv --param=link_flags='-L/custom/library/path'
72
73Some other common examples include:
74
75.. code-block:: bash
76
77  # Specify a custom compiler.
78  $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
79
80  # Enable warnings in the test suite
81  $ lit -sv --param=enable_warnings=true test/std
82
83  # Use UBSAN when running the tests.
84  $ lit -sv --param=use_sanitizer=Undefined
85
86
87LIT Options
88===========
89
90:program:`lit` [*options*...] [*filenames*...]
91
92Command Line Options
93--------------------
94
95To use these options you pass them on the LIT command line as --param NAME or
96--param NAME=VALUE. Some options have default values specified during CMake's
97configuration. Passing the option on the command line will override the default.
98
99.. program:: lit
100
101.. option:: cxx_under_test=<path/to/compiler>
102
103  Specify the compiler used to build the tests.
104
105.. option:: std=<standard version>
106
107  **Values**: c++98, c++03, c++11, c++14, c++1z
108
109  Change the standard version used when building the tests.
110
111.. option:: libcxx_site_config=<path/to/lit.site.cfg>
112
113  Specify the site configuration to use when running the tests.  This option
114  overrides the enviroment variable LIBCXX_SITE_CONFIG.
115
116.. option:: libcxx_headers=<path/to/headers>
117
118  Specify the libc++ headers that are tested. By default the headers in the
119  source tree are used.
120
121.. option:: libcxx_library=<path/to/libc++.so>
122
123  Specify the libc++ library that is tested. By default the library in the
124  build directory is used. This option cannot be used when use_system_lib is
125  provided.
126
127.. option:: use_system_lib=<bool>
128
129  **Default**: False
130
131  Enable or disable testing against the installed version of libc++ library.
132  Note: This does not use the installed headers.
133
134.. option:: use_lit_shell=<bool>
135
136  Enable or disable the use of LIT's internal shell in ShTests. If the
137  environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
138  the default value. Otherwise the default value is True on Windows and False
139  on every other platform.
140
141.. option:: no_default_flags=<bool>
142
143  **Default**: False
144
145  Disable all default compile and link flags from being added. When this
146  option is used only flags specified using the compile_flags and link_flags
147  will be used.
148
149.. option:: compile_flags="<list-of-args>"
150
151  Specify additional compile flags as a space delimited string.
152  Note: This options should not be used to change the standard version used.
153
154.. option:: link_flags="<list-of-args>"
155
156  Specify additional link flags as a space delimited string.
157
158.. option:: debug_level=<level>
159
160  **Values**: 0, 1
161
162  Enable the use of debug mode. Level 0 enables assertions and level 1 enables
163  assertions and debugging of iterator misuse.
164
165.. option:: use_sanitizer=<sanitizer name>
166
167  **Values**: Memory, MemoryWithOrigins, Address, Undefined
168
169  Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
170  building libc++ then that sanitizer will be used by default.
171
172.. option:: color_diagnostics
173
174  Enable the use of colorized compile diagnostics. If the color_diagnostics
175  option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
176  present then color diagnostics will be enabled.
177
178
179Environment Variables
180---------------------
181
182.. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
183
184  Specify the site configuration to use when running the tests.
185  Also see :option:`libcxx_site_config`.
186
187.. envvar:: LIBCXX_COLOR_DIAGNOSTICS
188
189  If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt
190  to use color diagnostic outputs from the compiler.
191  Also see :option:`color_diagnostics`.
192