1==========================
2Source-based Code Coverage
3==========================
4
5.. contents::
6   :local:
7
8Introduction
9============
10
11This document explains how to use clang's source-based code coverage feature.
12It's called "source-based" because it operates on AST and preprocessor
13information directly. This allows it to generate very precise coverage data.
14
15Clang ships two other code coverage implementations:
16
17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
18  various sanitizers. It can provide up to edge-level coverage.
19
20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
21  This is enabled by ``-ftest-coverage`` or ``--coverage``.
22
23From this point onwards "code coverage" will refer to the source-based kind.
24
25The code coverage workflow
26==========================
27
28The code coverage workflow consists of three main steps:
29
30* Compiling with coverage enabled.
31
32* Running the instrumented program.
33
34* Creating coverage reports.
35
36The next few sections work through a complete, copy-'n-paste friendly example
37based on this program:
38
39.. code-block:: cpp
40
41    % cat <<EOF > foo.cc
42    #define BAR(x) ((x) || (x))
43    template <typename T> void foo(T x) {
44      for (unsigned I = 0; I < 10; ++I) { BAR(I); }
45    }
46    int main() {
47      foo<int>(0);
48      foo<float>(0);
49      return 0;
50    }
51    EOF
52
53Compiling with coverage enabled
54===============================
55
56To compile code with coverage enabled, pass ``-fprofile-instr-generate
57-fcoverage-mapping`` to the compiler:
58
59.. code-block:: console
60
61    # Step 1: Compile with coverage enabled.
62    % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
63
64Note that linking together code with and without coverage instrumentation is
65supported. Uninstrumented code simply won't be accounted for in reports.
66
67Running the instrumented program
68================================
69
70The next step is to run the instrumented program. When the program exits it
71will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
72environment variable. If that variable does not exist, the profile is written
73to ``default.profraw`` in the current directory of the program. If
74``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing
75directory structure will be created.  Additionally, the following special
76**pattern strings** are rewritten:
77
78* "%p" expands out to the process ID.
79
80* "%h" expands out to the hostname of the machine running the program.
81
82* "%Nm" expands out to the instrumented binary's signature. When this pattern
83  is specified, the runtime creates a pool of N raw profiles which are used for
84  on-line profile merging. The runtime takes care of selecting a raw profile
85  from the pool, locking it, and updating it before the program exits.  If N is
86  not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must
87  be between 1 and 9. The merge pool specifier can only occur once per filename
88  pattern.
89
90* "%c" expands out to nothing, but enables a mode in which profile counter
91  updates are continuously synced to a file. This means that if the
92  instrumented program crashes, or is killed by a signal, perfect coverage
93  information can still be recovered. Continuous mode is not yet compatible with
94  the "%Nm" merging mode described above, does not support value profiling for
95  PGO, and is only supported on Darwin. Support for Linux may be mostly
96  complete but requires testing, and support for Fuchsia/Windows may require
97  more extensive changes: please get involved if you are interested in porting
98  this feature.
99
100.. code-block:: console
101
102    # Step 2: Run the program.
103    % LLVM_PROFILE_FILE="foo.profraw" ./foo
104
105Creating coverage reports
106=========================
107
108Raw profiles have to be **indexed** before they can be used to generate
109coverage reports. This is done using the "merge" tool in ``llvm-profdata``
110(which can combine multiple raw profiles and index them at the same time):
111
112.. code-block:: console
113
114    # Step 3(a): Index the raw profile.
115    % llvm-profdata merge -sparse foo.profraw -o foo.profdata
116
117There are multiple different ways to render coverage reports. The simplest
118option is to generate a line-oriented report:
119
120.. code-block:: console
121
122    # Step 3(b): Create a line-oriented coverage report.
123    % llvm-cov show ./foo -instr-profile=foo.profdata
124
125This report includes a summary view as well as dedicated sub-views for
126templated functions and their instantiations. For our example program, we get
127distinct views for ``foo<int>(...)`` and ``foo<float>(...)``.  If
128``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
129region counts (even in macro expansions):
130
131.. code-block:: none
132
133        1|   20|#define BAR(x) ((x) || (x))
134                               ^20     ^2
135        2|    2|template <typename T> void foo(T x) {
136        3|   22|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
137                                       ^22     ^20  ^20^20
138        4|    2|}
139    ------------------
140    | void foo<int>(int):
141    |      2|    1|template <typename T> void foo(T x) {
142    |      3|   11|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
143    |                                     ^11     ^10  ^10^10
144    |      4|    1|}
145    ------------------
146    | void foo<float>(int):
147    |      2|    1|template <typename T> void foo(T x) {
148    |      3|   11|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
149    |                                     ^11     ^10  ^10^10
150    |      4|    1|}
151    ------------------
152
153To generate a file-level summary of coverage statistics instead of a
154line-oriented report, try:
155
156.. code-block:: console
157
158    # Step 3(c): Create a coverage summary.
159    % llvm-cov report ./foo -instr-profile=foo.profdata
160    Filename           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
161    --------------------------------------------------------------------------------------------------------------------------------------
162    /tmp/foo.cc             13                 0   100.00%           3                 0   100.00%          13                 0   100.00%
163    --------------------------------------------------------------------------------------------------------------------------------------
164    TOTAL                   13                 0   100.00%           3                 0   100.00%          13                 0   100.00%
165
166The ``llvm-cov`` tool supports specifying a custom demangler, writing out
167reports in a directory structure, and generating html reports. For the full
168list of options, please refer to the `command guide
169<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_.
170
171A few final notes:
172
173* The ``-sparse`` flag is optional but can result in dramatically smaller
174  indexed profiles. This option should not be used if the indexed profile will
175  be reused for PGO.
176
177* Raw profiles can be discarded after they are indexed. Advanced use of the
178  profile runtime library allows an instrumented program to merge profiling
179  information directly into an existing raw profile on disk. The details are
180  out of scope.
181
182* The ``llvm-profdata`` tool can be used to merge together multiple raw or
183  indexed profiles. To combine profiling data from multiple runs of a program,
184  try e.g:
185
186  .. code-block:: console
187
188      % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
189
190Exporting coverage data
191=======================
192
193Coverage data can be exported into JSON using the ``llvm-cov export``
194sub-command. There is a comprehensive reference which defines the structure of
195the exported data at a high level in the llvm-cov source code.
196
197Interpreting reports
198====================
199
200There are four statistics tracked in a coverage summary:
201
202* Function coverage is the percentage of functions which have been executed at
203  least once. A function is considered to be executed if any of its
204  instantiations are executed.
205
206* Instantiation coverage is the percentage of function instantiations which
207  have been executed at least once. Template functions and static inline
208  functions from headers are two kinds of functions which may have multiple
209  instantiations.
210
211* Line coverage is the percentage of code lines which have been executed at
212  least once. Only executable lines within function bodies are considered to be
213  code lines.
214
215* Region coverage is the percentage of code regions which have been executed at
216  least once. A code region may span multiple lines (e.g in a large function
217  body with no control flow). However, it's also possible for a single line to
218  contain multiple code regions (e.g in "return x || y && z").
219
220Of these four statistics, function coverage is usually the least granular while
221region coverage is the most granular. The project-wide totals for each
222statistic are listed in the summary.
223
224Format compatibility guarantees
225===============================
226
227* There are no backwards or forwards compatibility guarantees for the raw
228  profile format. Raw profiles may be dependent on the specific compiler
229  revision used to generate them. It's inadvisable to store raw profiles for
230  long periods of time.
231
232* Tools must retain **backwards** compatibility with indexed profile formats.
233  These formats are not forwards-compatible: i.e, a tool which uses format
234  version X will not be able to understand format version (X+k).
235
236* Tools must also retain **backwards** compatibility with the format of the
237  coverage mappings emitted into instrumented binaries. These formats are not
238  forwards-compatible.
239
240* The JSON coverage export format has a (major, minor, patch) version triple.
241  Only a major version increment indicates a backwards-incompatible change. A
242  minor version increment is for added functionality, and patch version
243  increments are for bugfixes.
244
245Using the profiling runtime without static initializers
246=======================================================
247
248By default the compiler runtime uses a static initializer to determine the
249profile output path and to register a writer function. To collect profiles
250without using static initializers, do this manually:
251
252* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared
253  library and executable. When the linker finds a definition of this symbol, it
254  knows to skip loading the object which contains the profiling runtime's
255  static initializer.
256
257* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it
258  once from each instrumented executable. This function parses
259  ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files
260  at that path. To get the same behavior without truncating existing files,
261  pass a filename pattern string to ``void __llvm_profile_set_filename(char
262  *)``.  These calls can be placed anywhere so long as they precede all calls
263  to ``__llvm_profile_write_file``.
264
265* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write
266  out a profile. This function returns 0 when it succeeds, and a non-zero value
267  otherwise. Calling this function multiple times appends profile data to an
268  existing on-disk raw profile.
269
270In C++ files, declare these as ``extern "C"``.
271
272Collecting coverage reports for the llvm project
273================================================
274
275To prepare a coverage report for llvm (and any of its sub-projects), add
276``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw
277profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html
278report, run ``llvm/utils/prepare-code-coverage-artifact.py``.
279
280To specify an alternate directory for raw profiles, use
281``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use
282``-DLLVM_PROFILE_MERGE_POOL_SIZE``.
283
284Drawbacks and limitations
285=========================
286
287* Prior to version 2.26, the GNU binutils BFD linker is not able link programs
288  compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode.  Possible
289  workarounds include disabling ``--gc-sections``, upgrading to a newer version
290  of BFD, or using the Gold linker.
291
292* Code coverage does not handle unpredictable changes in control flow or stack
293  unwinding in the presence of exceptions precisely. Consider the following
294  function:
295
296  .. code-block:: cpp
297
298      int f() {
299        may_throw();
300        return 0;
301      }
302
303  If the call to ``may_throw()`` propagates an exception into ``f``, the code
304  coverage tool may mark the ``return`` statement as executed even though it is
305  not. A call to ``longjmp()`` can have similar effects.
306