1==========================
2Source-based Code Coverage
3==========================
4
5.. contents::
6   :local:
7
8Introduction
9============
10
11This document explains how to use clang's source-based code coverage feature.
12It's called "source-based" because it operates on AST and preprocessor
13information directly. This allows it to generate very precise coverage data.
14
15Clang ships two other code coverage implementations:
16
17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
18  various sanitizers. It can provide up to edge-level coverage.
19
20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
21
22From this point onwards "code coverage" will refer to the source-based kind.
23
24The code coverage workflow
25==========================
26
27The code coverage workflow consists of three main steps:
28
29* Compiling with coverage enabled.
30
31* Running the instrumented program.
32
33* Creating coverage reports.
34
35The next few sections work through a complete, copy-'n-paste friendly example
36based on this program:
37
38.. code-block:: cpp
39
40    % cat <<EOF > foo.cc
41    #define BAR(x) ((x) || (x))
42    template <typename T> void foo(T x) {
43      for (unsigned I = 0; I < 10; ++I) { BAR(I); }
44    }
45    int main() {
46      foo<int>(0);
47      foo<float>(0);
48      return 0;
49    }
50    EOF
51
52Compiling with coverage enabled
53===============================
54
55To compile code with coverage enabled, pass ``-fprofile-instr-generate
56-fcoverage-mapping`` to the compiler:
57
58.. code-block:: console
59
60    # Step 1: Compile with coverage enabled.
61    % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
62
63Note that linking together code with and without coverage instrumentation is
64supported: any uninstrumented code simply won't be accounted for.
65
66Running the instrumented program
67================================
68
69The next step is to run the instrumented program. When the program exits it
70will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
71environment variable. If that variable does not exist, the profile is written
72to ``default.profraw`` in the current directory of the program. If
73``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing
74directory structure will be created.  Additionally, the following special
75**pattern strings** are rewritten:
76
77* "%p" expands out to the process ID.
78
79* "%h" expands out to the hostname of the machine running the program.
80
81.. code-block:: console
82
83    # Step 2: Run the program.
84    % LLVM_PROFILE_FILE="foo.profraw" ./foo
85
86Creating coverage reports
87=========================
88
89Raw profiles have to be **indexed** before they can be used to generate
90coverage reports. This is done using the "merge" tool in ``llvm-profdata``, so
91named because it can combine and index profiles at the same time:
92
93.. code-block:: console
94
95    # Step 3(a): Index the raw profile.
96    % llvm-profdata merge -sparse foo.profraw -o foo.profdata
97
98There are multiple different ways to render coverage reports. One option is to
99generate a line-oriented report:
100
101.. code-block:: console
102
103    # Step 3(b): Create a line-oriented coverage report.
104    % llvm-cov show ./foo -instr-profile=foo.profdata
105
106To demangle any C++ identifiers in the ouput, use:
107
108.. code-block:: console
109
110    % llvm-cov show ./foo -instr-profile=foo.profdata | c++filt -n
111
112This report includes a summary view as well as dedicated sub-views for
113templated functions and their instantiations. For our example program, we get
114distinct views for ``foo<int>(...)`` and ``foo<float>(...)``.  If
115``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
116region counts (even in macro expansions):
117
118.. code-block:: cpp
119
120       20|    1|#define BAR(x) ((x) || (x))
121                               ^20     ^2
122        2|    2|template <typename T> void foo(T x) {
123       22|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
124                                       ^22     ^20  ^20^20
125        2|    4|}
126    ------------------
127    | void foo<int>(int):
128    |      1|    2|template <typename T> void foo(T x) {
129    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
130    |                                     ^11     ^10  ^10^10
131    |      1|    4|}
132    ------------------
133    | void foo<float>(int):
134    |      1|    2|template <typename T> void foo(T x) {
135    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
136    |                                     ^11     ^10  ^10^10
137    |      1|    4|}
138    ------------------
139
140It's possible to generate a file-level summary of coverage statistics (instead
141of a line-oriented report) with:
142
143.. code-block:: console
144
145    # Step 3(c): Create a coverage summary.
146    % llvm-cov report ./foo -instr-profile=foo.profdata
147    Filename                    Regions    Miss   Cover Functions  Executed
148    -----------------------------------------------------------------------
149    /tmp/foo.cc                      13       0 100.00%         3   100.00%
150    -----------------------------------------------------------------------
151    TOTAL                            13       0 100.00%         3   100.00%
152
153A few final notes:
154
155* The ``-sparse`` flag is optional but can result in dramatically smaller
156  indexed profiles. This option should not be used if the indexed profile will
157  be reused for PGO.
158
159* Raw profiles can be discarded after they are indexed. Advanced use of the
160  profile runtime library allows an instrumented program to merge profiling
161  information directly into an existing raw profile on disk. The details are
162  out of scope.
163
164* The ``llvm-profdata`` tool can be used to merge together multiple raw or
165  indexed profiles. To combine profiling data from multiple runs of a program,
166  try e.g:
167
168  .. code-block:: console
169
170      % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
171
172Format compatibility guarantees
173===============================
174
175* There are no backwards or forwards compatibility guarantees for the raw
176  profile format. Raw profiles may be dependent on the specific compiler
177  revision used to generate them. It's inadvisable to store raw profiles for
178  long periods of time.
179
180* Tools must retain **backwards** compatibility with indexed profile formats.
181  These formats are not forwards-compatible: i.e, a tool which uses format
182  version X will not be able to understand format version (X+k).
183
184* There is a third format in play: the format of the coverage mappings emitted
185  into instrumented binaries. Tools must retain **backwards** compatibility
186  with these formats. These formats are not forwards-compatible.
187
188Drawbacks and limitations
189=========================
190
191* Code coverage does not handle unpredictable changes in control flow or stack
192  unwinding in the presence of exceptions precisely. Consider the following
193  function:
194
195  .. code-block:: cpp
196
197      int f() {
198        may_throw();
199        return 0;
200      }
201
202  If the call to ``may_throw()`` propagates an exception into ``f``, the code
203  coverage tool may mark the ``return`` statement as executed even though it is
204  not. A call to ``longjmp()`` can have similar effects.
205