1===================================== 2Cross Translation Unit (CTU) Analysis 3===================================== 4 5Normally, static analysis works in the boundary of one translation unit (TU). 6However, with additional steps and configuration we can enable the analysis to inline the definition of a function from 7another TU. 8 9.. contents:: 10 :local: 11 12Overview 13________ 14CTU analysis can be used in a variety of ways. The importing of external TU definitions can work with pre-dumped PCH 15files or generating the necessary AST structure on-demand, during the analysis of the main TU. Driving the static 16analysis can also be implemented in multiple ways. The most direct way is to specify the necessary commandline options 17of the Clang frontend manually (and generate the prerequisite dependencies of the specific import method by hand). This 18process can be automated by other tools, like `CodeChecker <https://github.com/Ericsson/codechecker>`_ and scan-build-py 19(preference for the former). 20 21PCH-based analysis 22__________________ 23The analysis needs the PCH dumps of all the translations units used in the project. 24These can be generated by the Clang Frontend itself, and must be arranged in a specific way in the filesystem. 25The index, which maps symbols' USR names to PCH dumps containing them must also be generated by the 26`clang-extdef-mapping`. Entries in the index *must* have an `.ast` suffix if the goal 27is to use PCH-based analysis, as the lack of that extension signals that the entry is to be used as a source-file, and parsed on-demand. 28This tool uses a :doc:`compilation database <../../JSONCompilationDatabase>` to 29determine the compilation flags used. 30The analysis invocation must be provided with the directory which contains the dumps and the mapping files. 31 32 33Manual CTU Analysis 34################### 35Let's consider these source files in our minimal example: 36 37.. code-block:: cpp 38 39 // main.cpp 40 int foo(); 41 42 int main() { 43 return 3 / foo(); 44 } 45 46.. code-block:: cpp 47 48 // foo.cpp 49 int foo() { 50 return 0; 51 } 52 53And a compilation database: 54 55.. code-block:: bash 56 57 [ 58 { 59 "directory": "/path/to/your/project", 60 "command": "clang++ -c foo.cpp -o foo.o", 61 "file": "foo.cpp" 62 }, 63 { 64 "directory": "/path/to/your/project", 65 "command": "clang++ -c main.cpp -o main.o", 66 "file": "main.cpp" 67 } 68 ] 69 70We'd like to analyze `main.cpp` and discover the division by zero bug. 71In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file 72of `foo.cpp`: 73 74.. code-block:: bash 75 76 $ pwd $ /path/to/your/project 77 $ clang++ -emit-ast -o foo.cpp.ast foo.cpp 78 $ # Check that the .ast file is generated: 79 $ ls 80 compile_commands.json foo.cpp.ast foo.cpp main.cpp 81 $ 82 83The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the 84source files: 85 86.. code-block:: bash 87 88 $ clang-extdef-mapping -p . foo.cpp 89 c:@F@foo# /path/to/your/project/foo.cpp 90 $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt 91 92We have to modify `externalDefMap.txt` to contain the name of the `.ast` files instead of the source files: 93 94.. code-block:: bash 95 96 $ sed -i -e "s/.cpp/.cpp.ast/g" externalDefMap.txt 97 98We still have to further modify the `externalDefMap.txt` file to contain relative paths: 99 100.. code-block:: bash 101 102 $ sed -i -e "s|$(pwd)/||g" externalDefMap.txt 103 104Now everything is available for the CTU analysis. 105We have to feed Clang with CTU specific extra arguments: 106 107.. code-block:: bash 108 109 $ pwd 110 /path/to/your/project 111 $ clang++ --analyze \ 112 -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \ 113 -Xclang -analyzer-config -Xclang ctu-dir=. \ 114 -Xclang -analyzer-output=plist-multi-file \ 115 main.cpp 116 main.cpp:5:12: warning: Division by zero 117 return 3 / foo(); 118 ~~^~~~~~~ 119 1 warning generated. 120 $ # The plist file with the result is generated. 121 $ ls -F 122 compile_commands.json externalDefMap.txt foo.ast foo.cpp foo.cpp.ast main.cpp main.plist 123 $ 124 125This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use 126`CodeChecker` or `scan-build-py`. 127 128Automated CTU Analysis with CodeChecker 129####################################### 130The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang. 131Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes: 132 133.. code-block:: bash 134 135 $ CodeChecker analyze --ctu compile_commands.json -o reports 136 $ ls -F 137 compile_commands.json foo.cpp foo.cpp.ast main.cpp reports/ 138 $ tree reports 139 reports 140 ├── compile_cmd.json 141 ├── compiler_info.json 142 ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist 143 ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist 144 ├── metadata.json 145 └── unique_compile_commands.json 146 147 0 directories, 6 files 148 $ 149 150The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools. 151E.g. one may use `CodeChecker parse` to view the results in command line: 152 153.. code-block:: bash 154 155 $ CodeChecker parse reports 156 [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero] 157 return 3 / foo(); 158 ^ 159 160 Found 1 defect(s) in main.cpp 161 162 163 ----==== Summary ====---- 164 ----------------------- 165 Filename | Report count 166 ----------------------- 167 main.cpp | 1 168 ----------------------- 169 ----------------------- 170 Severity | Report count 171 ----------------------- 172 HIGH | 1 173 ----------------------- 174 ----=================---- 175 Total number of reports: 1 176 ----=================---- 177 178Or we can use `CodeChecker parse -e html` to export the results into HTML format: 179 180.. code-block:: bash 181 182 $ CodeChecker parse -e html -o html_out reports 183 $ firefox html_out/index.html 184 185Automated CTU Analysis with scan-build-py (don't do it) 186############################################################# 187We actively develop CTU with CodeChecker as the driver for this feature, `scan-build-py` is not actively developed for CTU. 188`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only. 189 190Example usage of scan-build-py: 191 192.. code-block:: bash 193 194 $ /your/path/to/llvm-project/clang/tools/scan-build-py/bin/analyze-build --ctu 195 analyze-build: Run 'scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk' to examine bug reports. 196 $ /your/path/to/llvm-project/clang/tools/scan-view/bin/scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk 197 Starting scan-view at: http://127.0.0.1:8181 198 Use Ctrl-C to exit. 199 [6336:6431:0717/175357.633914:ERROR:browser_process_sub_thread.cc(209)] Waited 5 ms for network service 200 Opening in existing browser session. 201 ^C 202 $ 203 204On-demand analysis 205__________________ 206The analysis produces the necessary AST structure of external TUs during analysis. This requires the 207exact compiler invocations for each TU, which can be generated by hand, or by tools driving the analyzer. 208The compiler invocation is a shell command that could be used to compile the TU-s main source file. 209The mapping from absolute source file paths of a TU to lists of compilation command segments used to 210compile said TU are given in YAML format referred to as `invocation list`, and must be passed as an 211analyer-config argument. 212The index, which maps function USR names to source files containing them must also be generated by the 213`clang-extdef-mapping`. Entries in the index must *not* have an `.ast` suffix if the goal 214is to use On-demand analysis, as that extension signals that the entry is to be used as an PCH-dump. 215The mapping of external definitions implicitly uses a 216:doc:`compilation database <../../JSONCompilationDatabase>` to determine the compilation flags used. 217The analysis invocation must be provided with the directory which contains the mapping 218files, and the `invocation list` which is used to determine compiler flags. 219 220 221Manual CTU Analysis 222################### 223 224Let's consider these source files in our minimal example: 225 226.. code-block:: cpp 227 228 // main.cpp 229 int foo(); 230 231 int main() { 232 return 3 / foo(); 233 } 234 235.. code-block:: cpp 236 237 // foo.cpp 238 int foo() { 239 return 0; 240 } 241 242The compilation database: 243 244.. code-block:: bash 245 246 [ 247 { 248 "directory": "/path/to/your/project", 249 "command": "clang++ -c foo.cpp -o foo.o", 250 "file": "foo.cpp" 251 }, 252 { 253 "directory": "/path/to/your/project", 254 "command": "clang++ -c main.cpp -o main.o", 255 "file": "main.cpp" 256 } 257 ] 258 259The `invocation list`: 260 261.. code-block:: bash 262 263 "/path/to/your/project/foo.cpp": 264 - "clang++" 265 - "-c" 266 - "/path/to/your/project/foo.cpp" 267 - "-o" 268 - "/path/to/your/project/foo.o" 269 270 "/path/to/your/project/main.cpp": 271 - "clang++" 272 - "-c" 273 - "/path/to/your/project/main.cpp" 274 - "-o" 275 - "/path/to/your/project/main.o" 276 277We'd like to analyze `main.cpp` and discover the division by zero bug. 278As we are using On-demand mode, we only need to create a CTU index file which holds the `USR` name and location of 279external definitions in the source files: 280 281.. code-block:: bash 282 283 $ clang-extdef-mapping -p . foo.cpp 284 c:@F@foo# /path/to/your/project/foo.cpp 285 $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt 286 287Now everything is available for the CTU analysis. 288We have to feed Clang with CTU specific extra arguments: 289 290.. code-block:: bash 291 292 $ pwd 293 /path/to/your/project 294 $ clang++ --analyze \ 295 -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \ 296 -Xclang -analyzer-config -Xclang ctu-dir=. \ 297 -Xclang -analyzer-config -Xclang ctu-invocation-list=invocations.yaml \ 298 -Xclang -analyzer-output=plist-multi-file \ 299 main.cpp 300 main.cpp:5:12: warning: Division by zero 301 return 3 / foo(); 302 ~~^~~~~~~ 303 1 warning generated. 304 $ # The plist file with the result is generated. 305 $ ls -F 306 compile_commands.json externalDefMap.txt foo.cpp main.cpp main.plist 307 $ 308 309This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use 310`CodeChecker` or `scan-build-py`. 311 312Automated CTU Analysis with CodeChecker 313####################################### 314The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang. 315Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes: 316 317.. code-block:: bash 318 319 $ CodeChecker analyze --ctu --ctu-ast-loading-mode on-demand compile_commands.json -o reports 320 $ ls -F 321 compile_commands.json foo.cpp main.cpp reports/ 322 $ tree reports 323 reports 324 ├── compile_cmd.json 325 ├── compiler_info.json 326 ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist 327 ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist 328 ├── metadata.json 329 └── unique_compile_commands.json 330 331 0 directories, 6 files 332 $ 333 334The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools. 335E.g. one may use `CodeChecker parse` to view the results in command line: 336 337.. code-block:: bash 338 339 $ CodeChecker parse reports 340 [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero] 341 return 3 / foo(); 342 ^ 343 344 Found 1 defect(s) in main.cpp 345 346 347 ----==== Summary ====---- 348 ----------------------- 349 Filename | Report count 350 ----------------------- 351 main.cpp | 1 352 ----------------------- 353 ----------------------- 354 Severity | Report count 355 ----------------------- 356 HIGH | 1 357 ----------------------- 358 ----=================---- 359 Total number of reports: 1 360 ----=================---- 361 362Or we can use `CodeChecker parse -e html` to export the results into HTML format: 363 364.. code-block:: bash 365 366 $ CodeChecker parse -e html -o html_out reports 367 $ firefox html_out/index.html 368 369Automated CTU Analysis with scan-build-py (don't do it) 370####################################################### 371We actively develop CTU with CodeChecker as the driver for feature, `scan-build-py` is not actively developed for CTU. 372`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only. 373 374Currently On-demand analysis is not supported with `scan-build-py`. 375 376