1================================= 2LLVM Testing Infrastructure Guide 3================================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 TestSuiteMakefileGuide 12 13Overview 14======== 15 16This document is the reference manual for the LLVM testing 17infrastructure. It documents the structure of the LLVM testing 18infrastructure, the tools needed to use it, and how to add and run 19tests. 20 21Requirements 22============ 23 24In order to use the LLVM testing infrastructure, you will need all of the 25software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or 26later. 27 28LLVM testing infrastructure organization 29======================================== 30 31The LLVM testing infrastructure contains two major categories of tests: 32regression tests and whole programs. The regression tests are contained 33inside the LLVM repository itself under ``llvm/test`` and are expected 34to always pass -- they should be run before every commit. 35 36The whole programs tests are referred to as the "LLVM test suite" (or 37"test-suite") and are in the ``test-suite`` module in subversion. For 38historical reasons, these tests are also referred to as the "nightly 39tests" in places, which is less ambiguous than "test-suite" and remains 40in use although we run them much more often than nightly. 41 42Regression tests 43---------------- 44 45The regression tests are small pieces of code that test a specific 46feature of LLVM or trigger a specific bug in LLVM. The language they are 47written in depends on the part of LLVM being tested. These tests are driven by 48the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and 49are located in the ``llvm/test`` directory. 50 51Typically when a bug is found in LLVM, a regression test containing just 52enough code to reproduce the problem should be written and placed 53somewhere underneath this directory. For example, it can be a small 54piece of LLVM IR distilled from an actual application or benchmark. 55 56``test-suite`` 57-------------- 58 59The test suite contains whole programs, which are pieces of code which 60can be compiled and linked into a stand-alone program that can be 61executed. These programs are generally written in high level languages 62such as C or C++. 63 64These programs are compiled using a user specified compiler and set of 65flags, and then executed to capture the program output and timing 66information. The output of these programs is compared to a reference 67output to ensure that the program is being compiled correctly. 68 69In addition to compiling and executing programs, whole program tests 70serve as a way of benchmarking LLVM performance, both in terms of the 71efficiency of the programs generated as well as the speed with which 72LLVM compiles, optimizes, and generates code. 73 74The test-suite is located in the ``test-suite`` Subversion module. 75 76Debugging Information tests 77--------------------------- 78 79The test suite contains tests to check quality of debugging information. 80The test are written in C based languages or in LLVM assembly language. 81 82These tests are compiled and run under a debugger. The debugger output 83is checked to validate of debugging information. See README.txt in the 84test suite for more information . This test suite is located in the 85``debuginfo-tests`` Subversion module. 86 87Quick start 88=========== 89 90The tests are located in two separate Subversion modules. The 91regressions tests are in the main "llvm" module under the directory 92``llvm/test`` (so you get these tests for free with the main LLVM tree). 93Use ``make check-all`` to run the regression tests after building LLVM. 94 95The more comprehensive test suite that includes whole programs in C and C++ 96is in the ``test-suite`` module. See :ref:`test-suite Quickstart 97<test-suite-quickstart>` for more information on running these tests. 98 99Regression tests 100---------------- 101 102To run all of the LLVM regression tests use the check-llvm target: 103 104.. code-block:: bash 105 106 % make check-llvm 107 108If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you 109can run the LLVM and Clang tests simultaneously using: 110 111.. code-block:: bash 112 113 % make check-all 114 115To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make 116variable to pass the required options to lit. For example, you can use: 117 118.. code-block:: bash 119 120 % make check LIT_ARGS="-v --vg --vg-leak" 121 122to enable testing with valgrind and with leak checking enabled. 123 124To run individual tests or subsets of tests, you can use the ``llvm-lit`` 125script which is built as part of LLVM. For example, to run the 126``Integer/BitPacked.ll`` test by itself you can run: 127 128.. code-block:: bash 129 130 % llvm-lit ~/llvm/test/Integer/BitPacked.ll 131 132or to run all of the ARM CodeGen tests: 133 134.. code-block:: bash 135 136 % llvm-lit ~/llvm/test/CodeGen/ARM 137 138For more information on using the :program:`lit` tool, see ``llvm-lit --help`` 139or the :doc:`lit man page <CommandGuide/lit>`. 140 141Debugging Information tests 142--------------------------- 143 144To run debugging information tests simply checkout the tests inside 145clang/test directory. 146 147.. code-block:: bash 148 149 % cd clang/test 150 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests 151 152These tests are already set up to run as part of clang regression tests. 153 154Regression test structure 155========================= 156 157The LLVM regression tests are driven by :program:`lit` and are located in the 158``llvm/test`` directory. 159 160This directory contains a large array of small tests that exercise 161various features of LLVM and to ensure that regressions do not occur. 162The directory is broken into several sub-directories, each focused on a 163particular area of LLVM. 164 165Writing new regression tests 166---------------------------- 167 168The regression test structure is very simple, but does require some 169information to be set. This information is gathered via ``configure`` 170and is written to a file, ``test/lit.site.cfg`` in the build directory. 171The ``llvm/test`` Makefile does this work for you. 172 173In order for the regression tests to work, each directory of tests must 174have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine 175how to run the tests. This file is just Python code and thus is very 176flexible, but we've standardized it for the LLVM regression tests. If 177you're adding a directory of tests, just copy ``lit.local.cfg`` from 178another directory to get running. The standard ``lit.local.cfg`` simply 179specifies which files to look in for tests. Any directory that contains 180only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit 181documentation <CommandGuide/lit>` for more information. 182 183Each test file must contain lines starting with "RUN:" that tell :program:`lit` 184how to run it. If there are no RUN lines, :program:`lit` will issue an error 185while running a test. 186 187RUN lines are specified in the comments of the test program using the 188keyword ``RUN`` followed by a colon, and lastly the command (pipeline) 189to execute. Together, these lines form the "script" that :program:`lit` 190executes to run the test case. The syntax of the RUN lines is similar to a 191shell's syntax for pipelines including I/O redirection and variable 192substitution. However, even though these lines may *look* like a shell 193script, they are not. RUN lines are interpreted by :program:`lit`. 194Consequently, the syntax differs from shell in a few ways. You can specify 195as many RUN lines as needed. 196 197:program:`lit` performs substitution on each RUN line to replace LLVM tool names 198with the full paths to the executable built for each tool (in 199``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does 200not invoke any stray LLVM tools in the user's path during testing. 201 202Each RUN line is executed on its own, distinct from other lines unless 203its last character is ``\``. This continuation character causes the RUN 204line to be concatenated with the next one. In this way you can build up 205long pipelines of commands without making huge line lengths. The lines 206ending in ``\`` are concatenated until a RUN line that doesn't end in 207``\`` is found. This concatenated set of RUN lines then constitutes one 208execution. :program:`lit` will substitute variables and arrange for the pipeline 209to be executed. If any process in the pipeline fails, the entire line (and 210test case) fails too. 211 212Below is an example of legal RUN lines in a ``.ll`` file: 213 214.. code-block:: llvm 215 216 ; RUN: llvm-as < %s | llvm-dis > %t1 217 ; RUN: llvm-dis < %s.bc-13 > %t2 218 ; RUN: diff %t1 %t2 219 220As with a Unix shell, the RUN lines permit pipelines and I/O 221redirection to be used. 222 223There are some quoting rules that you must pay attention to when writing 224your RUN lines. In general nothing needs to be quoted. :program:`lit` won't 225strip off any quote characters so they will get passed to the invoked program. 226To avoid this use curly braces to tell :program:`lit` that it should treat 227everything enclosed as one value. 228 229In general, you should strive to keep your RUN lines as simple as possible, 230using them only to run tools that generate textual output you can then examine. 231The recommended way to examine output to figure out if the test passes is using 232the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN 233lines is deprecated - please do not send or commit patches that use it.]* 234 235Put related tests into a single file rather than having a separate file per 236test. Check if there are files already covering your feature and consider 237adding your code there instead of creating a new file. 238 239Extra files 240----------- 241 242If your test requires extra files besides the file containing the ``RUN:`` 243lines, the idiomatic place to put them is in a subdirectory ``Inputs``. 244You can then refer to the extra files as ``%S/Inputs/foo.bar``. 245 246For example, consider ``test/Linker/ident.ll``. The directory structure is 247as follows:: 248 249 test/ 250 Linker/ 251 ident.ll 252 Inputs/ 253 ident.a.ll 254 ident.b.ll 255 256For convenience, these are the contents: 257 258.. code-block:: llvm 259 260 ;;;;; ident.ll: 261 262 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s 263 264 ; Verify that multiple input llvm.ident metadata are linked together. 265 266 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2} 267 ; CHECK-DAG: "Compiler V1" 268 ; CHECK-DAG: "Compiler V2" 269 ; CHECK-DAG: "Compiler V3" 270 271 ;;;;; Inputs/ident.a.ll: 272 273 !llvm.ident = !{!0, !1} 274 !0 = metadata !{metadata !"Compiler V1"} 275 !1 = metadata !{metadata !"Compiler V2"} 276 277 ;;;;; Inputs/ident.b.ll: 278 279 !llvm.ident = !{!0} 280 !0 = metadata !{metadata !"Compiler V3"} 281 282For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't 283actually participate in the test besides holding the ``RUN:`` lines. 284 285.. note:: 286 287 Some existing tests use ``RUN: true`` in extra files instead of just 288 putting the extra files in an ``Inputs/`` directory. This pattern is 289 deprecated. 290 291Fragile tests 292------------- 293 294It is easy to write a fragile test that would fail spuriously if the tool being 295tested outputs a full path to the input file. For example, :program:`opt` by 296default outputs a ``ModuleID``: 297 298.. code-block:: console 299 300 $ cat example.ll 301 define i32 @main() nounwind { 302 ret i32 0 303 } 304 305 $ opt -S /path/to/example.ll 306 ; ModuleID = '/path/to/example.ll' 307 308 define i32 @main() nounwind { 309 ret i32 0 310 } 311 312``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: 313 314.. code-block:: llvm 315 316 ; RUN: opt -S %s | FileCheck 317 318 define i32 @main() nounwind { 319 ; CHECK-NOT: load 320 ret i32 0 321 } 322 323This test will fail if placed into a ``download`` directory. 324 325To make your tests robust, always use ``opt ... < %s`` in the RUN line. 326:program:`opt` does not output a ``ModuleID`` when input comes from stdin. 327 328Platform-Specific Tests 329----------------------- 330 331Whenever adding tests that require the knowledge of a specific platform, 332either related to code generated, specific output or back-end features, 333you must make sure to isolate the features, so that buildbots that 334run on different architectures (and don't even compile all back-ends), 335don't fail. 336 337The first problem is to check for target-specific output, for example sizes 338of structures, paths and architecture names, for example: 339 340* Tests containing Windows paths will fail on Linux and vice-versa. 341* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. 342* Tests where the debug information calculates the size of types and structures. 343 344Also, if the test rely on any behaviour that is coded in any back-end, it must 345go in its own directory. So, for instance, code generator tests for ARM go 346into ``test/CodeGen/ARM`` and so on. Those directories contain a special 347``lit`` configuration file that ensure all tests in that directory will 348only run if a specific back-end is compiled and available. 349 350For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: 351 352.. code-block:: python 353 354 config.suffixes = ['.ll', '.c', '.cpp', '.test'] 355 if not 'ARM' in config.root.targets: 356 config.unsupported = True 357 358Other platform-specific tests are those that depend on a specific feature 359of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. 360 361For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture 362variants: 363 364.. code-block:: llvm 365 366 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 367 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 368 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 369 370And the checks are different: 371 372.. code-block:: llvm 373 374 ; SSE2: @test1 375 ; SSE2: psubusw LCPI0_0(%rip), %xmm0 376 ; AVX1: @test1 377 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 378 ; AVX2: @test1 379 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 380 381So, if you're testing for a behaviour that you know is platform-specific or 382depends on special features of sub-architectures, you must add the specific 383triple, test with the specific FileCheck and put it into the specific 384directory that will filter out all other architectures. 385 386 387Substitutions 388------------- 389 390Besides replacing LLVM tool names the following substitutions are performed in 391RUN lines: 392 393``%%`` 394 Replaced by a single ``%``. This allows escaping other substitutions. 395 396``%s`` 397 File path to the test case's source. This is suitable for passing on the 398 command line as the input to an LLVM tool. 399 400 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` 401 402``%S`` 403 Directory path to the test case's source. 404 405 Example: ``/home/user/llvm/test/MC/ELF`` 406 407``%t`` 408 File path to a temporary file name that could be used for this test case. 409 The file name won't conflict with other test cases. You can append to it 410 if you need multiple temporaries. This is useful as the destination of 411 some redirected output. 412 413 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` 414 415``%T`` 416 Directory of ``%t``. 417 418 Example: ``/home/user/llvm.build/test/MC/ELF/Output`` 419 420``%{pathsep}`` 421 422 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). 423 424 425**LLVM-specific substitutions:** 426 427``%shlibext`` 428 The suffix for the host platforms shared library files. This includes the 429 period as the first character. 430 431 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) 432 433``%exeext`` 434 The suffix for the host platforms executable files. This includes the 435 period as the first character. 436 437 Example: ``.exe`` (Windows), empty on Linux. 438 439``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` 440 The number of the line where this substitution is used, with an optional 441 integer offset. This can be used in tests with multiple RUN lines, which 442 reference test file's line numbers. 443 444 445**Clang-specific substitutions:** 446 447``%clang`` 448 Invokes the Clang driver. 449 450``%clang_cpp`` 451 Invokes the Clang driver for C++. 452 453``%clang_cl`` 454 Invokes the CL-compatible Clang driver. 455 456``%clangxx`` 457 Invokes the G++-compatible Clang driver. 458 459``%clang_cc1`` 460 Invokes the Clang frontend. 461 462``%itanium_abi_triple``, ``%ms_abi_triple`` 463 These substitutions can be used to get the current target triple adjusted to 464 the desired ABI. For example, if the test suite is running with the 465 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to 466 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without 467 constraining it to a specific triple. 468 469To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. 470 471 472Options 473------- 474 475The llvm lit configuration allows to customize some things with user options: 476 477``llc``, ``opt``, ... 478 Substitute the respective llvm tool name with a custom command line. This 479 allows to specify custom paths and default arguments for these tools. 480 Example: 481 482 % llvm-lit "-Dllc=llc -verify-machineinstrs" 483 484``run_long_tests`` 485 Enable the execution of long running tests. 486 487``llvm_site_config`` 488 Load the specified lit configuration instead of the default one. 489 490 491Other Features 492-------------- 493 494To make RUN line writing easier, there are several helper programs. These 495helpers are in the PATH when running tests, so you can just call them using 496their name. For example: 497 498``not`` 499 This program runs its arguments and then inverts the result code from it. 500 Zero result codes become 1. Non-zero result codes become 0. 501 502Sometimes it is necessary to mark a test case as "expected fail" or 503XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` 504on a line near the top of the file. This signals that the test case 505should succeed if the test fails. Such test cases are counted separately 506by the testing tool. To specify an expected fail, use the XFAIL keyword 507in the comments of the test program followed by a colon and one or more 508failure patterns. Each failure pattern can be either ``*`` (to specify 509fail everywhere), or a part of a target triple (indicating the test 510should fail on that platform), or the name of a configurable feature 511(for example, ``loadable_module``). If there is a match, the test is 512expected to fail. If not, the test is expected to succeed. To XFAIL 513everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` 514line: 515 516.. code-block:: llvm 517 518 ; XFAIL: darwin,sun 519 520To make the output more useful, :program:`lit` will scan 521the lines of the test case for ones that contain a pattern that matches 522``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number 523that is related to the test case. The number after "PR" specifies the 524LLVM bugzilla number. When a PR number is specified, it will be used in 525the pass/fail reporting. This is useful to quickly get some context when 526a test fails. 527 528Finally, any line that contains "END." will cause the special 529interpretation of lines to terminate. This is generally done right after 530the last RUN: line. This has two side effects: 531 532(a) it prevents special interpretation of lines that are part of the test 533 program, not the instructions to the test case, and 534 535(b) it speeds things up for really big test cases by avoiding 536 interpretation of the remainder of the file. 537 538``test-suite`` Overview 539======================= 540 541The ``test-suite`` module contains a number of programs that can be 542compiled and executed. The ``test-suite`` includes reference outputs for 543all of the programs, so that the output of the executed program can be 544checked for correctness. 545 546``test-suite`` tests are divided into three types of tests: MultiSource, 547SingleSource, and External. 548 549- ``test-suite/SingleSource`` 550 551 The SingleSource directory contains test programs that are only a 552 single source file in size. These are usually small benchmark 553 programs or small programs that calculate a particular value. Several 554 such programs are grouped together in each directory. 555 556- ``test-suite/MultiSource`` 557 558 The MultiSource directory contains subdirectories which contain 559 entire programs with multiple source files. Large benchmarks and 560 whole applications go here. 561 562- ``test-suite/External`` 563 564 The External directory contains Makefiles for building code that is 565 external to (i.e., not distributed with) LLVM. The most prominent 566 members of this directory are the SPEC 95 and SPEC 2000 benchmark 567 suites. The ``External`` directory does not contain these actual 568 tests, but only the Makefiles that know how to properly compile these 569 programs from somewhere else. When using ``LNT``, use the 570 ``--test-externals`` option to include these tests in the results. 571 572.. _test-suite-quickstart: 573 574``test-suite`` Quickstart 575------------------------- 576 577The modern way of running the ``test-suite`` is focused on testing and 578benchmarking complete compilers using the 579`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. 580 581For more information on using LNT to execute the ``test-suite``, please 582see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ 583documentation. 584 585``test-suite`` Makefiles 586------------------------ 587 588Historically, the ``test-suite`` was executed using a complicated setup 589of Makefiles. The LNT based approach above is recommended for most 590users, but there are some testing scenarios which are not supported by 591the LNT approach. In addition, LNT currently uses the Makefile setup 592under the covers and so developers who are interested in how LNT works 593under the hood may want to understand the Makefile based setup. 594 595For more information on the ``test-suite`` Makefile setup, please see 596the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. 597