1================================= 2LLVM Testing Infrastructure Guide 3================================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 TestSuiteMakefileGuide 12 13Overview 14======== 15 16This document is the reference manual for the LLVM testing 17infrastructure. It documents the structure of the LLVM testing 18infrastructure, the tools needed to use it, and how to add and run 19tests. 20 21Requirements 22============ 23 24In order to use the LLVM testing infrastructure, you will need all of the 25software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or 26later. 27 28If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also 29need a development version of zlib (zlib1g-dev is known to work on several Linux 30distributions). 31 32LLVM testing infrastructure organization 33======================================== 34 35The LLVM testing infrastructure contains two major categories of tests: 36regression tests and whole programs. The regression tests are contained 37inside the LLVM repository itself under ``llvm/test`` and are expected 38to always pass -- they should be run before every commit. 39 40The whole programs tests are referred to as the "LLVM test suite" (or 41"test-suite") and are in the ``test-suite`` module in subversion. For 42historical reasons, these tests are also referred to as the "nightly 43tests" in places, which is less ambiguous than "test-suite" and remains 44in use although we run them much more often than nightly. 45 46Regression tests 47---------------- 48 49The regression tests are small pieces of code that test a specific 50feature of LLVM or trigger a specific bug in LLVM. The language they are 51written in depends on the part of LLVM being tested. These tests are driven by 52the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and 53are located in the ``llvm/test`` directory. 54 55Typically when a bug is found in LLVM, a regression test containing just 56enough code to reproduce the problem should be written and placed 57somewhere underneath this directory. For example, it can be a small 58piece of LLVM IR distilled from an actual application or benchmark. 59 60``test-suite`` 61-------------- 62 63The test suite contains whole programs, which are pieces of code which 64can be compiled and linked into a stand-alone program that can be 65executed. These programs are generally written in high level languages 66such as C or C++. 67 68These programs are compiled using a user specified compiler and set of 69flags, and then executed to capture the program output and timing 70information. The output of these programs is compared to a reference 71output to ensure that the program is being compiled correctly. 72 73In addition to compiling and executing programs, whole program tests 74serve as a way of benchmarking LLVM performance, both in terms of the 75efficiency of the programs generated as well as the speed with which 76LLVM compiles, optimizes, and generates code. 77 78The test-suite is located in the ``test-suite`` Subversion module. 79 80Debugging Information tests 81--------------------------- 82 83The test suite contains tests to check quality of debugging information. 84The test are written in C based languages or in LLVM assembly language. 85 86These tests are compiled and run under a debugger. The debugger output 87is checked to validate of debugging information. See README.txt in the 88test suite for more information . This test suite is located in the 89``debuginfo-tests`` Subversion module. 90 91Quick start 92=========== 93 94The tests are located in two separate Subversion modules. The 95regressions tests are in the main "llvm" module under the directory 96``llvm/test`` (so you get these tests for free with the main LLVM tree). 97Use ``make check-all`` to run the regression tests after building LLVM. 98 99The more comprehensive test suite that includes whole programs in C and C++ 100is in the ``test-suite`` module. See :ref:`test-suite Quickstart 101<test-suite-quickstart>` for more information on running these tests. 102 103Regression tests 104---------------- 105 106To run all of the LLVM regression tests use the check-llvm target: 107 108.. code-block:: bash 109 110 % make check-llvm 111 112If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you 113can run the LLVM and Clang tests simultaneously using: 114 115.. code-block:: bash 116 117 % make check-all 118 119To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make 120variable to pass the required options to lit. For example, you can use: 121 122.. code-block:: bash 123 124 % make check LIT_ARGS="-v --vg --vg-leak" 125 126to enable testing with valgrind and with leak checking enabled. 127 128To run individual tests or subsets of tests, you can use the ``llvm-lit`` 129script which is built as part of LLVM. For example, to run the 130``Integer/BitPacked.ll`` test by itself you can run: 131 132.. code-block:: bash 133 134 % llvm-lit ~/llvm/test/Integer/BitPacked.ll 135 136or to run all of the ARM CodeGen tests: 137 138.. code-block:: bash 139 140 % llvm-lit ~/llvm/test/CodeGen/ARM 141 142For more information on using the :program:`lit` tool, see ``llvm-lit --help`` 143or the :doc:`lit man page <CommandGuide/lit>`. 144 145Debugging Information tests 146--------------------------- 147 148To run debugging information tests simply checkout the tests inside 149clang/test directory. 150 151.. code-block:: bash 152 153 % cd clang/test 154 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests 155 156These tests are already set up to run as part of clang regression tests. 157 158Regression test structure 159========================= 160 161The LLVM regression tests are driven by :program:`lit` and are located in the 162``llvm/test`` directory. 163 164This directory contains a large array of small tests that exercise 165various features of LLVM and to ensure that regressions do not occur. 166The directory is broken into several sub-directories, each focused on a 167particular area of LLVM. 168 169Writing new regression tests 170---------------------------- 171 172The regression test structure is very simple, but does require some 173information to be set. This information is gathered via ``configure`` 174and is written to a file, ``test/lit.site.cfg`` in the build directory. 175The ``llvm/test`` Makefile does this work for you. 176 177In order for the regression tests to work, each directory of tests must 178have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine 179how to run the tests. This file is just Python code and thus is very 180flexible, but we've standardized it for the LLVM regression tests. If 181you're adding a directory of tests, just copy ``lit.local.cfg`` from 182another directory to get running. The standard ``lit.local.cfg`` simply 183specifies which files to look in for tests. Any directory that contains 184only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit 185documentation <CommandGuide/lit>` for more information. 186 187Each test file must contain lines starting with "RUN:" that tell :program:`lit` 188how to run it. If there are no RUN lines, :program:`lit` will issue an error 189while running a test. 190 191RUN lines are specified in the comments of the test program using the 192keyword ``RUN`` followed by a colon, and lastly the command (pipeline) 193to execute. Together, these lines form the "script" that :program:`lit` 194executes to run the test case. The syntax of the RUN lines is similar to a 195shell's syntax for pipelines including I/O redirection and variable 196substitution. However, even though these lines may *look* like a shell 197script, they are not. RUN lines are interpreted by :program:`lit`. 198Consequently, the syntax differs from shell in a few ways. You can specify 199as many RUN lines as needed. 200 201:program:`lit` performs substitution on each RUN line to replace LLVM tool names 202with the full paths to the executable built for each tool (in 203``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does 204not invoke any stray LLVM tools in the user's path during testing. 205 206Each RUN line is executed on its own, distinct from other lines unless 207its last character is ``\``. This continuation character causes the RUN 208line to be concatenated with the next one. In this way you can build up 209long pipelines of commands without making huge line lengths. The lines 210ending in ``\`` are concatenated until a RUN line that doesn't end in 211``\`` is found. This concatenated set of RUN lines then constitutes one 212execution. :program:`lit` will substitute variables and arrange for the pipeline 213to be executed. If any process in the pipeline fails, the entire line (and 214test case) fails too. 215 216Below is an example of legal RUN lines in a ``.ll`` file: 217 218.. code-block:: llvm 219 220 ; RUN: llvm-as < %s | llvm-dis > %t1 221 ; RUN: llvm-dis < %s.bc-13 > %t2 222 ; RUN: diff %t1 %t2 223 224As with a Unix shell, the RUN lines permit pipelines and I/O 225redirection to be used. 226 227There are some quoting rules that you must pay attention to when writing 228your RUN lines. In general nothing needs to be quoted. :program:`lit` won't 229strip off any quote characters so they will get passed to the invoked program. 230To avoid this use curly braces to tell :program:`lit` that it should treat 231everything enclosed as one value. 232 233In general, you should strive to keep your RUN lines as simple as possible, 234using them only to run tools that generate textual output you can then examine. 235The recommended way to examine output to figure out if the test passes is using 236the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN 237lines is deprecated - please do not send or commit patches that use it.]* 238 239Put related tests into a single file rather than having a separate file per 240test. Check if there are files already covering your feature and consider 241adding your code there instead of creating a new file. 242 243Extra files 244----------- 245 246If your test requires extra files besides the file containing the ``RUN:`` 247lines, the idiomatic place to put them is in a subdirectory ``Inputs``. 248You can then refer to the extra files as ``%S/Inputs/foo.bar``. 249 250For example, consider ``test/Linker/ident.ll``. The directory structure is 251as follows:: 252 253 test/ 254 Linker/ 255 ident.ll 256 Inputs/ 257 ident.a.ll 258 ident.b.ll 259 260For convenience, these are the contents: 261 262.. code-block:: llvm 263 264 ;;;;; ident.ll: 265 266 ; RUN: llvm-link %S/Inputs/ident.a.ll %S/Inputs/ident.b.ll -S | FileCheck %s 267 268 ; Verify that multiple input llvm.ident metadata are linked together. 269 270 ; CHECK-DAG: !llvm.ident = !{!0, !1, !2} 271 ; CHECK-DAG: "Compiler V1" 272 ; CHECK-DAG: "Compiler V2" 273 ; CHECK-DAG: "Compiler V3" 274 275 ;;;;; Inputs/ident.a.ll: 276 277 !llvm.ident = !{!0, !1} 278 !0 = metadata !{metadata !"Compiler V1"} 279 !1 = metadata !{metadata !"Compiler V2"} 280 281 ;;;;; Inputs/ident.b.ll: 282 283 !llvm.ident = !{!0} 284 !0 = metadata !{metadata !"Compiler V3"} 285 286For symmetry reasons, ``ident.ll`` is just a dummy file that doesn't 287actually participate in the test besides holding the ``RUN:`` lines. 288 289.. note:: 290 291 Some existing tests use ``RUN: true`` in extra files instead of just 292 putting the extra files in an ``Inputs/`` directory. This pattern is 293 deprecated. 294 295Fragile tests 296------------- 297 298It is easy to write a fragile test that would fail spuriously if the tool being 299tested outputs a full path to the input file. For example, :program:`opt` by 300default outputs a ``ModuleID``: 301 302.. code-block:: console 303 304 $ cat example.ll 305 define i32 @main() nounwind { 306 ret i32 0 307 } 308 309 $ opt -S /path/to/example.ll 310 ; ModuleID = '/path/to/example.ll' 311 312 define i32 @main() nounwind { 313 ret i32 0 314 } 315 316``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: 317 318.. code-block:: llvm 319 320 ; RUN: opt -S %s | FileCheck 321 322 define i32 @main() nounwind { 323 ; CHECK-NOT: load 324 ret i32 0 325 } 326 327This test will fail if placed into a ``download`` directory. 328 329To make your tests robust, always use ``opt ... < %s`` in the RUN line. 330:program:`opt` does not output a ``ModuleID`` when input comes from stdin. 331 332Platform-Specific Tests 333----------------------- 334 335Whenever adding tests that require the knowledge of a specific platform, 336either related to code generated, specific output or back-end features, 337you must make sure to isolate the features, so that buildbots that 338run on different architectures (and don't even compile all back-ends), 339don't fail. 340 341The first problem is to check for target-specific output, for example sizes 342of structures, paths and architecture names, for example: 343 344* Tests containing Windows paths will fail on Linux and vice-versa. 345* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. 346* Tests where the debug information calculates the size of types and structures. 347 348Also, if the test rely on any behaviour that is coded in any back-end, it must 349go in its own directory. So, for instance, code generator tests for ARM go 350into ``test/CodeGen/ARM`` and so on. Those directories contain a special 351``lit`` configuration file that ensure all tests in that directory will 352only run if a specific back-end is compiled and available. 353 354For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: 355 356.. code-block:: python 357 358 config.suffixes = ['.ll', '.c', '.cpp', '.test'] 359 if not 'ARM' in config.root.targets: 360 config.unsupported = True 361 362Other platform-specific tests are those that depend on a specific feature 363of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. 364 365For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture 366variants: 367 368.. code-block:: llvm 369 370 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 371 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 372 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 373 374And the checks are different: 375 376.. code-block:: llvm 377 378 ; SSE2: @test1 379 ; SSE2: psubusw LCPI0_0(%rip), %xmm0 380 ; AVX1: @test1 381 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 382 ; AVX2: @test1 383 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 384 385So, if you're testing for a behaviour that you know is platform-specific or 386depends on special features of sub-architectures, you must add the specific 387triple, test with the specific FileCheck and put it into the specific 388directory that will filter out all other architectures. 389 390 391Substitutions 392------------- 393 394Besides replacing LLVM tool names the following substitutions are performed in 395RUN lines: 396 397``%%`` 398 Replaced by a single ``%``. This allows escaping other substitutions. 399 400``%s`` 401 File path to the test case's source. This is suitable for passing on the 402 command line as the input to an LLVM tool. 403 404 Example: ``/home/user/llvm/test/MC/ELF/foo_test.s`` 405 406``%S`` 407 Directory path to the test case's source. 408 409 Example: ``/home/user/llvm/test/MC/ELF`` 410 411``%t`` 412 File path to a temporary file name that could be used for this test case. 413 The file name won't conflict with other test cases. You can append to it 414 if you need multiple temporaries. This is useful as the destination of 415 some redirected output. 416 417 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` 418 419``%T`` 420 Directory of ``%t``. 421 422 Example: ``/home/user/llvm.build/test/MC/ELF/Output`` 423 424``%{pathsep}`` 425 426 Expands to the path separator, i.e. ``:`` (or ``;`` on Windows). 427 428 429**LLVM-specific substitutions:** 430 431``%shlibext`` 432 The suffix for the host platforms shared library files. This includes the 433 period as the first character. 434 435 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) 436 437``%exeext`` 438 The suffix for the host platforms executable files. This includes the 439 period as the first character. 440 441 Example: ``.exe`` (Windows), empty on Linux. 442 443``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` 444 The number of the line where this substitution is used, with an optional 445 integer offset. This can be used in tests with multiple RUN lines, which 446 reference test file's line numbers. 447 448 449**Clang-specific substitutions:** 450 451``%clang`` 452 Invokes the Clang driver. 453 454``%clang_cpp`` 455 Invokes the Clang driver for C++. 456 457``%clang_cl`` 458 Invokes the CL-compatible Clang driver. 459 460``%clangxx`` 461 Invokes the G++-compatible Clang driver. 462 463``%clang_cc1`` 464 Invokes the Clang frontend. 465 466``%itanium_abi_triple``, ``%ms_abi_triple`` 467 These substitutions can be used to get the current target triple adjusted to 468 the desired ABI. For example, if the test suite is running with the 469 ``i686-pc-win32`` target, ``%itanium_abi_triple`` will expand to 470 ``i686-pc-mingw32``. This allows a test to run with a specific ABI without 471 constraining it to a specific triple. 472 473To add more substituations, look at ``test/lit.cfg`` or ``lit.local.cfg``. 474 475 476Options 477------- 478 479The llvm lit configuration allows to customize some things with user options: 480 481``llc``, ``opt``, ... 482 Substitute the respective llvm tool name with a custom command line. This 483 allows to specify custom paths and default arguments for these tools. 484 Example: 485 486 % llvm-lit "-Dllc=llc -verify-machineinstrs" 487 488``run_long_tests`` 489 Enable the execution of long running tests. 490 491``llvm_site_config`` 492 Load the specified lit configuration instead of the default one. 493 494 495Other Features 496-------------- 497 498To make RUN line writing easier, there are several helper programs. These 499helpers are in the PATH when running tests, so you can just call them using 500their name. For example: 501 502``not`` 503 This program runs its arguments and then inverts the result code from it. 504 Zero result codes become 1. Non-zero result codes become 0. 505 506Sometimes it is necessary to mark a test case as "expected fail" or 507XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` 508on a line near the top of the file. This signals that the test case 509should succeed if the test fails. Such test cases are counted separately 510by the testing tool. To specify an expected fail, use the XFAIL keyword 511in the comments of the test program followed by a colon and one or more 512failure patterns. Each failure pattern can be either ``*`` (to specify 513fail everywhere), or a part of a target triple (indicating the test 514should fail on that platform), or the name of a configurable feature 515(for example, ``loadable_module``). If there is a match, the test is 516expected to fail. If not, the test is expected to succeed. To XFAIL 517everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` 518line: 519 520.. code-block:: llvm 521 522 ; XFAIL: darwin,sun 523 524To make the output more useful, :program:`lit` will scan 525the lines of the test case for ones that contain a pattern that matches 526``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number 527that is related to the test case. The number after "PR" specifies the 528LLVM bugzilla number. When a PR number is specified, it will be used in 529the pass/fail reporting. This is useful to quickly get some context when 530a test fails. 531 532Finally, any line that contains "END." will cause the special 533interpretation of lines to terminate. This is generally done right after 534the last RUN: line. This has two side effects: 535 536(a) it prevents special interpretation of lines that are part of the test 537 program, not the instructions to the test case, and 538 539(b) it speeds things up for really big test cases by avoiding 540 interpretation of the remainder of the file. 541 542.. _test-suite-overview: 543 544``test-suite`` Overview 545======================= 546 547The ``test-suite`` module contains a number of programs that can be 548compiled and executed. The ``test-suite`` includes reference outputs for 549all of the programs, so that the output of the executed program can be 550checked for correctness. 551 552``test-suite`` tests are divided into three types of tests: MultiSource, 553SingleSource, and External. 554 555- ``test-suite/SingleSource`` 556 557 The SingleSource directory contains test programs that are only a 558 single source file in size. These are usually small benchmark 559 programs or small programs that calculate a particular value. Several 560 such programs are grouped together in each directory. 561 562- ``test-suite/MultiSource`` 563 564 The MultiSource directory contains subdirectories which contain 565 entire programs with multiple source files. Large benchmarks and 566 whole applications go here. 567 568- ``test-suite/External`` 569 570 The External directory contains Makefiles for building code that is 571 external to (i.e., not distributed with) LLVM. The most prominent 572 members of this directory are the SPEC 95 and SPEC 2000 benchmark 573 suites. The ``External`` directory does not contain these actual 574 tests, but only the Makefiles that know how to properly compile these 575 programs from somewhere else. When using ``LNT``, use the 576 ``--test-externals`` option to include these tests in the results. 577 578.. _test-suite-quickstart: 579 580``test-suite`` Quickstart 581------------------------- 582 583The modern way of running the ``test-suite`` is focused on testing and 584benchmarking complete compilers using the 585`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. 586 587For more information on using LNT to execute the ``test-suite``, please 588see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ 589documentation. 590 591``test-suite`` Makefiles 592------------------------ 593 594Historically, the ``test-suite`` was executed using a complicated setup 595of Makefiles. The LNT based approach above is recommended for most 596users, but there are some testing scenarios which are not supported by 597the LNT approach. In addition, LNT currently uses the Makefile setup 598under the covers and so developers who are interested in how LNT works 599under the hood may want to understand the Makefile based setup. 600 601For more information on the ``test-suite`` Makefile setup, please see 602the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. 603