1*dc213305SDaniel Sanders======================================== 2*dc213305SDaniel SandersCompiler-rt Testing Infrastructure Guide 3*dc213305SDaniel Sanders======================================== 4*dc213305SDaniel Sanders 5*dc213305SDaniel Sanders.. contents:: 6*dc213305SDaniel Sanders :local: 7*dc213305SDaniel Sanders 8*dc213305SDaniel SandersOverview 9*dc213305SDaniel Sanders======== 10*dc213305SDaniel Sanders 11*dc213305SDaniel SandersThis document is the reference manual for the compiler-rt modifications to the 12*dc213305SDaniel Sanderstesting infrastructure. Documentation for the infrastructure itself can be found at 13*dc213305SDaniel Sanders:ref:`llvm_testing_guide`. 14*dc213305SDaniel Sanders 15*dc213305SDaniel SandersLLVM testing infrastructure organization 16*dc213305SDaniel Sanders======================================== 17*dc213305SDaniel Sanders 18*dc213305SDaniel SandersThe compiler-rt testing infrastructure contains regression tests which are run 19*dc213305SDaniel Sandersas part of the usual ``make check-all`` and are expected to always pass -- they 20*dc213305SDaniel Sandersshould be run before every commit. 21*dc213305SDaniel Sanders 22*dc213305SDaniel SandersQuick start 23*dc213305SDaniel Sanders=========== 24*dc213305SDaniel Sanders 25*dc213305SDaniel SandersThe regressions tests are in the "compiler-rt" module and are normally checked 26*dc213305SDaniel Sandersout in the directory ``llvm/projects/compiler-rt/test``. Use ``make check-all`` 27*dc213305SDaniel Sandersto run the regression tests after building compiler-rt. 28*dc213305SDaniel Sanders 29*dc213305SDaniel SandersREQUIRES, XFAIL, etc. 30*dc213305SDaniel Sanders--------------------- 31*dc213305SDaniel Sanders 32*dc213305SDaniel SandersSometimes it is necessary to restrict a test to a specific target or mark it as 33*dc213305SDaniel Sandersan "expected fail" or XFAIL. This is normally achieved using ``REQUIRES:`` or 34*dc213305SDaniel Sanders``XFAIL:`` with a substring of LLVM's default target triple. Unfortunately, the 35*dc213305SDaniel Sandersbehaviour of this is somewhat quirky in compiler-rt. There are two main 36*dc213305SDaniel Sanderspitfalls to avoid. 37*dc213305SDaniel Sanders 38*dc213305SDaniel SandersThe first pitfall is that these directives perform a substring match on the 39*dc213305SDaniel Sanderstriple and as such ``XFAIL: mips`` affects more triples than expected. For 40*dc213305SDaniel Sandersexample, ``mips-linux-gnu``, ``mipsel-linux-gnu``, ``mips64-linux-gnu``, and 41*dc213305SDaniel Sanders``mips64el-linux-gnu`` will all match a ``XFAIL: mips`` directive. Including a 42*dc213305SDaniel Sanderstrailing ``-`` such as in ``XFAIL: mips-`` can help to mitigate this quirk but 43*dc213305SDaniel Sanderseven that has issues as described below. 44*dc213305SDaniel Sanders 45*dc213305SDaniel SandersThe second pitfall is that the default target triple is often inappropriate for 46*dc213305SDaniel Sanderscompiler-rt tests since compiler-rt tests may be compiled for multiple targets. 47*dc213305SDaniel SandersFor example, a typical build on an ``x86_64-linux-gnu`` host will often run the 48*dc213305SDaniel Sanderstests for both x86_64 and i386. In this situation ``XFAIL: x86_64`` will mark 49*dc213305SDaniel Sandersboth the x86_64 and i386 tests as an expected failure while ``XFAIL: i386`` 50*dc213305SDaniel Sanderswill have no effect at all. 51*dc213305SDaniel Sanders 52*dc213305SDaniel SandersTo remedy both pitfalls, compiler-rt tests provide a feature string which can 53*dc213305SDaniel Sandersbe used to specify a single target. This string is of the form 54*dc213305SDaniel Sanders``target-is-${arch}`` where ``${arch}}`` is one of the values from the 55*dc213305SDaniel Sandersfollowing lines of the CMake output:: 56*dc213305SDaniel Sanders 57*dc213305SDaniel Sanders -- Compiler-RT supported architectures: x86_64;i386 58*dc213305SDaniel Sanders -- Builtin supported architectures: i386;x86_64 59*dc213305SDaniel Sanders 60*dc213305SDaniel SandersSo for example ``XFAIL: target-is-x86_64`` will mark a test as expected to fail 61*dc213305SDaniel Sanderson x86_64 without also affecting the i386 test and ``XFAIL: target-is-i386`` 62*dc213305SDaniel Sanderswill mark a test as expected to fail on i386 even if the default target triple 63*dc213305SDaniel Sandersis ``x86_64-linux-gnu``. Directives that use these ``target-is-${arch}`` string 64*dc213305SDaniel Sandersrequire exact matches so ``XFAIL: target-is-mips``, 65*dc213305SDaniel Sanders``XFAIL: target-is-mipsel``, ``XFAIL: target-is-mips64``, and 66*dc213305SDaniel Sanders``XFAIL: target-is-mips64el`` all refer to different MIPS targets. 67