[lit] Support reading arguments from a file

This allows reading arguments from file using the response file syntax.
We would like to use this in the LLVM build to pass test suites from
subbuilds.

D

[lit] Support reading arguments from a file

This allows reading arguments from file using the response file syntax.
We would like to use this in the LLVM build to pass test suites from
subbuilds.

Differential Revision: https://reviews.llvm.org/D132437

(cherry picked from commit b52820edec6f3d95edef7859a56ef30878b08382)

show more ...

[test] Add lit helper for windows paths

This adds 2 new lit helpers `%{fs-src-root}` and `%{fs-sep}`, these
allow writing tests that correctly handle slashes on Windows. In the
case of tests like cl

[test] Add lit helper for windows paths

This adds 2 new lit helpers `%{fs-src-root}` and `%{fs-sep}`, these
allow writing tests that correctly handle slashes on Windows. In the
case of tests like clang/test/CodeGen/debug-prefix-map.c, these are
unable to correctly test behavior on both platforms, unless they fork
and add OS requirements, because the relevant logic hits host specific
codepaths like checking if paths are absolute.

Differential Revision: https://reviews.llvm.org/D111457

show more ...

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default t

[llvm] [lit] Support forcing lexical test order

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.

Differential Revision: https://reviews.llvm.org/D107695

show more ...

[lit] Add --xfail-not/LIT_XFAIL_NOT

For example, I need this lately in my CI config:

LIT_XFAIL_NOT='libomptarget :: nvptx64-nvidia-cuda :: unified_shared_memory/api.c'

That test specifies an XFAIL

[lit] Add --xfail-not/LIT_XFAIL_NOT

For example, I need this lately in my CI config:

LIT_XFAIL_NOT='libomptarget :: nvptx64-nvidia-cuda :: unified_shared_memory/api.c'

That test specifies an XFAIL directive, but I get an XPASS result.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D106022

show more ...

[lit] Extend --xfail/LIT_XFAIL to take full test name

The new documentation entry gives an example use case from
libomptarget.

Reviewed By: yln, jhenderson, davezarzycki

Differential Revision: htt

[lit] Extend --xfail/LIT_XFAIL to take full test name

The new documentation entry gives an example use case from
libomptarget.

Reviewed By: yln, jhenderson, davezarzycki

Differential Revision: https://reviews.llvm.org/D105208

show more ...

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2)

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2) A very recently introduced "early_tests" feature.
3) The `--incremental` flag forces failing tests to run first.

All of these approaches have problems.

1) The `is_early` feature was until very recently undocumented. Nevertheless it still lacks testing and is a imprecise way of optimizing test starting times.
2) The `early_tests` feature requires manual updates and doesn't scale.
3) `--incremental` is undocumented, untested, and it requires modifying the *source* file system by "touching" the file. This "touch" based approach is arguably a hack because it confuses editors (because it looks like the test was modified behind the back of the editor) and "touching" the test source file doesn't work if the test suite is read only from the perspective of `lit` (via advanced filesystem/build tricks).

This patch attempts to simplify and address all of the above problems.

This patch formalizes, documents, tests, and defaults lit to recording the execution time of tests and then reordering all tests during the next execution. By reordering the tests, high core count machines run faster, sometimes significantly so.

This patch also always runs failing tests first, which is a positive user experience win for those that didn't know about the hidden `--incremental` flag.

Finally, if users want, they can _optionally_ commit the test timing data (or a subset thereof) back to the repository to accelerate bots and first-time runs of the test suite.

Reviewed By: jhenderson, yln

Differential Revision: https://reviews.llvm.org/D98179

show more ...

[lit] Add --ignore-fail

For some build configurations, `check-all` calls lit multiple times to
run multiple lit test suites. Most recently, I've found this to be
true when configuring openmp as par

[lit] Add --ignore-fail

For some build configurations, `check-all` calls lit multiple times to
run multiple lit test suites. Most recently, I've found this to be
true when configuring openmp as part of `LLVM_ENABLE_RUNTIMES`, but
this is not the first time.

If one test suite fails, none of the remaining test suites run, so you
cannot determine if your patch has broken them. It can then be
frustrating to try to determine which `check-` targets will run the
remaining tests without getting stuck on the failing tests.

When such cases arise, it is probably best to adjust the cmake
configuration for `check-all` to run all test suites as part of one
lit invocation. Because that fix will likely not be implemented and
land immediately, this patch introduces `--ignore-fail` to serve as a
workaround for developers trying to see test results until it does
land:

```
$ LIT_OPTS=--ignore-fail ninja check-all
```

One problem with `--ignore-fail` is that it makes it challenging to
detect test failures in a script, perhaps in CI. This problem should
serve as motivation to actually fix the cmake configuration instead of
continuing to use `--ignore-fail` indefinitely.

Reviewed By: jhenderson, thopre

Differential Revision: https://reviews.llvm.org/D96371

show more ...

[lit] Add --xfail and --filter-out (inverse of --filter)

In semi-automated environments, XFAILing or filtering out known regressions without actually committing changes or temporarily modifying the

[lit] Add --xfail and --filter-out (inverse of --filter)

In semi-automated environments, XFAILing or filtering out known regressions without actually committing changes or temporarily modifying the test suite can be quite useful.

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D96662

show more ...

[lit] Add "early_tests" config option

With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve perf

[lit] Add "early_tests" config option

With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve performance (for high-end systems) by listing just a few of the slowest tests up front.

Reviewed By: jdenny, jhenderson

Differential Revision: https://reviews.llvm.org/D96594

show more ...

Add lit config for dir with standalone tests

Some test systems do not use lit for test discovery but only for its
substitution and test selection because they use another way of managing
test collec

Add lit config for dir with standalone tests

Some test systems do not use lit for test discovery but only for its
substitution and test selection because they use another way of managing
test collections, e.g. CTest. This forces those tests to be invoked with
lit --no-indirectly-run-check. When a mix of lit version is in use, it
requires to detect the availability of that option.

This commit provides a new config option standalone_tests to signal a
directory made of tests meant to run as standalone. When this option is
set, lit skips test discovery and the indirectly run check. It also adds
the missing documentation for --no-indirectly-run-check.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D94766

show more ...

fix doc typo to cycle bots

[docs] [lit] Add a more helpful description for lit.py's -s flag.

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D82808

[LIT] Correcting max-failures option in lit documentation.

[LIT] NFC adding max-failures option to lit documentation.

Differential Revision: https://reviews.llvm.org/D81337

[docs] Fix :option: links

[docs] Fix :option: links

[docs] Document lit's --timeout=N flag

[lit] Move the recursiveExpansionLimit setting to TestingConfig

The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some tes

[lit] Move the recursiveExpansionLimit setting to TestingConfig

The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.

Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.

Differential Revision: https://reviews.llvm.org/D77415

show more ...

[lit] Recursively expand substitutions

This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cx

[lit] Recursively expand substitutions

This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.

Differential Revision: https://reviews.llvm.org/D76178

show more ...

[lit] NFC: Document missing result codes

These result codes already exist, but they were not documented. I assume
this is an oversight when adding these result codes.

[llvm] NFC: fix trivial typos in documents

Reviewers: hans, Jim

Reviewed By: Jim

Subscribers: jvesely, nhaehnle, mgorny, arphaman, bmahjour, kerbowa, llvm-commits

Tags: #llvm

Differential Revisi

[llvm] NFC: fix trivial typos in documents

Reviewers: hans, Jim

Reviewed By: Jim

Subscribers: jvesely, nhaehnle, mgorny, arphaman, bmahjour, kerbowa, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D73017

show more ...

[lit] Document the undocumented pre-defined substitutions

[lit] Bring back `--threads` option alias

Bring back `--threads` option which was lost in the move of the
command line argument parsing code to cl_arguments.py. Update docs
since `--workers` is pre

[lit] Bring back `--threads` option alias

Bring back `--threads` option which was lost in the move of the
command line argument parsing code to cl_arguments.py. Update docs
since `--workers` is preferred.

llvm-svn: 374432

show more ...

[lit] Parse command-line options from LIT_OPTS

Similar to `FILECHECK_OPTS` for FileCheck, `LIT_OPTS` makes it easy to
adjust lit behavior when running the test suite via ninja. For
example:

```
$

[lit] Parse command-line options from LIT_OPTS

Similar to `FILECHECK_OPTS` for FileCheck, `LIT_OPTS` makes it easy to
adjust lit behavior when running the test suite via ninja. For
example:

```
$ LIT_OPTS='--time-tests -vv --filter=threadprivate' \
ninja check-clang-openmp
```

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D64135

llvm-svn: 365313

show more ...

[docs][tools] Add missing "program" tags to rst files

Sphinx allows for definitions of command-line options using
`.. option <name>` and references to those options via `:option:<name>`.
However, it

[docs][tools] Add missing "program" tags to rst files

Sphinx allows for definitions of command-line options using
`.. option <name>` and references to those options via `:option:<name>`.
However, it looks like there is no scoping of these options by default,
meaning that links can end up pointing to incorrect documents. See for
example the llvm-mca document, which contains references to -o that,
prior to this patch, pointed to a different document. What's worse is
that these links appear to be non-deterministic in which one is picked
(on my machine, some references end up pointing to opt, whereas on the
live docs, they point to llvm-dwarfdump, for example).

The fix is to add the .. program <name> tag. This essentially namespaces
the options (definitions and references) to the named program, ensuring
that the links are kept correct.

Reviwed by: andreadb

Differential Revision: https://reviews.llvm.org/D63873

llvm-svn: 364538

show more ...