[Lit] Pass through DFLTCC from the surrounding environment

Many distros ship zlib with the IBM Z deflate hardware acceleration
patch [1]. Sometimes it's desirable to disable the acceleration, for
ex

[Lit] Pass through DFLTCC from the surrounding environment

Many distros ship zlib with the IBM Z deflate hardware acceleration
patch [1]. Sometimes it's desirable to disable the acceleration, for
example, for reproducibility. This can be done by exporting DFLTCC=0.
llvm-lit clears this environment variable, which causes
compress-debug-sections-zlib.test fail on z15 and later machines. Add
DFLTCC to the list of variables to keep.

[1] https://github.com/madler/zlib/pull/410

Reviewed By: abrachet

Differential Revision: https://reviews.llvm.org/D130253

show more ...

Revert "[lit] Passthrough CLANG_MODULE_CACHE_PATH env var"

This reverts commit 6466c9abf3674bade1f6ee859f24ebc7aaf9cd88.

[lit] Passthrough CLANG_MODULE_CACHE_PATH env var

This environment variable can be set to control module
caching. It disables caching by setting the variable
empty. As such, it needs to be handled d

[lit] Passthrough CLANG_MODULE_CACHE_PATH env var

This environment variable can be set to control module
caching. It disables caching by setting the variable
empty. As such, it needs to be handled differently
from other environment variables here which are
assumed to not be empty.

show more ...

[lit] pass LLVM_SYMBOLIZER_PATH through to tests.

Currently several buildbots give unsymbolized traces on crash.
I suspect these are configuring the symbolizer in this way and regressed in
D122251 o

[lit] pass LLVM_SYMBOLIZER_PATH through to tests.

Currently several buildbots give unsymbolized traces on crash.
I suspect these are configuring the symbolizer in this way and regressed in
D122251 or thereabouts.

Trying this coupled with a reland of patch that failed on a couple of bots with
no useful stacktrace...

show more ...

[lit] Forward more sanitizer env in TestingConfig

[NFC] Reformat a part of TestingConfig.py

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtest

[Reland][lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk, abrachet

Differential Revision: https://reviews.llvm.org/D122251

show more ...

Revert "Reland "[lit] Use sharding for GoogleTest format""

This reverts commit 948f3deca91a66caf4a618f826ff6de8277bed9c.

Reland "[lit] Use sharding for GoogleTest format"

This relands commit a87ba5c86d5d72defdbcdb278baad6515ec99463.

Adjust llvm/utils/lit/tests/googletest-timeout.py for new test output.

Revert "[lit] Use sharding for GoogleTest format"

This reverts commit a87ba5c86d5d72defdbcdb278baad6515ec99463.

Breaks bots:
https://lab.llvm.org/buildbot/#/builders/196/builds/10454

[lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instea

[lit] Use sharding for GoogleTest format

This helps lit unit test performance by a lot, especially on windows. The performance gain comes from launching one gtest executable for many subtests instead of one (this is the current situation).

The shards are executed by the test runner and the results are stored in the
json format supported by the GoogleTest. Later in the test reporting stage,
all test results in the json file are retrieved to continue the test results
summary etc.

On my Win10 desktop, before this patch: `check-clang-unit`: 177s, `check-llvm-unit`: 38s; after this patch: `check-clang-unit`: 37s, `check-llvm-unit`: 11s.
On my Linux machine, before this patch: `check-clang-unit`: 46s, `check-llvm-unit`: 8s; after this patch: `check-clang-unit`: 7s, `check-llvm-unit`: 4s.

Reviewed By: yln, rnk

Differential Revision: https://reviews.llvm.org/D122251

show more ...

[lit] Pass the COMSPEC variable through to test processes on Windows

This variable is necessary for the system() function for running
external processes.

This is needes for some libcxx tests. With

[lit] Pass the COMSPEC variable through to test processes on Windows

This variable is necessary for the system() function for running
external processes.

This is needes for some libcxx tests. With the current libcxx test
infrastructure, all OS environment variables are passed through, but
with the new "from scratch" libcxx test setup, we only pass through
the variables listed here.

Differential Revision: https://reviews.llvm.org/D115363

show more ...

Have lit preserve SOURCE_DATE_EPOCH

This environment variable has been standardized for reproducible builds. Setting
it can help to have reproducible tests too, so keep it as part of the testing
env

Have lit preserve SOURCE_DATE_EPOCH

This environment variable has been standardized for reproducible builds. Setting
it can help to have reproducible tests too, so keep it as part of the testing
env when set.

See https://reproducible-builds.org/docs/source-date-epoch/

Differential Revision: https://reviews.llvm.org/D108332

show more ...

[debuginfo-tests] Fix environment variable used to specify LLDB

Currently, if the user specifies the environment variable 'CLANG', tests
will attempt to use the value as a path to the clang executab

[debuginfo-tests] Fix environment variable used to specify LLDB

Currently, if the user specifies the environment variable 'CLANG', tests
will attempt to use the value as a path to the clang executable.
Previously, lldb could also be specified via the CLANG environment
variable, but this was almost certainly a bug, because that meant both
clang and lldb would have the same path. This patch changes the
environment variable for lldb to 'LLDB'.

Reviewed by: thopre, teemperor

Differential Revision: https://reviews.llvm.org/D101982

show more ...

[lit] Pass the USERPROFILE variable through on Windows

When running in a Windows Container, the Git for Windows Unix tools
(C:\Program Files\Git\usr\bin) just hang if this variable isn't
passed thro

[lit] Pass the USERPROFILE variable through on Windows

When running in a Windows Container, the Git for Windows Unix tools
(C:\Program Files\Git\usr\bin) just hang if this variable isn't
passed through.

Currently, running the LLVM/clang tests in a Windows Container fails
if that directory is added to the path, but succeeds after this change.
(After this change, the previously used GnuWin tools can be left out
entirely, too, as lit automatically picks up the Git for Windows tools
if necessary.)

Differential Revision: https://reviews.llvm.org/D98858

show more ...

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2)

[lit] Sort test start times based on prior test timing data

Lit as it exists today has three hacks that allow users to run tests earlier:

1) An entire test suite can set the `is_early` boolean.
2) A very recently introduced "early_tests" feature.
3) The `--incremental` flag forces failing tests to run first.

All of these approaches have problems.

1) The `is_early` feature was until very recently undocumented. Nevertheless it still lacks testing and is a imprecise way of optimizing test starting times.
2) The `early_tests` feature requires manual updates and doesn't scale.
3) `--incremental` is undocumented, untested, and it requires modifying the *source* file system by "touching" the file. This "touch" based approach is arguably a hack because it confuses editors (because it looks like the test was modified behind the back of the editor) and "touching" the test source file doesn't work if the test suite is read only from the perspective of `lit` (via advanced filesystem/build tricks).

This patch attempts to simplify and address all of the above problems.

This patch formalizes, documents, tests, and defaults lit to recording the execution time of tests and then reordering all tests during the next execution. By reordering the tests, high core count machines run faster, sometimes significantly so.

This patch also always runs failing tests first, which is a positive user experience win for those that didn't know about the hidden `--incremental` flag.

Finally, if users want, they can _optionally_ commit the test timing data (or a subset thereof) back to the repository to accelerate bots and first-time runs of the test suite.

Reviewed By: jhenderson, yln

Differential Revision: https://reviews.llvm.org/D98179

show more ...

[lit] Add "early_tests" config option

With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve perf

[lit] Add "early_tests" config option

With enough cores, the slowest tests can significantly change the total testing time if they happen to run late. With this change, a test suite can improve performance (for high-end systems) by listing just a few of the slowest tests up front.

Reviewed By: jdenny, jhenderson

Differential Revision: https://reviews.llvm.org/D96594

show more ...

Add lit config for dir with standalone tests

Some test systems do not use lit for test discovery but only for its
substitution and test selection because they use another way of managing
test collec

Add lit config for dir with standalone tests

Some test systems do not use lit for test discovery but only for its
substitution and test selection because they use another way of managing
test collections, e.g. CTest. This forces those tests to be invoked with
lit --no-indirectly-run-check. When a mix of lit version is in use, it
requires to detect the availability of that option.

This commit provides a new config option standalone_tests to signal a
directory made of tests meant to run as standalone. When this option is
set, lit skips test discovery and the indirectly run check. It also adds
the missing documentation for --no-indirectly-run-check.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D94766

show more ...

[Lit] Pass through SSH_AUTH_SOCK from the surrounding environment

This allows running Lit tests that run ssh without having to manually
enter a password (which is inconvenient), by just having ssh-a

[Lit] Pass through SSH_AUTH_SOCK from the surrounding environment

This allows running Lit tests that run ssh without having to manually
enter a password (which is inconvenient), by just having ssh-agent
setup properly when running the test suite.

show more ...

Change filecheck default to dump input on failure

Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which real

Change filecheck default to dump input on failure

Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.

Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.

Differential Revision: https://reviews.llvm.org/D81422

show more ...

[lit] Move the recursiveExpansionLimit setting to TestingConfig

The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some tes

[lit] Move the recursiveExpansionLimit setting to TestingConfig

The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.

Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.

Differential Revision: https://reviews.llvm.org/D77415

show more ...

[lit] Allow setting parallelism groups to None

Check that we do not crash if a parallelism group is explicitly set to
None. Permits usage of the following pattern.

[lit.common.cfg]
lit_config.par

[lit] Allow setting parallelism groups to None

Check that we do not crash if a parallelism group is explicitly set to
None. Permits usage of the following pattern.

[lit.common.cfg]
lit_config.parallelism_groups['my_group'] = None
if <condition>:
lit_config.parallelism_groups['my_group'] = 3

[project/lit.cfg]
config.parallelism_group = 'my_group'

Reviewers: rnk

Differential Revision: https://reviews.llvm.org/D58305

llvm-svn: 354912

show more ...

[lit] Pass more environment variables through to child processes.

This arose when I was trying to have a substitution which invoked a
python script P, and that python script tried to invoke clang-cl

[lit] Pass more environment variables through to child processes.

This arose when I was trying to have a substitution which invoked a
python script P, and that python script tried to invoke clang-cl (or
even cl). Since we invoke P with a custom environment, it doesn't
inherit the environment of the parent, and then when we go to invoke
clang-cl, it's unable to find the MSVC installation directory. There
were many more I could have passed through which are set by vcvarsall,
but I tried to keep it simple and only pass through the important ones.

Differential Revision: https://reviews.llvm.org/D54963

llvm-svn: 347691

show more ...

Fix clang test suite on Windows by reverting part of r347216

Otherwise, the clang analyzer tests fail on Windows when attempting to
unpickle AnalyzerTest objects in the worker processes. The pattern

Fix clang test suite on Windows by reverting part of r347216

Otherwise, the clang analyzer tests fail on Windows when attempting to
unpickle AnalyzerTest objects in the worker processes. The pattern of,
add to path, import, remove from path, serialize, deserialize, doesn't
work. Once something gets added to the path, if we want to move it
across the wire for multiprocessing, we need to keep the module on
sys.path.

llvm-svn: 347254

show more ...

Fix some issues with LLDB's lit configuration files.

Recently I tried to port LLDB's lit configuration files over to use a
on the surface, but broke some cases that weren't broken before and also
ex

Fix some issues with LLDB's lit configuration files.

Recently I tried to port LLDB's lit configuration files over to use a
on the surface, but broke some cases that weren't broken before and also
exposed some additional problems with the old approach that we were just
getting lucky with.

When we set up a lit environment, the goal is to make it as hermetic as
possible. We should not be relying on PATH and enabling the use of
arbitrary shell commands. Instead, only whitelisted commands should be
allowed. These are, generally speaking, the lit builtins such as echo,
cd, etc, as well as anything for which substitutions have been
explicitly set up for. These substitutions should map to the build
output directory, but in some cases it's useful to be able to override
this (for example to point to an installed tools directory).

This is, of course, how it's supposed to work. What was actually
happening is that we were bringing in PATH and LD_LIBRARY_PATH and then
just running the given run line as a shell command. This led to problems
such as finding the wrong version of clang-cl on PATH since it wasn't
even a substitution, and flakiness / non-determinism since the
environment the tests were running in would change per-machine. On the
other hand, it also made other things possible. For example, we had some
tests that were explicitly running cl.exe and link.exe instead of
clang-cl and lld-link and the only reason it worked at all is because it
was finding them on PATH. Unfortunately we can't entirely get rid of
these tests, because they support a few things in debug info that
clang-cl and lld-link don't (notably, the LF_UDT_MOD_SRC_LINE record
which makes some of the tests fail.

The high level changes introduced in this patch are:

1. Removal of functionality - The lit test suite no longer respects
LLDB_TEST_C_COMPILER and LLDB_TEST_CXX_COMPILER. This means there is no
more support for gcc, but nobody was using this anyway (note: The
functionality is still there for the dotest suite, just not the lit test
suite). There is no longer a single substitution %cxx and %cc which maps
to <arbitrary-compiler>, you now explicitly specify the compiler with a
substitution like %clang or %clangxx or %clang_cl. We can revisit this
in the future when someone needs gcc.

2. Introduction of the LLDB_LIT_TOOLS_DIR directory. This does in spirit
what LLDB_TEST_C_COMPILER and LLDB_TEST_CXX_COMPILER used to do, but now
more friendly. If this is not specified, all tools are expected to be
the just-built tools. If it is specified, the tools which are not
themselves being tested but are being used to construct and run checks
(e.g. clang, FileCheck, llvm-mc, etc) will be searched for in this
directory first, then the build output directory.

3. Changes to core llvm lit files. The use_lld() and use_clang()
functions were introduced long ago in anticipation of using them in
lldb, but since they were never actually used anywhere but their
respective problems, there were some issues to be resolved regarding
generality and ability to use them outside their project.

4. Changes to .test files - These are all just replacing things like
clang-cl with %clang_cl and %cxx with %clangxx, etc.

5. Changes to lit.cfg.py - Previously we would load up some system
environment variables and then add some new things to them. Then do a
bunch of work building out our own substitutions. First, we delete the
system environment variable code, making the environment hermetic. Then,
we refactor the substitution logic into two separate helper functions,
one which sets up substitutions for the tools we want to test (which
must come from the build output directory), and another which sets up
substitutions for support tools (like compilers, etc).

6. New substitutions for MSVC -- Previously we relied on location of
MSVC by bringing in the entire parent's PATH and letting
subprocess.Popen just run the command line. Now we set up real
substitutions that should have the same effect. We use PATH to find
them, and then look for INCLUDE and LIB to construct a substitution
command line with appropriate /I and /LIBPATH: arguments. The nice thing
about this is that it opens the door to having separate %msvc-cl32 and
%msvc-cl64 substitutions, rather than only requiring the user to run
vcvars first. Because we can deduce the path to 32-bit libraries from
64-bit library directories, and vice versa. Without these substitutions
this would have been impossible.

Differential Revision: https://reviews.llvm.org/D54567

llvm-svn: 347216

show more ...