[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.

Differential Revision: https://reviews.llvm.org/D125484

show more ...

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and relate

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and related code-paths.

Differential Revision: https://reviews.llvm.org/D124558

show more ...

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appsp

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-mac-x64/b8818888168677577537/overview

show more ...

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, th

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, though. Testing the few special cases we have in `clang-scan-deps` for `clang-cl` should be good enough:

1. Deducing output path (and therefore target name in our make output).
2. Ignoring `-Xclang` arguments in step 1.
3. Deducing resource directory by invoking the compiler executuable.

This test de-duplicates the extra clang-cl test cases.

Reviewed By: dexonsmith, saudi

Differential Revision: https://reviews.llvm.org/D121812

show more ...

[clang][deps] Move invocation adjustments from `clang-scan-deps` to `DependencyScanning` library

The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. T

[clang][deps] Move invocation adjustments from `clang-scan-deps` to `DependencyScanning` library

The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. The goal is to setup `DependencyOutputOptions` by injecting `-M -MT <target>` and prevent the creation of output files.

This patch moves the logic into the `DependencyScanning` library, and uses the parsed `CompilerInvocation` instead of the raw command-line. The code simpler and can be used from the C++ API as well.

The `-o /dev/null` arguments are not necessary, since the `DependencyScanning` library only runs a preprocessing action, so there's no way it'll produce an actual object file.

Related: The `-M` argument implies `-w`, which would appear on the command-line of modular dependencies even though it was not on the original TU command line (see D104036).

Some related tests were updated.

Reviewed By: arphaman

Differential Revision: https://reviews.llvm.org/D104030

show more ...

Reland "[clang-scan-deps] Add support for clang-cl"

This reverts commit 199c39748292cbc89cd148a0d8364ebb1014ec38.
This time, clang-scan-deps's search for output argument in clang-cl command line wil

Reland "[clang-scan-deps] Add support for clang-cl"

This reverts commit 199c39748292cbc89cd148a0d8364ebb1014ec38.
This time, clang-scan-deps's search for output argument in clang-cl command line will now ignore arguments preceded by "-Xclang".
That way, it won't detect a /o argument in "-Xclang -ivfsoverlay -Xclang /opt/subpath"

Initial patch description:
clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews.llvm.org/D92191

show more ...

Revert "[clang-scan-deps] Add support for clang-cl"

This reverts commit bb26fa8c286bf524ed9235c3e293ad22ecf3e984.

[clang-scan-deps] Add support for clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https:/

[clang-scan-deps] Add support for clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews.llvm.org/D92191

show more ...

Revert "[clang-scan-deps] Support clang-cl"

Reverting, as it breaks build on mac.

This reverts commit 640ad769110f76d3391667249478ece88c835529.

[clang-scan-deps] Support clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews

[clang-scan-deps] Support clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews.llvm.org/D92191

show more ...

[clang][ScanDeps] Fix issue with multiple commands with the same input.

Previously, given a CompilationDatabase with two commands for the same
source file we would report that file twice with the un

[clang][ScanDeps] Fix issue with multiple commands with the same input.

Previously, given a CompilationDatabase with two commands for the same
source file we would report that file twice with the union of the
dependencies for each command both times.

This was due to the way `ClangTool` runs actions given an input source
file (see the comment in `DependencyScanningTool.cpp`). This commit adds
a `SingleCommandCompilationDatabase` that is created with each
`CompileCommand` in the original CDB, which is then used for each
`ClangTool` invocation. This gives us a single run of
`DependencyScanningAction` per `CompileCommand`.

I looked at using `AllTUsToolExecutor` which is a parallel tool
executor, but I'm not sure it's suitable for `clang-scan-deps` as it
does a lot more sharing of state than `AllTUsToolExecutor` expects.

Differential Revision: https://reviews.llvm.org/D69643

show more ...

[clang-scan-deps] Add dependency targets

Differential Revision: https://reviews.llvm.org/D67475

llvm-svn: 371697

[clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preproc

[clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor
blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive.
The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.

On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on
the compilation database for a recent LLVM and Clang (3511 files).

Differential Revision: https://reviews.llvm.org/D67127

llvm-svn: 371656

show more ...

[clang-scan-deps][NFC] Fix tests - prevent FileCheck matching test dir path

Differential Revision: https://reviews.llvm.org/D67379

llvm-svn: 371578

clang-scan-deps: do not spawn threads when LLVM_ENABLE_THREADS is disabled

llvm-svn: 368640

Mark clang-scan-deps test as requiring thread support

Otherwise the test calls a pure virtual method and crashes. Perhaps this
could be improved.

llvm-svn: 368354

[clang-scan-deps] Implementation of dependency scanner over minimized sources

This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are

[clang-scan-deps] Implementation of dependency scanner over minimized sources

This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are minimized using the dependency directives source minimizer.

A shared file system cache is used to ensure that the file system requests and source minimization
is performed only once. The cache assumes that the underlying filesystem won't change during the course
of the scan (or if it will, it will not affect the output), and it can't be evicted. This means that the
service and workers can be used for a single run of a dependency scanner, and can't be reused across multiple,
incremental runs. This is something that we'll most likely support in the future though.
Note that the driver still utilizes the underlying real filesystem.

This commit is also still missing the fast skipped PP block skipping optimization that I mentioned at EuroLLVM talk.
Additionally, the file manager is still not reused by the threads as well.

Differential Revision: https://reviews.llvm.org/D63907

llvm-svn: 368086

show more ...

[clang-scan-deps] print the dependencies to stdout
and remove the need to use -MD options in the CDB

Differential Revision: https://reviews.llvm.org/D63579

llvm-svn: 364088

[clang-scan-deps] initial outline of the tool that runs preprocessor to find
dependencies over a JSON compilation database

This commit introduces an outline for the clang-scan-deps tool that will be

[clang-scan-deps] initial outline of the tool that runs preprocessor to find
dependencies over a JSON compilation database

This commit introduces an outline for the clang-scan-deps tool that will be
used to implement fast dependency discovery phase using implicit modules for
explicit module builds.

The initial version of the tool works by computing non-modular header dependencies
for files in the compilation database without any optimizations
(i.e. without source minimization from r362459).
The tool spawns a number of worker threads to run the clang compiler workers in parallel.

The immediate goal for clang-scan-deps is to create a ClangScanDeps library
which will be used to build up this tool to use the source minimization and
caching multi-threaded filesystem to implement the optimized non-incremental
dependency scanning phase for a non-modular build. This will allow us to do
benchmarks and comparisons for performance that the minimization and caching give us

Differential Revision: https://reviews.llvm.org/D60233

llvm-svn: 363204

show more ...