[Tooling/DependencyScanning] Enable passing a `vfs::FileSystem` object to `DependencyScanningTool`

Also include a unit test to validate that the `vfs::FileSystem` object is properly used.

Different

[Tooling/DependencyScanning] Enable passing a `vfs::FileSystem` object to `DependencyScanningTool`

Also include a unit test to validate that the `vfs::FileSystem` object is properly used.

Differential Revision: https://reviews.llvm.org/D129912

show more ...

[clang] Don't use Optional::hasValue (NFC)

[clang][deps] Set -disable-free for module compilations

The command-line arguments for module builds are cc1 commands, so they
do not implicitly set -disable-free like a driver invocation, and
Tooli

[clang][deps] Set -disable-free for module compilations

The command-line arguments for module builds are cc1 commands, so they
do not implicitly set -disable-free like a driver invocation, and
Tooling will disable it for the scanning instance itself. Set
-disable-free explicitly so that separate invocations for building
modules will not pay for freeing memory unnecessarily.

Differential Revision: https://reviews.llvm.org/D127229

show more ...

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionality

Removes `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blocks
will be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.

* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

Replaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.

* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`

This is bringing the following benefits:

* Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
* Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time.
* Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).

For normal preprocessing measurements show differences are below the noise level.

Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.

Differential Revision: https://reviews.llvm.org/D125486
Differential Revision: https://reviews.llvm.org/D125487
Differential Revision: https://reviews.llvm.org/D125488

show more ...

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.

Differential Revision: https://reviews.llvm.org/D125484

show more ...

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and relate

[Tooling/DependencyScanning] Make skipping excluded PP ranges during dependency scanning the default

This is to improve maintenance a bit and remove need to maintain the additional option and related code-paths.

Differential Revision: https://reviews.llvm.org/D124558

show more ...

[clang][deps] Fix traversal of precompiled dependencies

The code for traversing precompiled dependencies is somewhat complicated and contains a dangling iterator bug.

This patch simplifies the code

[clang][deps] Fix traversal of precompiled dependencies

The code for traversing precompiled dependencies is somewhat complicated and contains a dangling iterator bug.

This patch simplifies the code and fixes the bug.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D121533

show more ...

[clang][deps] Disable global module index

While scanning dependencies of a TU that depends on a PCH, the scanner basically performs mixed implicit/explicit modular compilation. (Explicit modules com

[clang][deps] Disable global module index

While scanning dependencies of a TU that depends on a PCH, the scanner basically performs mixed implicit/explicit modular compilation. (Explicit modules come from the PCH.) This seems to trip up the global module index.

This patch disables global module index in the dependency scanner.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118890

show more ...

[clang][deps] NFC: Clean up wording (ignored vs minimized)

The filesystem used during dependency scanning does two things: it caches file entries and minimizes source file contents. We use the term

[clang][deps] NFC: Clean up wording (ignored vs minimized)

The filesystem used during dependency scanning does two things: it caches file entries and minimizes source file contents. We use the term "ignored file" in a couple of places, but it's not clear what exactly that means. This commit clears up the semantics, explicitly spelling out this relates to minimization.

show more ...

[clang][deps] NFC: Rename scanning CompilerInstance

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not

[clang][deps] NFC: Rename scanning CompilerInstance

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.

This patch gives a distinct name to the `CompilerInstance` that's used to run the implicit build during dependency scan.

Depends on D111724.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111725

show more ...

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and c

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.

In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.

We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.

There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.

Depends on D102923.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D102488

show more ...

[clang][deps] Sanitize both instances of DiagnosticOptions

During dependency scanning, we generally want to suppress -Werror. Apply the same logic to the DiagnosticOptions instance used for command-

[clang][deps] Sanitize both instances of DiagnosticOptions

During dependency scanning, we generally want to suppress -Werror. Apply the same logic to the DiagnosticOptions instance used for command-line parsing.

This fixes a test failure on the PS4 bot, where the system header directory could not be found, which was reported due to -Werror being on the command line and not being sanitized.

show more ...

[clang][deps] Use correct DiagnosticOptions for command-line handling

In this patch the dependency scanner starts using proper `DiagnosticOptions` parsed from the actual TU command-line in order to

[clang][deps] Use correct DiagnosticOptions for command-line handling

In this patch the dependency scanner starts using proper `DiagnosticOptions` parsed from the actual TU command-line in order to mimic what the actual compiler would do. The actual functionality will be enabled and tested in follow-up patches. (This split is necessary to avoid temporary regression.)

Depends on D108976.

Reviewed By: dexonsmith, arphaman

Differential Revision: https://reviews.llvm.org/D108982

show more ...

[clang][deps] NFC: Remove CompilationDatabase from DependencyScanningWorker API

This patch simplifies the dependency scanner API. Depends on D108979.

Reviewed By: dexonsmith

Differential Revision:

[clang][deps] NFC: Remove CompilationDatabase from DependencyScanningWorker API

This patch simplifies the dependency scanner API. Depends on D108979.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108980

show more ...

[clang][deps] NFC: Stop going through ClangTool

The dependency scanner currently uses `ClangTool` to invoke the dependency scanning action.

However, `ClangTool` seems to be the wrong level of abstr

[clang][deps] NFC: Stop going through ClangTool

The dependency scanner currently uses `ClangTool` to invoke the dependency scanning action.

However, `ClangTool` seems to be the wrong level of abstraction. It's intended to be run over a collection of compile commands, which we actively avoid via `SingleCommandCompilationDatabase`. It automatically injects `-fsyntax-only` and other flags, which we avoid by calling `clearArgumentsAdjusters()`. It deduces the resource directory based on the current executable path, which we'd like to change to deducing from `argv[0]`.

Internally, `ClangTool` uses `ToolInvocation` which seems to be more in line with what the dependency scanner tries to achieve. This patch switches to directly using `ToolInvocation` instead. NFC.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108979

show more ...

[clang][deps] Stop using `ClangTool` for virtual files

This patch changes how the dependency scanner creates the fake input file when scanning dependencies of a single module (introduced in D109485)

[clang][deps] Stop using `ClangTool` for virtual files

This patch changes how the dependency scanner creates the fake input file when scanning dependencies of a single module (introduced in D109485). The scanner now has its own `InMemoryFilesystem` which sits under the minimizing FS (when that's requested). This makes it possible to drop the duplicate work in `DependencyScanningActions::runInvocation` that sets up the main file ID. Besides that, this patch makes it possible to land D108979, where we drop `ClangTool` entirely.

Depends on D109485.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D109498

show more ...

[clang-scan-deps] Add an API for clang dependency scanner to perform
module lookup by name alone

This removes the need to create a fake source file that imports a
module.

rdar://64538073

Different

[clang-scan-deps] Add an API for clang dependency scanner to perform
module lookup by name alone

This removes the need to create a fake source file that imports a
module.

rdar://64538073

Differential Revision: https://reviews.llvm.org/D109485

show more ...

[clang][deps] Avoid minimizing PCH input files

This patch avoid minimizing input files that contributed to a PCH or its modules. This prevents the implicit modular build to fail on unexpected file s

[clang][deps] Avoid minimizing PCH input files

This patch avoid minimizing input files that contributed to a PCH or its modules. This prevents the implicit modular build to fail on unexpected file size. Depends on D106146.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D104536

show more ...

[clang][deps] Normalize ignored filenames in minimizing file system

This patch normalizes filenames in `DependencyScanningWorkerFilesystem` so that lookup of ignored files works correctly on Windows

[clang][deps] Normalize ignored filenames in minimizing file system

This patch normalizes filenames in `DependencyScanningWorkerFilesystem` so that lookup of ignored files works correctly on Windows (where `/` and `\` are equivalent).

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D106064

show more ...

[clang][deps] NFC: Stop using moved-from object

The dependency scanning worker uses `std::move` to "reset" `DependencyOutputOptions` in the `CompilerInstance` that performs the implicit build. It's

[clang][deps] NFC: Stop using moved-from object

The dependency scanning worker uses `std::move` to "reset" `DependencyOutputOptions` in the `CompilerInstance` that performs the implicit build. It's probably preferable to replace the object with value-initialized instance, rather than depending on the behavior of a moved-from object.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D104106

show more ...

[clang][deps] NFC: Handle `DependencyOutputOptions` only once

There's no need to pass `DependencyOutputOptions` to each call of `handleFileDependency`, since the options don't ever change.

This pat

[clang][deps] NFC: Handle `DependencyOutputOptions` only once

There's no need to pass `DependencyOutputOptions` to each call of `handleFileDependency`, since the options don't ever change.

This patch adds new `handleDependencyOutputOpts` method to the `DependencyConsumer` interface and the dependency scanner uses it to report the options only once.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D104104

show more ...

[clang][deps] Prevent unintended modifications of the original TU command-line

One of the goals of the dependency scanner is to generate command-lines that can be used to explicitly build modular de

[clang][deps] Prevent unintended modifications of the original TU command-line

One of the goals of the dependency scanner is to generate command-lines that can be used to explicitly build modular dependencies of a translation unit. The only modifications to these command-lines should be for the purposes of explicit modular build.

However, the current version of dependency scanner leaks its implementation details into the command-lines.

The first problem is that the `clang-scan-deps` tool adjusts the original textual command-line (adding `-Eonly -M -MT <target> -sys-header-deps -Wno-error -o /dev/null `, removing `--serialize-diagnostics`) in order to set up the `DependencyScanning` library. This has been addressed in D103461, D104012, D104030, D104031, D104033. With these patches, the `DependencyScanning` library receives the unmodified `CompilerInvocation`, sets it up and uses it for the implicit modular build.

Finally, to prevent leaking the implementation details to the resulting command-lines, this patch generates them from the **original** unmodified `CompilerInvocation` rather than from the one that drives the implicit build.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D104036

show more ...

[clang][deps] Move enabling system header deps from `clang-scan-deps` to `DependencyScanning` library

This patch moves enabling system header deps from `clang-scan-deps` into the `DependencyScanning

[clang][deps] Move enabling system header deps from `clang-scan-deps` to `DependencyScanning` library

This patch moves enabling system header deps from `clang-scan-deps` into the `DependencyScanning` library. This will make it easier to preserve semantics of the original TU command-line for modular dependencies (see D104036).

Reviewed By: arphaman

Differential Revision: https://reviews.llvm.org/D104033

show more ...

[clang][deps] Move injection of `-Wno-error` from `clang-scan-deps` to `DependencyScanning` library

This moves another piece of logic specific to `clang-scan-deps` into the `DependencyScanning` libr

[clang][deps] Move injection of `-Wno-error` from `clang-scan-deps` to `DependencyScanning` library

This moves another piece of logic specific to `clang-scan-deps` into the `DependencyScanning` library. This makes it easier to check how the original command-line looked like in the library and will enable the library to stop inventing `-Wno-error` for modular dependencies (see D104036).

Reviewed By: arphaman

Differential Revision: https://reviews.llvm.org/D104031

show more ...

[clang][deps] Move invocation adjustments from `clang-scan-deps` to `DependencyScanning` library

The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. T

[clang][deps] Move invocation adjustments from `clang-scan-deps` to `DependencyScanning` library

The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. The goal is to setup `DependencyOutputOptions` by injecting `-M -MT <target>` and prevent the creation of output files.

This patch moves the logic into the `DependencyScanning` library, and uses the parsed `CompilerInvocation` instead of the raw command-line. The code simpler and can be used from the C++ API as well.

The `-o /dev/null` arguments are not necessary, since the `DependencyScanning` library only runs a preprocessing action, so there's no way it'll produce an actual object file.

Related: The `-M` argument implies `-w`, which would appear on the command-line of modular dependencies even though it was not on the original TU command line (see D104036).

Some related tests were updated.

Reviewed By: arphaman

Differential Revision: https://reviews.llvm.org/D104030

show more ...