[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special

[Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.

Differential Revision: https://reviews.llvm.org/D125484

show more ...

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appsp

Revert "[clang][deps] NFC: De-duplicate clang-cl tests"

This reverts commit 30cb49b44e4e2af2e0627b2aa33df0ba57ab9e55.
It caused test failures on Fuchsia Toolchain Mac builds:
https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-mac-x64/b8818888168677577537/overview

show more ...

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, th

[clang][deps] NFC: De-duplicate clang-cl tests

In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.

We don't need to duplicate all test cases, though. Testing the few special cases we have in `clang-scan-deps` for `clang-cl` should be good enough:

1. Deducing output path (and therefore target name in our make output).
2. Ignoring `-Xclang` arguments in step 1.
3. Deducing resource directory by invoking the compiler executuable.

This test de-duplicates the extra clang-cl test cases.

Reviewed By: dexonsmith, saudi

Differential Revision: https://reviews.llvm.org/D121812

show more ...

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `File

[clang][deps] Simplify PREFIX definitions in tests

Instead of outputting the test directory into the JSON result file, parsing it with `FileCheck` and then potentially stripping it, simply use `FileCheck`'s `-D` option.

Note that we use `%/t` instead of `%t` in order to normalize to forward slashes on Windows, which matches what we do with `sed 's:\\\\\?:/:g'`.

Differential Revision: https://reviews.llvm.org/D121516

show more ...

[clang][deps] Generate '-fmodule-file=' only for direct dependencies

The `clang-scan-deps` tool currently generates `-fmodule-file=` command-line arguments for the whole transitive closure of modula

[clang][deps] Generate '-fmodule-file=' only for direct dependencies

The `clang-scan-deps` tool currently generates `-fmodule-file=` command-line arguments for the whole transitive closure of modular dependencies. This is not necessary, we only need to provide the direct dependencies on the command line. Information about transitive dependencies is stored within the `.pcm` files of direct dependencies. This makes the command lines shorter, but should be a NFC otherwise (unless there are bugs in the loading mechanism for explicit modules).

Depends on D120465.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118915

show more ...

[clang][deps] Disable implicit module maps

Since D113473, we don't report any module map files via `-fmodule-map-file=` in explicit builds. The ultimate goal here is to make sure Clang doesn't open/

[clang][deps] Disable implicit module maps

Since D113473, we don't report any module map files via `-fmodule-map-file=` in explicit builds. The ultimate goal here is to make sure Clang doesn't open/read/parse/evaluate unnecessary module maps.

However, implicit module maps still end up reading all reachable module maps. This patch disables implicit module maps in explicit builds.

Unfortunately, we still need to report some module map files that aren't encoded in PCM files of dependencies: module maps that are necessary to correctly evaluate includes in modules marked as `[no_undeclared_includes]`.

Depends on D120464.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D120465

show more ...

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation uni

[clang][deps] Return the whole TU command line

The dependency scanner already generates canonical -cc1 command lines that can be used to compile discovered modular dependencies.

For translation unit command lines, the scanner only generates additional driver arguments the build system is expected to append to the original command line.

While this works most of the time, there are situations where that's not the case. For example with `-Wunused-command-line-argument`, Clang will complain about the `-fmodules-cache-path=` argument that's not being used in explicit modular builds. Combine that with `-Werror` and the build outright fails.

To prevent such failures, this patch changes the dependency scanner to return the full driver command line to compile the original translation unit. This gives us more opportunities to massage the arguments into something reasonable.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118986

show more ...

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fn

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.

Reviewed By: bnbarham

Differential Revision: https://reviews.llvm.org/D113473

show more ...

[clang][deps] NFC: Fix an XFAIL test on Windows

The `sed` command ensures Windows-specific path separators (single and double backslashes) are replaced by forward slashes in the output file. FileChe

[clang][deps] NFC: Fix an XFAIL test on Windows

The `sed` command ensures Windows-specific path separators (single and double backslashes) are replaced by forward slashes in the output file. FileCheck can continue using forward slashes in paths this way.

show more ...

[clang][deps] NFC: Check the correct context hashes in tests

The `clang-scan-deps` tests for the full output format were written under the assumption that most TUs/modules have the same context hash

[clang][deps] NFC: Check the correct context hashes in tests

The `clang-scan-deps` tests for the full output format were written under the assumption that most TUs/modules have the same context hash. This is no longer true, since we're changing the original compilation options. This patch updates the tests, which no longer conflate multiple context hashes into a single FileCheck variable.

show more ...

[clang][deps] Add argument for customizing PCM paths

Dependency scanning currently performs an implicit build. When testing that Clang can build modules with the command-lines generated by `clang-sc

[clang][deps] Add argument for customizing PCM paths

Dependency scanning currently performs an implicit build. When testing that Clang can build modules with the command-lines generated by `clang-scan-deps`, the actual compilation would overwrite artifacts created during the scan, which makes debugging harder than it should be and can lead to errors in multi-step builds.

To prevent this, this patch adds new flag to `clang-scan-deps` that allows developers to customize the directory to use when generating module map paths, instead of always using the module cache. Moreover, the explicit context hash in now part of the PCM path, which will be useful in D102488, where the context hash can change due to command-line pruning.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D103516

show more ...

Reapply "[clang][deps] Support inferred modules"

This reverts commit 76b8754d and ensures the PCM files are created in the correct directory (not in the current working directory).

Revert "Reapply "[clang][deps] Support inferred modules""

This reverts commit c98833cdaad01787ea70ecdfabb05a7e142a6671.
The test `ClangScanDeps/modules-inferred-explicit-build.m` creates files
in th

Revert "Reapply "[clang][deps] Support inferred modules""

This reverts commit c98833cdaad01787ea70ecdfabb05a7e142a6671.
The test `ClangScanDeps/modules-inferred-explicit-build.m` creates files
in the current directory.

show more ...

Reapply "[clang][deps] Support inferred modules"

This reapplies commit 95033eb3 that reverted commit 1d9e8e13.

The tests were failing on Windows due to spaces and backslashes in paths not being han

Reapply "[clang][deps] Support inferred modules"

This reapplies commit 95033eb3 that reverted commit 1d9e8e13.

The tests were failing on Windows due to spaces and backslashes in paths not being handled carefully.

show more ...

Revert "[clang][deps] Support inferred modules"

This reverts commit 1d9e8e13

Something is off with handling Windows paths: http://45.33.8.238/win/38908/step_7.txt
Most likely introduced in https://

Revert "[clang][deps] Support inferred modules"

This reverts commit 1d9e8e13

Something is off with handling Windows paths: http://45.33.8.238/win/38908/step_7.txt
Most likely introduced in https://reviews.llvm.org/D102491

show more ...

[clang][deps] Support inferred modules

This patch adds support for inferred modules to the dependency scanner.

Effectively a cherry-pick of https://github.com/apple/llvm-project/pull/699 authored b

[clang][deps] Support inferred modules

This patch adds support for inferred modules to the dependency scanner.

Effectively a cherry-pick of https://github.com/apple/llvm-project/pull/699 authored by @Bigcheese with libclang and other changes omitted.

Contains following changes:

1. [Clang][ScanDeps] Ignore __inferred_module.map dependency.
* This shows up with inferred modules, but it doesn't exist on disk, so don't report it as a dependency.

2. [Clang][ScanDeps] Use the module map a module was inferred from for inferred modules.

Also includes a smoke test that uses clang-scan-deps output to perform an explicit build. There's no intention to duplicate whatever `test/Modules` contains, just to verify the produced command-line does "work" (with very loose definition of work).

Split from D100934.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D102495

show more ...

[clang][deps] Only generate absolute paths when asked to

Add option to `clang-scan-deps` to enable/disable generation of command-line arguments with absolute paths. This is essentially a revert of D

[clang][deps] Only generate absolute paths when asked to

Add option to `clang-scan-deps` to enable/disable generation of command-line arguments with absolute paths. This is essentially a revert of D100533, but with improved naming and added test.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D101051

show more ...

[clang][deps] Check extra args in tests

These flags are being generated by `clang-scan-deps` and it makes sense to ensure it keeps doing so.

[clang][deps] Include "-cc1" in the arguments

To simplify tools consuming dependency scanning results, prepend the "-cc1" argument by default.

Reviewed By: Bigcheese

Differential Revision: https:/

[clang][deps] Include "-cc1" in the arguments

To simplify tools consuming dependency scanning results, prepend the "-cc1" argument by default.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D100942

show more ...

Reland "[clang-scan-deps] Add support for clang-cl"

This reverts commit 199c39748292cbc89cd148a0d8364ebb1014ec38.
This time, clang-scan-deps's search for output argument in clang-cl command line wil

Reland "[clang-scan-deps] Add support for clang-cl"

This reverts commit 199c39748292cbc89cd148a0d8364ebb1014ec38.
This time, clang-scan-deps's search for output argument in clang-cl command line will now ignore arguments preceded by "-Xclang".
That way, it won't detect a /o argument in "-Xclang -ivfsoverlay -Xclang /opt/subpath"

Initial patch description:
clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews.llvm.org/D92191

show more ...

Revert "[clang-scan-deps] Add support for clang-cl"

This reverts commit bb26fa8c286bf524ed9235c3e293ad22ecf3e984.

[clang][deps] Generate the full command-line for modules

This patch uses the new `CompilerInvocation::generateCC1CommandLine` to generate the full canonical command line for modular dependencies, in

[clang][deps] Generate the full command-line for modules

This patch uses the new `CompilerInvocation::generateCC1CommandLine` to generate the full canonical command line for modular dependencies, instead of only appending additional arguments.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D100534

show more ...

[clang][deps] Remove the -full-command-line flag

This patch removes the `-full-command-line` option from `clang-scan-deps`. It's only used with `-format=experimental-full`, where omitting the comman

[clang][deps] Remove the -full-command-line flag

This patch removes the `-full-command-line` option from `clang-scan-deps`. It's only used with `-format=experimental-full`, where omitting the command lines doesn't make much sense. There are no tests without `-full-command-line`.

Depends on D100531.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D100533

show more ...

[clang][deps] Simplify function discovering .pcm and .modulemap files

This patch simplifies (and renames) the `appendCommonModuleArguments` function.

It no longer tries to construct the command lin

[clang][deps] Simplify function discovering .pcm and .modulemap files

This patch simplifies (and renames) the `appendCommonModuleArguments` function.

It no longer tries to construct the command line for explicitly building modules. Instead, it only performs the DFS traversal of modular dependencies and queries the callbacks to collect paths to `.pcm` and `.modulemap` files.

This makes it more flexible and usable in two contexts:

* Generating additional command line arguments for the main TU in modular build. The `std::vector<std::string>` output parameters can be used to manually generate appropriate command line flags.
* Generate full command line for a module. The output parameters can be the corresponding parts of `CompilerInvocation`. (In a follow-up patch.)

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D100531

show more ...

[clang-scan-deps] Add support for clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https:/

[clang-scan-deps] Add support for clang-cl

clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.

Differential Revision: https://reviews.llvm.org/D92191

show more ...