[clang][deps] Fix handling of -MT in module command-line

Follow-up to 6626f6fec3d3, this fixes the handling of -MT
* If no targets are provided, we need to invent one since cc1 expects
the driver

[clang][deps] Fix handling of -MT in module command-line

Follow-up to 6626f6fec3d3, this fixes the handling of -MT
* If no targets are provided, we need to invent one since cc1 expects
the driver to have handled it. The default is to use -o, quoting as
necessary for a make target.
* Fix the splitting for empty string, which was incorrectly treated as
{""} instead of {}.
* Add a way to test this behaviour in clang-scan-deps.

Differential Revision: https://reviews.llvm.org/D129607

show more ...

[clang][deps] Override dependency and serialized diag files for modules

When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in additio

[clang][deps] Override dependency and serialized diag files for modules

When building modules, override secondary outputs (dependency file,
dependency targets, serialized diagnostic file) in addition to the pcm
file path. This avoids inheriting per-TU command-line options that
cause non-determinism in the results (non-deterministic command-line for
the module build, non-determinism in which TU's .diag and .d files will
contain the module outputs). In clang-scan-deps we infer whether to
generate dependency or serialized diagnostic files based on an original
command-line. In a real build system this should be modeled explicitly.

Differential Revision: https://reviews.llvm.org/D129389

show more ...

[clang][deps] Sort submodules when calculating dependencies

Dependency scanning does not care about the order of submodules for
correctness, so sort the submodules so that we get the same
command-li

[clang][deps] Sort submodules when calculating dependencies

Dependency scanning does not care about the order of submodules for
correctness, so sort the submodules so that we get the same
command-lines to build the module across different TUs. The order of
inferred submodules can vary depending on the order of #includes in the
including TU.

Differential Revision: https://reviews.llvm.org/D128008

show more ...

[clang][deps] Further canonicalize implicit modules options in dep scan

Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for t

[clang][deps] Further canonicalize implicit modules options in dep scan

Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for the modules
cache path. This reduces uninteresting differences between
command-lines, which is particularly useful if there is a tool that can
cache the compilations.

Differential Revision: https://reviews.llvm.org/D127883

show more ...

[clang][deps] Make order of module dependencies deterministic

This fixes the underlying module dependencies, which had a
non-deterministic order, which was also visible in the order of calls to
Depe

[clang][deps] Make order of module dependencies deterministic

This fixes the underlying module dependencies, which had a
non-deterministic order, which was also visible in the order of calls to
DependencyConsumer methods. This was not directly observable in
the clang-scan-deps utility, because it was previously seeing a sorted
order from std::map in DependencyScanningTool. However, the underlying
API previously created a likely issue for any other clients. Note: if
you only apply the change from DependencyScanningTool, you can see the
issue in clang-scan-deps, and existing tests will fail
non-deterministicaly.

Differential Revision: https://reviews.llvm.org/D127243

show more ...

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<FileEntryRef>`.

With the API change in place, this patch then removes some uses of the deprecated `FileEntry::getName()` (e.g. in `DependencyGraph.cpp` and `ModuleDependencyCollector.cpp`).

Reviewed By: dexonsmith, bnbarham

Differential Revision: https://reviews.llvm.org/D123574

show more ...

[clang][deps] Don't prune search paths used by dependencies

When pruning header search paths (to reduce the number of modules we need to build explicitly), we can't prune the search paths used in (t

[clang][deps] Don't prune search paths used by dependencies

When pruning header search paths (to reduce the number of modules we need to build explicitly), we can't prune the search paths used in (transitive) dependencies of a module. Otherwise, we could end up with either of the following dependency graphs:

```
X:<hash1> -> Y:<hash2>
X:<hash1> -> Y:<hash3>
```

depending on the search paths of the translation unit we discovered `X` and `Y` from.

This patch fixes that.

Depends on D121295.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D121303

show more ...

[clang][deps] Remove '-fmodules-cache-path=' arguments

With explicit modules build, the '-fmodules-cache-path=' argument is unused.

This patch removes the argument to avoid warnings or errors (with

[clang][deps] Remove '-fmodules-cache-path=' arguments

With explicit modules build, the '-fmodules-cache-path=' argument is unused.

This patch removes the argument to avoid warnings or errors (with '-Werror') stemming from that.

Depends on D118915.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D120474

show more ...

[clang][deps] Generate '-fmodule-file=' only for direct dependencies

The `clang-scan-deps` tool currently generates `-fmodule-file=` command-line arguments for the whole transitive closure of modula

[clang][deps] Generate '-fmodule-file=' only for direct dependencies

The `clang-scan-deps` tool currently generates `-fmodule-file=` command-line arguments for the whole transitive closure of modular dependencies. This is not necessary, we only need to provide the direct dependencies on the command line. Information about transitive dependencies is stored within the `.pcm` files of direct dependencies. This makes the command lines shorter, but should be a NFC otherwise (unless there are bugs in the loading mechanism for explicit modules).

Depends on D120465.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D118915

show more ...

[clang][deps] Disable implicit module maps

Since D113473, we don't report any module map files via `-fmodule-map-file=` in explicit builds. The ultimate goal here is to make sure Clang doesn't open/

[clang][deps] Disable implicit module maps

Since D113473, we don't report any module map files via `-fmodule-map-file=` in explicit builds. The ultimate goal here is to make sure Clang doesn't open/read/parse/evaluate unnecessary module maps.

However, implicit module maps still end up reading all reachable module maps. This patch disables implicit module maps in explicit builds.

Unfortunately, we still need to report some module map files that aren't encoded in PCM files of dependencies: module maps that are necessary to correctly evaluate includes in modules marked as `[no_undeclared_includes]`.

Depends on D120464.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D120465

show more ...

Re-land "[Clang][ScanDeps] Use the virtual path for module maps"

This re-lands:
- 04192422c4e3b730c580498b8e948088cb15580b
- 015e08c6badad6b27404d6f94569e25c18d79049

Which I reverted in ea835171389

Re-land "[Clang][ScanDeps] Use the virtual path for module maps"

This re-lands:
- 04192422c4e3b730c580498b8e948088cb15580b
- 015e08c6badad6b27404d6f94569e25c18d79049

Which I reverted in ea835171389aa356b865bf9cb72ca8f4f84b64fd in error.

Differential Revision: https://reviews.llvm.org/D114206

show more ...

Revert "[Clang][ScanDeps] Use the virtual path for module maps"

This reverts commits:
- 04192422c4e3b730c580498b8e948088cb15580b.
- 015e08c6badad6b27404d6f94569e25c18d79049

D114206 was landed befor

Revert "[Clang][ScanDeps] Use the virtual path for module maps"

This reverts commits:
- 04192422c4e3b730c580498b8e948088cb15580b.
- 015e08c6badad6b27404d6f94569e25c18d79049

D114206 was landed before it was approved - and was landed knowing that
the test crashed on windows, without an xfail. The promised follow-up
commit with fixes has not appeared since it was promised on December 14th.

show more ...

[Clang][ScanDeps] Use the virtual path for module maps

Make clang-scan-deps use the virtual path for module maps instead of the on disk
path. This is needed so that modulemap relative lookups are do

[Clang][ScanDeps] Use the virtual path for module maps

Make clang-scan-deps use the virtual path for module maps instead of the on disk
path. This is needed so that modulemap relative lookups are done correctly in
the actual module builds. The file dependencies still use the on disk path as
that's what matters for build invalidation.

Differential Revision: https://reviews.llvm.org/D114206

show more ...

[clang][deps] Reset some benign codegen options

Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes the

[clang][deps] Reset some benign codegen options

Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes them to make builds deterministic and/or reduce the number of built modules.

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D112923

show more ...

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fn

[clang][deps] Don't emit `-fmodule-map-file=`

During explicit modules build, when all modules are provided via `-fmodule-file=<path>` and implicit modules and implicit module maps are disabled (`-fno-implicit-modules`, `-fno-implicit-module-maps`), we don't need to load the original module map files at all. This patch stops emitting the `-fmodule-map-file=` arguments we don't need, saving some compilation time due to avoiding parsing such module maps and making the command line shorter.

Reviewed By: bnbarham

Differential Revision: https://reviews.llvm.org/D113473

show more ...

[clang][deps] NFC: Rename building CompilerInvocation

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is n

[clang][deps] NFC: Rename building CompilerInvocation

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.

This patch gives descriptive name to the generated `CompilerInvocation` that can be used to derive the command-line to build a modular dependency.

Depends on D111725.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111728

show more ...

[clang][deps] NFC: Rename scanning CompilerInstance

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not

[clang][deps] NFC: Rename scanning CompilerInstance

The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.

This patch gives a distinct name to the `CompilerInstance` that's used to run the implicit build during dependency scan.

Depends on D111724.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111725

show more ...

[clang][deps] NFC: Remove redundant CompilerInstance reference

The `ModuleDepCollectorPP` class holds a reference to `ModuleDepCollector` as well as `ModuleDepCollector`'s `CompilerInstance`. The fa

[clang][deps] NFC: Remove redundant CompilerInstance reference

The `ModuleDepCollectorPP` class holds a reference to `ModuleDepCollector` as well as `ModuleDepCollector`'s `CompilerInstance`. The fact that these refer to the same object is non-obvious.

This patch removes the `CompilerInvocation` reference from `ModuleDepCollectorPP` and accesses it through `ModuleDepCollector` instead.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111724

show more ...

[clang][deps] Ensure reported context hash is strict

One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch

[clang][deps] Ensure reported context hash is strict

One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch ensures that strict context hash is enabled not only during the scan itself (and its minimized implicit build), but also when actually reporting the dependency.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111720

show more ...

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and c

[clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.

In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.

We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.

There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.

Depends on D102923.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D102488

show more ...

[clang][deps] Reset non-modular language and preprocessor options

There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an im

[clang][deps] Reset non-modular language and preprocessor options

There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an implicit module. This patch uses the logic for explicit modules as well.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108710

show more ...

[clang][deps] Use top-level modules as precompiled dependencies

The `ASTReader` populates `Module::PresumedModuleMapFile` only for top-level modules, not submodules. To avoid generating empty `-fmod

[clang][deps] Use top-level modules as precompiled dependencies

The `ASTReader` populates `Module::PresumedModuleMapFile` only for top-level modules, not submodules. To avoid generating empty `-fmodule-map-file=` arguments, make discovered modules depend on top-level precompiled modules. The granularity of submodules is not important here.

The documentation of `Module::PresumedModuleMapFile` says this field is non-empty only when building from preprocessed source. This means there can still be cases where the dependency scanner generates empty `-fmodule-map-file=` arguments. That's being addressed in separate patch: D108544.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108647

show more ...

[clang][deps] Collect precompiled deps from submodules too

In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level m

[clang][deps] Collect precompiled deps from submodules too

In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level modules.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D108540

show more ...

[llvm][clang][NFC] updates inline licence info

Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception li

[llvm][clang][NFC] updates inline licence info

Some files still contained the old University of Illinois Open Source
Licence header. This patch replaces that with the Apache 2 with LLVM
Exception licence.

Differential Revision: https://reviews.llvm.org/D107528

show more ...

[clang-scan-deps] ignore top-level module dependencies that aren't actually imported

Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the

[clang-scan-deps] ignore top-level module dependencies that aren't actually imported

Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the headers are imported textually and the module isn't actually
built. However, the dependency scanner could still record it as a potential dependency
if the module was reimported and thus recorded by the preprocessor callbacks.
This change avoids collecting this kind of module as a dependency by verifying that we don't
collect top level modules without actual PCM files.

Differential Revision: https://reviews.llvm.org/D106100

show more ...