[C++20][Modules] Implement include translation.

This addresses [cpp.include]/7

(when encountering #include header-name)

If the header identified by the header-name denotes an importable header, it

[C++20][Modules] Implement include translation.

This addresses [cpp.include]/7

(when encountering #include header-name)

If the header identified by the header-name denotes an importable header, it
is implementation-defined whether the #include preprocessing directive is
instead replaced by an import directive.

In this implementation, include translation is performed _only_ for headers
in the Global Module fragment, so:
```
module;
#include "will-be-translated.h" // IFF the header unit is available.

export module M;
#include "will-not-be-translated.h" // even if the header unit is available
```
The reasoning is that, in general, includes in the module purview would not
be validly translatable (they would have to immediately follow the module
decl and without any other intervening decls). Otherwise that would violate
the rules on contiguous import directives.

This would be quite complex to track in the preprocessor, and for relatively
little gain (the user can 'import "will-not-be-translated.h";' instead.)

TODO: This is one area where it becomes increasingly difficult to disambiguate
clang modules in C++ from C++ standard modules. That needs to be addressed in
both the driver and the FE.

Differential Revision: https://reviews.llvm.org/D128981

show more ...

[Clang][Preprocessor] Fix inconsistent `FLT_EVAL_METHOD` when compiling vs preprocessing

When running `clang -E -Ofast` on macOS, the `__FLT_EVAL_METHOD__` macro is `0`, which causes the following t

[Clang][Preprocessor] Fix inconsistent `FLT_EVAL_METHOD` when compiling vs preprocessing

When running `clang -E -Ofast` on macOS, the `__FLT_EVAL_METHOD__` macro is `0`, which causes the following typedef to be emitted into the preprocessed source: `typedef float float_t`.

However, when running `clang -c -Ofast`, `__FLT_EVAL_METHOD__` is `-1`, and `typedef long double float_t` is emitted.

This causes build errors for certain projects, which are not reproducible when compiling from preprocessed source.

The issue is that `__FLT_EVAL_METHOD__` is configured in `Sema::Sema` which is not executed when running in `-E` mode.

This change moves that logic into the preprocessor initialization method, which is invoked correctly in `-E` mode.

rdar://96134605
rdar://92748429

Differential Revision: https://reviews.llvm.org/D128814

show more ...

[Clang] Rename StringLiteral::isAscii() => isOrdinary() [NFC]

"Ascii" StringLiteral instances are actually narrow strings
that are UTF-8 encoded and do not have an encoding prefix.
(UTF8 StringLiter

[Clang] Rename StringLiteral::isAscii() => isOrdinary() [NFC]

"Ascii" StringLiteral instances are actually narrow strings
that are UTF-8 encoded and do not have an encoding prefix.
(UTF8 StringLiteral are also UTF-8 encoded strings, but with
the u8 prefix.

To avoid possible confusion both with actuall ASCII strings,
and with future works extending the set of literal encodings
supported by clang, this rename StringLiteral::isAscii() to
isOrdinary(), matching C++ standard terminology.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D128762

show more ...

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionality

Removes `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blocks
will be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.

* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

Replaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.

* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`

This is bringing the following benefits:

* Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
* Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time.
* Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).

For normal preprocessing measurements show differences are below the noise level.

Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.

Differential Revision: https://reviews.llvm.org/D125486
Differential Revision: https://reviews.llvm.org/D125487
Differential Revision: https://reviews.llvm.org/D125488

show more ...

[clang][preprocessor] Allow calling DumpToken() on annotation tokens

Differential Revision: https://reviews.llvm.org/D122659

Add support for floating-point option `ffp-eval-method` and for
`pragma clang fp eval_method`.

Differential Revision: https://reviews.llvm.org/D109239

[clang][lex] Include tracking: simplify and move to preprocessor

This patch replaces the exact include count of each file in `HeaderFileInfo` with a set of included files in `Preprocessor`.

The num

[clang][lex] Include tracking: simplify and move to preprocessor

This patch replaces the exact include count of each file in `HeaderFileInfo` with a set of included files in `Preprocessor`.

The number of includes isn't a property of a header file but rather a preprocessor state. The exact number of includes is not used anywhere except statistic tracking.

Reviewed By: vsapsai

Differential Revision: https://reviews.llvm.org/D114095

show more ...

[clang][lex] NFC: Simplify calls to `LookupFile`

The `{HeaderSearch,Preprocessor}::LookupFile()` functions take an out-parameter `const DirectoryLookup *&`. Most callers end up creating a `const Dir

[clang][lex] NFC: Simplify calls to `LookupFile`

The `{HeaderSearch,Preprocessor}::LookupFile()` functions take an out-parameter `const DirectoryLookup *&`. Most callers end up creating a `const DirectoryLookup *` variable that's otherwise unused.

This patch changes the out-parameter from reference to a pointer, making it possible to simply pass `nullptr` to the function without the ceremony.

Reviewed By: ahoppen

Differential Revision: https://reviews.llvm.org/D117312

show more ...

[clang] Use llvm::is_contained (NFC)

[clang][modules] Default `SourceLocation` parameter in `HeaderSearch::lookupModule`

This fixes an LLDB build failure where the `ImportLoc` argument is missing: https://lab.llvm.org/buildbot#builders

[clang][modules] Default `SourceLocation` parameter in `HeaderSearch::lookupModule`

This fixes an LLDB build failure where the `ImportLoc` argument is missing: https://lab.llvm.org/buildbot#builders/68/builds/19975

This change also makes it possible to drop `SourceLocation()` in `Preprocessor::getCurrentModule`.

show more ...

[clang][modules] NFC: Propagate import `SourceLocation` into `HeaderSearch::lookupModule`

This patch propagates the import `SourceLocation` into `HeaderSearch::lookupModule`. This enables remarks on

[clang][modules] NFC: Propagate import `SourceLocation` into `HeaderSearch::lookupModule`

This patch propagates the import `SourceLocation` into `HeaderSearch::lookupModule`. This enables remarks on search path usage (implemented in D102923) to point to the source code that initiated header search.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D111557

show more ...

Implement #pragma clang final extension

This patch adds a new preprocessor extension ``#pragma clang final``
which enables warning on undefinition and re-definition of macros.

The intent of this wa

Implement #pragma clang final extension

This patch adds a new preprocessor extension ``#pragma clang final``
which enables warning on undefinition and re-definition of macros.

The intent of this warning is to extend beyond ``-Wmacro-redefined`` to
warn against any and all alterations to macros that are marked `final`.

This warning is part of the ``-Wpedantic-macros`` diagnostics group.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D108567

show more ...

Effectively revert 33c3d8a916c / D33782

This change would treat the token `or` in system headers as an
identifier, and elsewhere as an operator. As reported in
llvm.org/pr42427, many users classify

Effectively revert 33c3d8a916c / D33782

This change would treat the token `or` in system headers as an
identifier, and elsewhere as an operator. As reported in
llvm.org/pr42427, many users classify their third party library headers
as "system" headers to suppress warnings. There's no clean way to
separate Windows SDK headers from user headers.

Clang is still able to parse old Windows SDK headers if C++ operator
names are disabled. Traditionally this was controlled by
`-fno-operator-names`, but is now also enabled with `/permissive` since
D103773. This change will prevent `clang-cl` from parsing <query.h> from
the Windows SDK out of the box, but there are multiple ways to work
around that:
- Pass `/clang:-fno-operator-names`
- Pass `/permissive`
- Pass `-DQUERY_H_RESTRICTION_PERMISSIVE`

In all of these modes, the operator names will consistently be available
or not available, instead of depending on whether the code is in a
system header.

I added a release note for this, since it may break straightforward
users of the Windows SDK.

Fixes PR42427

Differential Revision: https://reviews.llvm.org/D108720

show more ...

Implement #pragma clang restrict_expansion

This patch adds `#pragma clang restrict_expansion ` to enable flagging
macros as unsafe for header use. This is to allow macros that may have
ABI implicati

Implement #pragma clang restrict_expansion

This patch adds `#pragma clang restrict_expansion ` to enable flagging
macros as unsafe for header use. This is to allow macros that may have
ABI implications to be avoided in headers that have ABI stability
promises.

Using macros in headers (particularly public headers) can cause a
variety of issues relating to ABI and modules. This new pragma logs
warnings when using annotated macros outside the main source file.

This warning is added under a new diagnostics group -Wpedantic-macros

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D107095

show more ...

Support macro deprecation #pragma clang deprecated

This patch adds `#pragma clang deprecated` to enable deprecation of
preprocessor macros.

The macro must be defined before `#pragma clang deprecate

Support macro deprecation #pragma clang deprecated

This patch adds `#pragma clang deprecated` to enable deprecation of
preprocessor macros.

The macro must be defined before `#pragma clang deprecated`. When
deprecating a macro a custom message may be optionally provided.

Warnings are emitted at the use site of a deprecated macro, and can be
controlled via the `-Wdeprecated` warning group.

This patch takes some rough inspiration and a few lines of code from
https://reviews.llvm.org/D67935.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D106732

show more ...

[Preprocessor] Implement -fminimize-whitespace.

This patch adds the -fminimize-whitespace with the following effects:

* If combined with -E, remove as much non-line-breaking whitespace as
possi

[Preprocessor] Implement -fminimize-whitespace.

This patch adds the -fminimize-whitespace with the following effects:

* If combined with -E, remove as much non-line-breaking whitespace as
possible.

* If combined with -E -P, removes as much whitespace as possible,
including line-breaks.

The motivation is to reduce the amount of insignificant changes in the
preprocessed output with source files where only whitespace has been
changed (add/remove comments, clang-format, etc.) which is in particular
useful with ccache.

A patch for ccache for using this flag has been proposed to ccache as well:
https://github.com/ccache/ccache/pull/815, which will use
-fnormalize-whitespace when clang-13 has been detected, and additionally
uses -P in "unify_mode". ccache already had a unify_mode in an older
version which was removed because of problems that using the
preprocessor itself does not have (such that the custom tokenizer did
not recognize C++11 raw strings).

This patch slightly reorganizes which part is responsible for adding
newlines that are required for semantics. It is now either
startNewLineIfNeeded() or MoveToLine() but never both; this avoids the
ShouldUpdateCurrentLine workaround and avoids redundant lines being
inserted in some cases. It also fixes a mandatory newline not inserted
after a _Pragma("...") that is expanded into a #pragma.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D104601

show more ...

Add support for #elifdef and #elifndef

WG14 adopted N2645 and WG21 EWG has accepted P2334 in principle (still
subject to full EWG vote + CWG review + plenary vote), which add
support for #elifdef as

Add support for #elifdef and #elifndef

WG14 adopted N2645 and WG21 EWG has accepted P2334 in principle (still
subject to full EWG vote + CWG review + plenary vote), which add
support for #elifdef as shorthand for #elif defined and #elifndef as
shorthand for #elif !defined. This patch adds support for the new
preprocessor directives.

show more ...

[CodeCompletion] Avoid spurious signature help for init-list args

Somewhat surprisingly, signature help is emitted as a side-effect of
computing the expected type of a function argument.
The reason

[CodeCompletion] Avoid spurious signature help for init-list args

Somewhat surprisingly, signature help is emitted as a side-effect of
computing the expected type of a function argument.
The reason is that both actions require enumerating the possible
function signatures and running partial overload resolution, and doing
this twice would be wasteful and complicated.

Change #1: document this, it's subtle :-)

However, sometimes we need to compute the expected type without having
reached the code completion cursor yet - in particular to allow
completion of designators.
eb4ab3358cd4dc834a761191b5531b38114f7b13 did this but introduced a
regression - it emits signature help in the wrong location as a side-effect.

Change #2: only emit signature help if the code completion cursor was reached.

Currently there is PP.isCodeCompletionReached(), but we can't use it
because it's set *after* running code completion.
It'd be nice to set this implicitly when the completion token is lexed,
but ConsumeCodeCompletionToken() makes this complicated.

Change #3: call cutOffParsing() *first* when seeing a completion token.

After this, the fact that the Sema::Produce*SignatureHelp() functions
are even more confusing, as they only sometimes do that.
I don't want to rename them in this patch as it's another large
mechanical change, but we should soon.

Change #4: prepare to rename ProduceSignatureHelp() to GuessArgumentType() etc.

Differential Revision: https://reviews.llvm.org/D98488

show more ...

Don't allow __VA_OPT__ to be detected by #ifdef.

More study has discovered this to not actually be useful: because
current C++20 implementations reject `#ifdef __VA_OPT__`, this can't
really be used

Don't allow __VA_OPT__ to be detected by #ifdef.

More study has discovered this to not actually be useful: because
current C++20 implementations reject `#ifdef __VA_OPT__`, this can't
really be used as a feature-test mechanism. And it's not too hard to
detect __VA_OPT__ without this, for example:

#define THIRD_ARG(a, b, c, ...) c
#define HAS_VA_OPT(...) THIRD_ARG(__VA_OPT__(,), 1, 0, )
#if HAS_VA_OPT(?)

Partially reverts 0436ec2128c9775ba13b0308937238fc79673fdd.

show more ...

Permit __VA_OPT__ in all language modes and allow it to be detected with #ifdef.

These changes are intended to give code a path to move away from the GNU
,##__VA_ARGS__ extension, which is non-confo

Permit __VA_OPT__ in all language modes and allow it to be detected with #ifdef.

These changes are intended to give code a path to move away from the GNU
,##__VA_ARGS__ extension, which is non-conforming in some situations and
which we'd like to disable in our conforming mode in those cases.

show more ...

clang/Basic: Replace SourceManager::getMemoryBufferForFile, NFC

Replace `SourceManager::getMemoryBufferForFile`, which returned a
dereferenceable `MemoryBuffer*` and had a `bool*Invalid` out paramet

clang/Basic: Replace SourceManager::getMemoryBufferForFile, NFC

Replace `SourceManager::getMemoryBufferForFile`, which returned a
dereferenceable `MemoryBuffer*` and had a `bool*Invalid` out parameter,
with `getMemoryBufferForFileOrNone` (returning
`Optional<MemoryBufferRef>`) and `getMemoryBufferForFileOrFake`
(returning `MemoryBufferRef`).

Differential Revision: https://reviews.llvm.org/D89429

show more ...

[Coverage] Add empty line regions to SkippedRegions

Differential Revision: https://reviews.llvm.org/D84988

Reland [Coverage] Add comment to skipped regions

Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=45757.
Add comment to skipped regions so we don't track execution count for lines containing o

Reland [Coverage] Add comment to skipped regions

Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=45757.
Add comment to skipped regions so we don't track execution count for lines containing only comments.

Differential Revision: https://reviews.llvm.org/D83592

show more ...

Revert abd45154b "[Coverage] Add comment to skipped regions"

This casued assertions during Chromium builds. See comment on the code review

> Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=4

Revert abd45154b "[Coverage] Add comment to skipped regions"

This casued assertions during Chromium builds. See comment on the code review

> Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=45757.
> Add comment to skipped regions so we don't track execution count for lines containing only comments.
>
> Differential Revision: https://reviews.llvm.org/D84208

This reverts commit abd45154bdb6b76c5b480455eacc8c75b08242aa and the
follow-up 87d725473380652bbe845fd2fbd9c0507a55172f.

show more ...

[Coverage] Add comment to skipped regions

Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=45757.
Add comment to skipped regions so we don't track execution count for lines containing only com

[Coverage] Add comment to skipped regions

Bug filled here: https://bugs.llvm.org/show_bug.cgi?id=45757.
Add comment to skipped regions so we don't track execution count for lines containing only comments.

Differential Revision: https://reviews.llvm.org/D84208

show more ...