[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<

[clang][lex] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective()

This patch changes type of the `File` parameter in `PPCallbacks::InclusionDirective()` from `const FileEntry *` to `Optional<FileEntryRef>`.

With the API change in place, this patch then removes some uses of the deprecated `FileEntry::getName()` (e.g. in `DependencyGraph.cpp` and `ModuleDependencyCollector.cpp`).

Reviewed By: dexonsmith, bnbarham

Differential Revision: https://reviews.llvm.org/D123574

show more ...

[C++20][Modules][HU 5/5] Add fdirectives-only mode for preprocessing output.

When the -fdirectives-only option is used together with -E, the preprocessor
output reflects evaluation of if/then/else d

[C++20][Modules][HU 5/5] Add fdirectives-only mode for preprocessing output.

When the -fdirectives-only option is used together with -E, the preprocessor
output reflects evaluation of if/then/else directives.

As such, it preserves defines and undefs of macros that are still live after
such processing. The intent is that this output could be consumed as input
to generate considered a C++20 header unit.

We strip out any (unused) defines that come from built-in, built-in-file or
command line; these are re-added when the preprocessed source is consumed.

Differential Revision: https://reviews.llvm.org/D121099

show more ...

[clang] fix out of bounds access in an empty string when lexing a _Pragma with missing string token

The lexer can attempt to lex a _Pragma and crash with an out of bounds string access when it's
lex

[clang] fix out of bounds access in an empty string when lexing a _Pragma with missing string token

The lexer can attempt to lex a _Pragma and crash with an out of bounds string access when it's
lexing a _Pragma whose string token is an invalid buffer, e.g. when a module header file from which the macro
expansion for that token was deleted from the file system.

Differential Revision: https://reviews.llvm.org/D116052

show more ...

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLEx

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLExtras, which will now
never be called.
- Calling it without llvm:: breaks C++17 compat

show more ...

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no buil

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no build
breakage expected).

show more ...

[clang] Use true/false instead of 1/0 (NFC)

Identified with modernize-use-bool-literals.

[Preprocessor] Fix newline before/after _Pragma.

The PragmaAssumeNonNullHandler (and maybe others) passes an invalid
SourceLocation to its callback, hence PrintPreprocessedOutput does not
know how m

[Preprocessor] Fix newline before/after _Pragma.

The PragmaAssumeNonNullHandler (and maybe others) passes an invalid
SourceLocation to its callback, hence PrintPreprocessedOutput does not
know how many lines to insert between the previous token and the
pragma and does nothing.

With this patch we instead assume that the unknown token is on the same
line as the previous such that we can call the procedure that also emits
semantically significant whitespace.

Fixes bug reported here: https://reviews.llvm.org/D104601#3105044

show more ...

[Preprocessor] Fix warning: left and right subexpressions are identical. NFCI.

This is reported by msvc as
warning C6287: redundant code: the left and right subexpressions are identical

EmittedDire

[Preprocessor] Fix warning: left and right subexpressions are identical. NFCI.

This is reported by msvc as
warning C6287: redundant code: the left and right subexpressions are identical

EmittedDirectiveOnThisLine implies EmittedTokensOnThisLine
making this an NFC change. To be on the safe side and because both of
them are checked at other places as well, we continue to check both.

Compiler warning reported here:
https://reviews.llvm.org/D104601#2957333

show more ...

[clang] Let PPCallbacks::PragmaWarning() pass specifier as enum instead of string

Differential Revision: https://reviews.llvm.org/D110635

[Preprocessor] Elide empty line(s) at start of file.

In -P mode, PrintPPOutputPPCallbacks::MoveToLine started at least one
newline if current and target line number mismatched. The method is also
ca

[Preprocessor] Elide empty line(s) at start of file.

In -P mode, PrintPPOutputPPCallbacks::MoveToLine started at least one
newline if current and target line number mismatched. The method is also
called when entering a new file, be it the main file or an include file.
In this situation line numbers always almost mismatch, resulting in a
newline for each occurance even if no tokens have been printed
in-between.

Empty lines at the beginning of the output must be trimmed because it
may be parsed by scripts expecting the result to appear on the first
output line, as done by LibreOffice's configure script.

Fix by only emitting a newline if tokens have been printed so far using
the EmittedTokensOnThisLine flag. Also adding a test case of FileChanged
callbacks occuring with empty include files.

This fixes llvm.org/PR51616

show more ...

Fix unknown parameter Wdocumentation warning. NFC.

[Preprocessor] Ensure newline after #pragma introduced by -fms-extensions.

The -fms-extensions converts __pragma (and _Pragma) into a #pragma that
has to occur at the beginning of a line and end wit

[Preprocessor] Ensure newline after #pragma introduced by -fms-extensions.

The -fms-extensions converts __pragma (and _Pragma) into a #pragma that
has to occur at the beginning of a line and end with a newline. This
patch ensures that the newline after the #pragma is added even if
Token::isAtStartOfLine() indicated that we should not start a newline.

Committing relying post-commit review since the change is small, some
downstream uses might be blocked without this fix, and to make clear the
decision of the new -fminimize-whitespace feature (fix on main, revert
on clang-13.x branch) suggested by @aaron.ballman in D104601.

Differential Revision: https://reviews.llvm.org/D107183

show more ...

[Preprocessor] -E -P: Ensure newline after 8 skipped lines.

The implementation of -fminimize-whitespace (D104601) revised the logic
when to emit newlines. There was no case to handle when more than

[Preprocessor] -E -P: Ensure newline after 8 skipped lines.

The implementation of -fminimize-whitespace (D104601) revised the logic
when to emit newlines. There was no case to handle when more than
8 lines were skippped in -P (DisableLineMarkers) mode and instead fell
through the case intended for -fminimize-whitespace, i.e. emit nothing.
This patch will emit one newline in this case.

The newline logic is slightly reorganized. The `-P -fminimize-whitespace`
case is handled explicitly and emitting at least one newline is the new
fallback case. The choice between emitting a line marker or up to
7 empty lines is now a choice only with enabled line markers. The up to
8 newlines likely are fewer characters than a line directive, but
in -P mode this had the paradoxic effect that it would print up to
7 empty lines, but none at all if more than 8 lines had to be skipped.
Now with DisableLineMarkers, we don't consider printing empty lines
(just start a new line) which matches gcc's behavior.

The line-directive-output-mincol.c test is replaced with a more
comprehensive test skip-empty-lines.c also testing the more than
8 skipped lines behaviour with all flag combinations.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D106924

show more ...

[Preprocessor] Implement -fminimize-whitespace.

This patch adds the -fminimize-whitespace with the following effects:

* If combined with -E, remove as much non-line-breaking whitespace as
possi

[Preprocessor] Implement -fminimize-whitespace.

This patch adds the -fminimize-whitespace with the following effects:

* If combined with -E, remove as much non-line-breaking whitespace as
possible.

* If combined with -E -P, removes as much whitespace as possible,
including line-breaks.

The motivation is to reduce the amount of insignificant changes in the
preprocessed output with source files where only whitespace has been
changed (add/remove comments, clang-format, etc.) which is in particular
useful with ccache.

A patch for ccache for using this flag has been proposed to ccache as well:
https://github.com/ccache/ccache/pull/815, which will use
-fnormalize-whitespace when clang-13 has been detected, and additionally
uses -P in "unify_mode". ccache already had a unify_mode in an older
version which was removed because of problems that using the
preprocessor itself does not have (such that the custom tokenizer did
not recognize C++11 raw strings).

This patch slightly reorganizes which part is responsible for adding
newlines that are required for semantics. It is now either
startNewLineIfNeeded() or MoveToLine() but never both; this avoids the
ShouldUpdateCurrentLine workaround and avoids redundant lines being
inserted in some cases. It also fixes a mandatory newline not inserted
after a _Pragma("...") that is expanded into a #pragma.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D104601

show more ...

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

[PragmaHandler] Expose `#pragma` location

Currently, a pragma AST node's recorded location starts at the
namespace token (such as `omp` in the case of OpenMP) after the
`#pragma` token, and the `#pr

[PragmaHandler] Expose `#pragma` location

Currently, a pragma AST node's recorded location starts at the
namespace token (such as `omp` in the case of OpenMP) after the
`#pragma` token, and the `#pragma` location isn't available. However,
the `#pragma` location can be useful when, for example, rewriting a
directive using Clang's Rewrite facility.

This patch makes `#pragma` locations available in any `PragmaHandler`
but it doesn't yet make use of them.

This patch also uses the new `struct PragmaIntroducer` to simplify
`Preprocessor::HandlePragmaDirective`. It doesn't do the same for
`PPCallbacks::PragmaDirective` because that changes the API documented
in `clang-tools-extra/docs/pp-trace.rst`, and I'm not sure about
backward compatibility guarantees there.

Reviewed By: ABataev, lebedev.ri, aaron.ballman

Differential Revision: https://reviews.llvm.org/D61643

llvm-svn: 361335

show more ...

[Lex] Allow to consume tokens while preprocessing

Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
exp

[Lex] Allow to consume tokens while preprocessing

Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
expanded tokens without re-runnig the preprocessor with minimal changes
to the preprocessor and minimal performance penalty when preprocessing
without recording the tokens.

The added hook is very low-level and reconstructing the expanded token
stream requires more work in the client code, the actual algorithm to
collect the tokens using this hook can be found in the follow-up change.

Reviewers: rsmith

Reviewed By: rsmith

Subscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59885

llvm-svn: 361007

show more ...

[C++20] Implement context-sensitive header-name lexing and pp-import parsing in the preprocessor.

llvm-svn: 358231

Add PragmaHandler for MSVC pragma execution_character_set

__pragma(execution_character_set(push, "UTF-8")) is used in
TraceLoggingProvider.h. This commit implements a no-op handler for
compatability

Add PragmaHandler for MSVC pragma execution_character_set

__pragma(execution_character_set(push, "UTF-8")) is used in
TraceLoggingProvider.h. This commit implements a no-op handler for
compatability, similar to how the flag -fexec_charset is handled.

Patch by Matt Gardner!

Differential Revision: https://reviews.llvm.org/D58530

llvm-svn: 356185

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Fix combining pragma __debug dump & parser_crash with -E

Previously these would be transformed into annotation tokens and the
preprocessor would then assume they were real tokens with source
locatio

Fix combining pragma __debug dump & parser_crash with -E

Previously these would be transformed into annotation tokens and the
preprocessor would then assume they were real tokens with source
locations and assert/UB.

Other pragmas that produce annotation tokens aren't a problem because
they aren't handled if the parser isn't hooked up - ParsePragma.cpp
registers those handlers & isn't run for pure preprocessing. So they're
treated as unknown pragmas & printed verbatim by the preprocessor.

Perhaps these pragmas should be treated the same way? But they got mixed
in with other __debug pragmas that do need to be handled during
preprocessing.

The third __debug pragma that produces an annotation token is 'captured'
- which had its own fix for this issue - by not inserting the annotation
token in the first place if it detected that it was in preprocessing
mode. I've removed that fix (from Lex/Pragma.cpp) in favor of the more
general one in Frontend/PrintPreprocessedOutput.cpp.

llvm-svn: 346928

show more ...

Remove trailing space

sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h}

llvm-svn: 338291

Reland '[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective'

This commit relands r331904.

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, an

Reland '[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective'

This commit relands r331904.

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are
system
headers.

Differential Revision: https://reviews.llvm.org/D46614

llvm-svn: 332021

show more ...

Revert "[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective"

This reverts commit r331904 because of a memory leak.

llvm-svn: 331932

[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This

[clang] Adding CharacteristicKind to PPCallbacks::InclusionDirective

Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are system
headers.

Differential Revision: https://reviews.llvm.org/D46614

llvm-svn: 331904

show more ...