[Lex] Introduce `PPCallbacks::LexedFileChanged()` preprocessor callback

This is a preprocessor callback focused on the lexed file changing, without conflating effects of line number directives and o

[Lex] Introduce `PPCallbacks::LexedFileChanged()` preprocessor callback

This is a preprocessor callback focused on the lexed file changing, without conflating effects of line number directives and other pragmas.
A client that only cares about what files the lexer processes, like dependency generation, can use this more straightforward
callback instead of `PPCallbacks::FileChanged()`. Clients that want the pragma directive effects as well can keep using `FileChanged()`.

A use case where `PPCallbacks::LexedFileChanged()` is particularly simpler to use than `FileChanged()` is in a situation
where a client wants to keep track of lexed file changes that include changes from/to the predefines buffer, where it becomes
unnecessary complicated trying to use `FileChanged()` while filtering out the pragma directives effects callbacks.

Also take the opportunity to provide information about the prior `FileID` the `Lexer` moved from, even when entering a new file.

Differential Revision: https://reviews.llvm.org/D128947

show more ...

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionality

Removes `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blocks
will be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.

* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

Replaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.

* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`

This is bringing the following benefits:

* Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
* Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time.
* Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).

For normal preprocessing measurements show differences are below the noise level.

Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.

Differential Revision: https://reviews.llvm.org/D125486
Differential Revision: https://reviews.llvm.org/D125487
Differential Revision: https://reviews.llvm.org/D125488

show more ...

Revert "Revert "Revert "[clang][pp] adds '#pragma include_instead'"""

> Includes regression test for problem noted by @hans.
> is reverts commit 973de71.
>
> Differential Revision: https://reviews.l

Revert "Revert "Revert "[clang][pp] adds '#pragma include_instead'"""

> Includes regression test for problem noted by @hans.
> is reverts commit 973de71.
>
> Differential Revision: https://reviews.llvm.org/D106898

Feature implemented as-is is fairly expensive and hasn't been used by
libc++. A potential reimplementation is possible if libc++ become
interested in this feature again.

Differential Revision: https://reviews.llvm.org/D123885

show more ...

Serialize PragmaAssumeNonNullLoc to support preambles

Previously, if a `#pragma clang assume_nonnull begin` was at the
end of a premable with a `#pragma clang assume_nonnull end` at the
end of the m

Serialize PragmaAssumeNonNullLoc to support preambles

Previously, if a `#pragma clang assume_nonnull begin` was at the
end of a premable with a `#pragma clang assume_nonnull end` at the
end of the main file, clang would diagnose an unterminated begin in
the preamble and an unbalanced end in the main file.

With this change, those errors no longer occur and the case above is
now properly handled. I've added a corresponding test to clangd,
which makes use of preambles, in order to verify this works as
expected.

Differential Revision: https://reviews.llvm.org/D122179

show more ...

[clang][lex] Introduce `ConstSearchDirIterator`

The `const DirectoryLookup *` out-parameter of `{HeaderSearch,Preprocessor}::LookupFile()` is assigned the most recently used search directory, which

[clang][lex] Introduce `ConstSearchDirIterator`

The `const DirectoryLookup *` out-parameter of `{HeaderSearch,Preprocessor}::LookupFile()` is assigned the most recently used search directory, which callers use to implement `#include_next`.

From the function signature it's not obvious the `const DirectoryLookup *` is being used as an iterator. This patch introduces `ConstSearchDirIterator` to make that affordance obvious. This would've prevented a bug that occurred after initially landing D116750.

Reviewed By: ahoppen

Differential Revision: https://reviews.llvm.org/D117566

show more ...

[clang][lex] Refactor check for the first file include

This patch refactors the code that checks whether a file has just been included for the first time.

The `HeaderSearch::FirstTimeLexingFile` fu

[clang][lex] Refactor check for the first file include

This patch refactors the code that checks whether a file has just been included for the first time.

The `HeaderSearch::FirstTimeLexingFile` function is removed and the information is threaded to the original call site from `HeaderSearch::ShouldEnterIncludeFile`. This will make it possible to avoid tracking the number of includes in a follow up patch.

Depends on D114092.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D114093

show more ...

[clang] Use llvm::{count,count_if,find_if,all_of,none_of} (NFC)

Revert "Revert "[clang][pp] adds '#pragma include_instead'""

Includes regression test for problem noted by @hans.
This reverts commit 973de7185606a21fd5e9d5e8c014fbf898c0e72f.

Differential Revision

Revert "Revert "[clang][pp] adds '#pragma include_instead'""

Includes regression test for problem noted by @hans.
This reverts commit 973de7185606a21fd5e9d5e8c014fbf898c0e72f.

Differential Revision: https://reviews.llvm.org/D106898

show more ...

Revert "[clang][pp] adds '#pragma include_instead'"

> `#pragma clang include_instead(<header>)` is a pragma that can be used
> by system headers (and only system headers) to indicate to a tool that

Revert "[clang][pp] adds '#pragma include_instead'"

> `#pragma clang include_instead(<header>)` is a pragma that can be used
> by system headers (and only system headers) to indicate to a tool that
> the file containing said pragma is an implementation-detail header and
> should not be directly included by user code.
>
> The library alternative is very messy code that can be seen in the first
> diff of D106124, and we'd rather avoid that with something more
> universal.
>
> This patch takes the first step by warning a user when they include a
> detail header in their code, and suggests alternative headers that the
> user should include instead. Future work will involve adding a fixit to
> automate the process, as well as cleaning up modules diagnostics to not
> suggest said detail headers. Other tools, such as clangd can also take
> advantage of this pragma to add the correct user headers.
>
> Differential Revision: https://reviews.llvm.org/D106394

This caused compiler crashes in Chromium builds involving PCH and an include
directive with macro expansion, when Token::getLiteralData() returned null. See
the code review for details.

This reverts commit e8a64e5491260714c79dab65d1aa73245931d314.

show more ...

[clang][pp] adds '#pragma include_instead'

`#pragma clang include_instead(<header>)` is a pragma that can be used
by system headers (and only system headers) to indicate to a tool that
the file cont

[clang][pp] adds '#pragma include_instead'

`#pragma clang include_instead(<header>)` is a pragma that can be used
by system headers (and only system headers) to indicate to a tool that
the file containing said pragma is an implementation-detail header and
should not be directly included by user code.

The library alternative is very messy code that can be seen in the first
diff of D106124, and we'd rather avoid that with something more
universal.

This patch takes the first step by warning a user when they include a
detail header in their code, and suggests alternative headers that the
user should include instead. Future work will involve adding a fixit to
automate the process, as well as cleaning up modules diagnostics to not
suggest said detail headers. Other tools, such as clangd can also take
advantage of this pragma to add the correct user headers.

Differential Revision: https://reviews.llvm.org/D106394

show more ...

Lexer: Update the Lexer to use MemoryBufferRef, NFC

Update `Lexer` / `Lexer::Lexer` to use `MemoryBufferRef` instead of
`MemoryBuffer*`. Callers that were acquiring a `MemoryBuffer*` via
`SourceMana

Lexer: Update the Lexer to use MemoryBufferRef, NFC

Update `Lexer` / `Lexer::Lexer` to use `MemoryBufferRef` instead of
`MemoryBuffer*`. Callers that were acquiring a `MemoryBuffer*` via
`SourceManager::getBuffer` were updated, such that if they checked
`Invalid` they use `getBufferOrNone` and otherwise `getBufferOrFake`.

Differential Revision: https://reviews.llvm.org/D89398

show more ...

[clang][module] Improve incomplete-umbrella warning

Change the warning message for -Wincomplete-umbrella to report the location of the umbrella header;

Differential Revision: https://reviews.llvm.o

[clang][module] Improve incomplete-umbrella warning

Change the warning message for -Wincomplete-umbrella to report the location of the umbrella header;

Differential Revision: https://reviews.llvm.org/D82118

show more ...

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include

[clang] Make sure argument expansion locations are correct in presence of predefined buffer

Summary:
Macro argument expansion logic relies on skipping file IDs that created
as a result of an include. Unfortunately it fails to do that for
predefined buffer since it doesn't have a valid insertion location.

As a result of that any file ID created for an include inside the
predefined buffers breaks the traversal logic in
SourceManager::computeMacroArgsCache.

To fix this issue we first record number of created FIDs for predefined
buffer, and then skip them explicitly in source manager.

Another solution would be to just give predefined buffers a valid source
location, but it is unclear where that should be..

Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78649

show more ...

Avoid including FileManager.h from SourceManager.h

Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in

Avoid including FileManager.h from SourceManager.h

Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in FileManager.h and LLVM's FS headers.

The main code change here is to sink SM::createFileID into the cpp file.
I reason that this is not performance critical because it doesn't happen
on the diagnostic path, it happens along the paths of macro expansion
(could be hot) and new includes (less hot).

Saves some includes:
309 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileManager.h
272 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileSystemOptions.h
271 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/VirtualFileSystem.h
267 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/FileSystem.h
266 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Chrono.h

Differential Revision: https://reviews.llvm.org/D75406

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[NFCI]Create CommonAttributeInfo Type as base type of *Attr and ParsedAttr.

In order to enable future improvements to our attribute diagnostics,
this moves info from ParsedAttr into CommonAttributeI

[NFCI]Create CommonAttributeInfo Type as base type of *Attr and ParsedAttr.

In order to enable future improvements to our attribute diagnostics,
this moves info from ParsedAttr into CommonAttributeInfo, then makes
this type the base of the *Attr and ParsedAttr types. Quite a bit of
refactoring took place, including removing a bunch of redundant Spelling
Index propogation.

Differential Revision: https://reviews.llvm.org/D67368

llvm-svn: 371875

show more ...

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
o

[Clang] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942

show more ...

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan

[clang] Adopt new FileManager error-returning APIs

Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.

Signed-off-by: Harlan Haskins <[email protected]>
llvm-svn: 367616

show more ...

[modules] Add PP callbacks for entering and leaving a submodule.

llvm-svn: 365153

[Lex] Allow to consume tokens while preprocessing

Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
exp

[Lex] Allow to consume tokens while preprocessing

Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
expanded tokens without re-runnig the preprocessor with minimal changes
to the preprocessor and minimal performance penalty when preprocessing
without recording the tokens.

The added hook is very low-level and reconstructing the expanded token
stream requires more work in the client code, the actual algorithm to
collect the tokens using this hook can be found in the follow-up change.

Reviewers: rsmith

Reviewed By: rsmith

Subscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59885

llvm-svn: 361007

show more ...

Basic: Return a reference from FileManager::getVirtualFileSystem, NFC

FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clea

Basic: Return a reference from FileManager::getVirtualFileSystem, NFC

FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clear by returning a
reference from its accessor.

https://reviews.llvm.org/D59388

llvm-svn: 357038

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

PTH-- Remove feature entirely-

When debugging a boost build with a modified
version of Clang, I discovered that the PTH implementation
stores TokenKind in 8 bits. However, we currently have 368
Toke

PTH-- Remove feature entirely-

When debugging a boost build with a modified
version of Clang, I discovered that the PTH implementation
stores TokenKind in 8 bits. However, we currently have 368
TokenKinds.

The result is that the value gets truncated and the wrong token
gets picked up when including PTH files. It seems that this will
go wrong every time someone uses a token that uses the 9th bit.

Upon asking on IRC, it was brought up that this was a highly
experimental features that was considered a failure. I discovered
via googling that BoostBuild (mostly Boost.Math) is the only user of
this
feature, using the CC1 flag directly. I believe that this can be
transferred over to normal PCH with minimal effort:
https://github.com/boostorg/build/issues/367

Based on advice on IRC and research showing that this is a nearly
completely unused feature, this patch removes it entirely.

Note: I considered leaving the build-flags in place and making them
emit an error/warning, however since I've basically identified and
warned the only user, it seemed better to just remove them.

Differential Revision: https://reviews.llvm.org/D54547

Change-Id: If32744275ef1f585357bd6c1c813d96973c4d8d9
llvm-svn: 348266

show more ...

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang

Lift VFS from clang to llvm (NFC)

This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.

RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html

Differential revision: https://reviews.llvm.org/D52783

llvm-svn: 344140

show more ...

[VFS] vfs::directory_iterator yields path and file type instead of full Status

Summary:
Most callers I can find are using only `getName()`. Type is used by the
recursive iterator.

Now we don't have

[VFS] vfs::directory_iterator yields path and file type instead of full Status

Summary:
Most callers I can find are using only `getName()`. Type is used by the
recursive iterator.

Now we don't have to call stat() on every listed file (on most platforms).
Exceptions are e.g. Solaris where readdir() doesn't include type information.
On those platforms we'll still stat() - see D51918.

The result is significantly faster (stat() can be slow).
My motivation: this may allow us to improve clang IO on large TUs with long
include search paths. Caching readdir() results may allow us to skip many stat()
and open() operations on nonexistent files.

Reviewers: bkramer

Subscribers: fedor.sergeev, cfe-commits

Differential Revision: https://reviews.llvm.org/D51921

llvm-svn: 342232

show more ...