Changes in PPLexerChange.cpp

0d3a2b4c - [Lex] Introduce `PPCallbacks::LexedFileChanged()` preprocessor callback

Thu, 30 Jun 2022 21:04:14 +0000

[Lex] Introduce `PPCallbacks::LexedFileChanged()` preprocessor callbackThis is a preprocessor callback focused on the lexed file changing, without conflating effects of line number directives and other pragmas.A client that only cares about what files the lexer processes, like dependency generation, can use this more straightforwardcallback instead of `PPCallbacks::FileChanged()`. Clients that want the pragma directive effects as well can keep using `FileChanged()`.A use case where `PPCallbacks::LexedFileChanged()` is particularly simpler to use than `FileChanged()` is in a situationwhere a client wants to keep track of lexed file changes that include changes from/to the predefines buffer, where it becomesunnecessary complicated trying to use `FileChanged()` while filtering out the pragma directives effects callbacks.Also take the opportunity to provide information about the prior `FileID` the `Lexer` moved from, even when entering a new file.Differential Revision: https://reviews.llvm.org/D128947 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

b4c83a13 - [Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

Thu, 12 May 2022 04:56:29 +0000

[Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sourcesThis is a commit with the following changes:* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionalityRemoves `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blockswill be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sourcesReplaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`This is bringing the following benefits: * Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources. * Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time. * Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).For normal preprocessing measurements show differences are below the noise level.Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.Differential Revision: https://reviews.llvm.org/D125486Differential Revision: https://reviews.llvm.org/D125487Differential Revision: https://reviews.llvm.org/D125488 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

e9a902c7 - Revert "Revert "Revert "[clang][pp] adds '#pragma include_instead'"""

Sat, 16 Apr 2022 00:22:43 +0000

Revert "Revert "Revert "[clang][pp] adds '#pragma include_instead'"""> Includes regression test for problem noted by @hans.> is reverts commit 973de71.>> Differential Revision: https://reviews.llvm.org/D106898Feature implemented as-is is fairly expensive and hasn't been used bylibc++. A potential reimplementation is possible if libc++ becomeinterested in this feature again.Differential Revision: https://reviews.llvm.org/D123885 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

d9739f29 - Serialize PragmaAssumeNonNullLoc to support preambles

Mon, 21 Mar 2022 17:45:21 +0000

Serialize PragmaAssumeNonNullLoc to support preamblesPreviously, if a `#pragma clang assume_nonnull begin` was at theend of a premable with a `#pragma clang assume_nonnull end` at theend of the main file, clang would diagnose an unterminated begin inthe preamble and an unbalanced end in the main file.With this change, those errors no longer occur and the case above isnow properly handled. I've added a corresponding test to clangd,which makes use of preambles, in order to verify this works asexpected.Differential Revision: https://reviews.llvm.org/D122179 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

7631c366 - [clang][lex] Introduce `ConstSearchDirIterator`

Tue, 15 Feb 2022 09:11:00 +0000

[clang][lex] Introduce `ConstSearchDirIterator`The `const DirectoryLookup *` out-parameter of `{HeaderSearch,Preprocessor}::LookupFile()` is assigned the most recently used search directory, which callers use to implement `#include_next`.From the function signature it's not obvious the `const DirectoryLookup *` is being used as an iterator. This patch introduces `ConstSearchDirIterator` to make that affordance obvious. This would've prevented a bug that occurred after initially landing D116750.Reviewed By: ahoppenDifferential Revision: https://reviews.llvm.org/D117566 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

197576c4 - [clang][lex] Refactor check for the first file include

Thu, 18 Nov 2021 11:34:54 +0000

[clang][lex] Refactor check for the first file includeThis patch refactors the code that checks whether a file has just been included for the first time.The `HeaderSearch::FirstTimeLexingFile` function is removed and the information is threaded to the original call site from `HeaderSearch::ShouldEnterIncludeFile`. This will make it possible to avoid tracking the number of includes in a follow up patch.Depends on D114092.Reviewed By: dexonsmithDifferential Revision: https://reviews.llvm.org/D114093 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

16ceb44e - [clang] Use llvm::{count,count_if,find_if,all_of,none_of} (NFC)

Mon, 25 Oct 2021 16:14:45 +0000

[clang] Use llvm::{count,count_if,find_if,all_of,none_of} (NFC) List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

08719541 - Revert "Revert "[clang][pp] adds '#pragma include_instead'""

Tue, 27 Jul 2021 15:36:03 +0000

Revert "Revert "[clang][pp] adds '#pragma include_instead'""Includes regression test for problem noted by @hans.This reverts commit 973de7185606a21fd5e9d5e8c014fbf898c0e72f.Differential Revision: https://reviews.llvm.org/D106898 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

973de718 - Revert "[clang][pp] adds '#pragma include_instead'"

Tue, 27 Jul 2021 15:22:01 +0000

Revert "[clang][pp] adds '#pragma include_instead'"> `#pragma clang include_instead(

)` is a pragma that can be used> by system headers (and only system headers) to indicate to a tool that> the file containing said pragma is an implementation-detail header and> should not be directly included by user code.>> The library alternative is very messy code that can be seen in the first> diff of D106124, and we'd rather avoid that with something more> universal.>> This patch takes the first step by warning a user when they include a> detail header in their code, and suggests alternative headers that the> user should include instead. Future work will involve adding a fixit to> automate the process, as well as cleaning up modules diagnostics to not> suggest said detail headers. Other tools, such as clangd can also take> advantage of this pragma to add the correct user headers.>> Differential Revision: https://reviews.llvm.org/D106394This caused compiler crashes in Chromium builds involving PCH and an includedirective with macro expansion, when Token::getLiteralData() returned null. Seethe code review for details.This reverts commit e8a64e5491260714c79dab65d1aa73245931d314. List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

e8a64e54 - [clang][pp] adds '#pragma include_instead'

Thu, 22 Jul 2021 20:30:43 +0000

[clang][pp] adds '#pragma include_instead'`#pragma clang include_instead(

)` is a pragma that can be usedby system headers (and only system headers) to indicate to a tool thatthe file containing said pragma is an implementation-detail header andshould not be directly included by user code.The library alternative is very messy code that can be seen in the firstdiff of D106124, and we'd rather avoid that with something moreuniversal.This patch takes the first step by warning a user when they include adetail header in their code, and suggests alternative headers that theuser should include instead. Future work will involve adding a fixit toautomate the process, as well as cleaning up modules diagnostics to notsuggest said detail headers. Other tools, such as clangd can also takeadvantage of this pragma to add the correct user headers.Differential Revision: https://reviews.llvm.org/D106394 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

b3eff6b7 - Lexer: Update the Lexer to use MemoryBufferRef, NFC

Wed, 14 Oct 2020 15:11:09 +0000

Lexer: Update the Lexer to use MemoryBufferRef, NFCUpdate `Lexer` / `Lexer::Lexer` to use `MemoryBufferRef` instead of`MemoryBuffer*`. Callers that were acquiring a `MemoryBuffer*` via`SourceManager::getBuffer` were updated, such that if they checked`Invalid` they use `getBufferOrNone` and otherwise `getBufferOrFake`.Differential Revision: https://reviews.llvm.org/D89398 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

ed79827a - [clang][module] Improve incomplete-umbrella warning

Tue, 16 Jun 2020 23:39:50 +0000

[clang][module] Improve incomplete-umbrella warningChange the warning message for -Wincomplete-umbrella to report the location of the umbrella header;Differential Revision: https://reviews.llvm.org/D82118 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

411a254a - [clang] Make sure argument expansion locations are correct in presence of predefined buffer

Wed, 22 Apr 2020 14:37:27 +0000

[clang] Make sure argument expansion locations are correct in presence of predefined bufferSummary:Macro argument expansion logic relies on skipping file IDs that createdas a result of an include. Unfortunately it fails to do that forpredefined buffer since it doesn't have a valid insertion location.As a result of that any file ID created for an include inside thepredefined buffers breaks the traversal logic inSourceManager::computeMacroArgsCache.To fix this issue we first record number of created FIDs for predefinedbuffer, and then skip them explicitly in source manager.Another solution would be to just give predefined buffers a valid sourcelocation, but it is unclear where that should be..Reviewers: sammccallSubscribers: cfe-commitsTags: #clangDifferential Revision: https://reviews.llvm.org/D78649 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

e08464fb - Avoid including FileManager.h from SourceManager.h

Sat, 29 Feb 2020 17:10:42 +0000

Avoid including FileManager.h from SourceManager.hMost clients of SourceManager.h need to do things like turning sourcelocations into file & line number pairs, but this doesn't requirebringing in FileManager.h and LLVM's FS headers.The main code change here is to sink SM::createFileID into the cpp file.I reason that this is not performance critical because it doesn't happenon the diagnostic path, it happens along the paths of macro expansion(could be hot) and new includes (less hot).Saves some includes: 309 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileManager.h 272 - /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileSystemOptions.h 271 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/VirtualFileSystem.h 267 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/FileSystem.h 266 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Chrono.hDifferential Revision: https://reviews.llvm.org/D75406 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

adcd0268 - Make llvm::StringRef to std::string conversions explicit.

Tue, 28 Jan 2020 19:23:46 +0000

Make llvm::StringRef to std::string conversions explicit.This is how it should've been and brings it more in line withstd::string_view. There should be no functional change here.This is mostly mechanical from a custom clang-tidy check, with a lot ofmanual fixups. It uncovers a lot of minor inefficiencies.This doesn't actually modify StringRef yet, I'll do that in a follow-up. List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

6a24e806 - [NFCI]Create CommonAttributeInfo Type as base type of *Attr and ParsedAttr.

Fri, 13 Sep 2019 17:39:31 +0000

[NFCI]Create CommonAttributeInfo Type as base type of *Attr and ParsedAttr.In order to enable future improvements to our attribute diagnostics,this moves info from ParsedAttr into CommonAttributeInfo, then makesthis type the base of the *Attr and ParsedAttr types. Quite a bit ofrefactoring took place, including removing a bunch of redundant SpellingIndex propogation.Differential Revision: https://reviews.llvm.org/D67368llvm-svn: 371875 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

2b3d49b6 - [Clang] Migrate llvm::make_unique to std::make_unique

Wed, 14 Aug 2019 23:04:18 +0000

[Clang] Migrate llvm::make_unique to std::make_uniqueNow that we've moved to C++14, we no longer need the llvm::make_uniqueimplementation from STLExtras.h. This patch is a mechanical replacementof (hopefully) all the llvm::make_unique instances across the monorepo.Differential revision: https://reviews.llvm.org/D66259llvm-svn: 368942 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

8d323d15 - [clang] Adopt new FileManager error-returning APIs

Thu, 01 Aug 2019 21:31:56 +0000

[clang] Adopt new FileManager error-returning APIsUpdate the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.Signed-off-by: Harlan Haskins llvm-svn: 367616 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

16de81c1 - [modules] Add PP callbacks for entering and leaving a submodule.

Thu, 04 Jul 2019 19:06:52 +0000

[modules] Add PP callbacks for entering and leaving a submodule.llvm-svn: 365153 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp

929af673 - [Lex] Allow to consume tokens while preprocessing

Fri, 17 May 2019 09:32:05 +0000

[Lex] Allow to consume tokens while preprocessingSummary:By adding a hook to consume all tokens produced by the preprocessor.The intention of this change is to make it possible to consume theexpanded tokens without re-runnig the preprocessor with minimal changesto the preprocessor and minimal performance penalty when preprocessingwithout recording the tokens.The added hook is very low-level and reconstructing the expanded tokenstream requires more work in the client code, the actual algorithm tocollect the tokens using this hook can be found in the follow-up change.Reviewers: rsmithReviewed By: rsmithSubscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commitsTags: #clangDifferential Revision: https://reviews.llvm.org/D59885llvm-svn: 361007 List of files: /llvm-project-15.0.7/clang/lib/Lex/PPLexerChange.cpp