[ThinLTO] Efficiency fix for writing type id records in per-module indexes

Summary:
In D49565/r337503, the type id record writing was fixed so that only
referenced type ids were emitted into each pe

[ThinLTO] Efficiency fix for writing type id records in per-module indexes

Summary:
In D49565/r337503, the type id record writing was fixed so that only
referenced type ids were emitted into each per-module index for ThinLTO
distributed builds. However, this still left an efficiency issue: each
per-module index checked all type ids for membership in the referenced
set, yielding O(M*N) performance (M indexes and N type ids).

Change the TypeIdMap in the summary to be indexed by GUID, to facilitate
correlating with type identifier GUIDs referenced in the function
summary TypeIdInfo structures. This allowed simplifying other
places where a map from type id GUID to type id map entry was previously
being used to aid this correlation.

Also fix AsmWriter code to handle the rare case of type id GUID
collision.

For a large internal application, this reduced the thin link time by
almost 15%.

Reviewers: pcc, vitalybuka

Subscribers: mehdi_amini, inglorion, steven_wu, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D51330

llvm-svn: 343021

show more ...

[IR] Add a boolean field in DILocation to know if a line must covered or not

Summary:
Some lines have a hit counter where they should not have one.
For example, in C++, some cleanup is adding at the

[IR] Add a boolean field in DILocation to know if a line must covered or not

Summary:
Some lines have a hit counter where they should not have one.
For example, in C++, some cleanup is adding at the end of a scope represented by a '}'.
So such a line has a hit counter where a user expects to not have one.
The goal of the patch is to add this information in DILocation which is used to get the covered lines in GCOVProfiling.cpp.
A following patch in clang will add this information when generating IR (https://reviews.llvm.org/D49916).

Reviewers: marco-c, davidxl, vsk, javed.absar, rnk

Reviewed By: rnk

Subscribers: eraman, xur, danielcdh, aprantl, rnk, dblaikie, #debug-info, vsk, llvm-commits, sylvestre.ledru

Tags: #debug-info

Differential Revision: https://reviews.llvm.org/D49915

llvm-svn: 342631

show more ...

[ThinLTO] Write TYPE_IDs for types used in functions imported by aliases

Summary:
ThinLTO imports alias as a copy of a aliasee, so when we import such functions with type tests we will
need type ids

[ThinLTO] Write TYPE_IDs for types used in functions imported by aliases

Summary:
ThinLTO imports alias as a copy of a aliasee, so when we import such functions with type tests we will
need type ids used by function. However after D49565 we pick types only during processing of
FunctionSummary which is not happening for such aliesees.

Example:
Unit U1 with a type, a functions F with the type check, and an alias A to the function.
Unit U2 with only call to the alias A.

In particular, this happens when we use -mconstructor-aliases, which is default.
So if c++ unit only creates instance of the class, without calling any other methods it will lack of
necessary type ids, which will result in false CFI reports.

Reviewers: tejohnson, eugenis

Subscribers: pcc, mehdi_amini, inglorion, eraman, hiraditya, steven_wu, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D52201

llvm-svn: 342574

show more ...

[ThinLTO] Extract getReferencedTypeIds from [NFC]

Summary: Refactoring before D52201

Reviewers: eugenis, tejohnson

Subscribers: pcc, inglorion, hiraditya, dexonsmith, llvm-commits

Differential Re

[ThinLTO] Extract getReferencedTypeIds from [NFC]

Summary: Refactoring before D52201

Reviewers: eugenis, tejohnson

Subscribers: pcc, inglorion, hiraditya, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D52203

llvm-svn: 342573

show more ...

[x86/SLH] Add a real Clang flag and LLVM IR attribute for Speculative
Load Hardening.

Wires up the existing pass to work with a proper IR attribute rather
than just a hidden/internal flag. The inter

[x86/SLH] Add a real Clang flag and LLVM IR attribute for Speculative
Load Hardening.

Wires up the existing pass to work with a proper IR attribute rather
than just a hidden/internal flag. The internal flag continues to work
for now, but I'll likely remove it soon.

Most of the churn here is adding the IR attribute. I talked about this
Kristof Beyls and he seemed at least initially OK with this direction.
The idea of using a full attribute here is that we *do* expect at least
some forms of this for other architectures. There isn't anything
*inherently* x86-specific about this technique, just that we only have
an implementation for x86 at the moment.

While we could potentially expose this as a Clang-level attribute as
well, that seems like a good question to defer for the moment as it
isn't 100% clear whether that or some other programmer interface (or
both?) would be best. We'll defer the programmer interface side of this
for now, but at least get to the point where the feature can be enabled
without relying on implementation details.

This also allows us to do something that was really hard before: we can
enable *just* the indirect call retpolines when using SLH. For x86, we
don't have any other way to mitigate indirect calls. Other architectures
may take a different approach of course, and none of this is surfaced to
user-level flags.

Differential Revision: https://reviews.llvm.org/D51157

llvm-svn: 341363

show more ...

Allow creating llvm::Function in non-zero address spaces

Most users won't have to worry about this as all of the
'getOrInsertFunction' functions on Module will default to the program
address space.

Allow creating llvm::Function in non-zero address spaces

Most users won't have to worry about this as all of the
'getOrInsertFunction' functions on Module will default to the program
address space.

An overload has been added to Function::Create to abstract away the
details for most callers.

This is based on https://reviews.llvm.org/D37054 but without the changes to
make passing a Module to Function::Create() mandatory. I have also added
some more tests and fixed the LLParser to accept call instructions for
types in the program address space.

Reviewed By: bjope

Differential Revision: https://reviews.llvm.org/D47541

llvm-svn: 340519

show more ...

DebugInfo: Add metadata support for disabling DWARF pub sections

In cases where the debugger load time is a worthwhile tradeoff (or less
costly - such as loading from a DWP instead of a variety of D

DebugInfo: Add metadata support for disabling DWARF pub sections

In cases where the debugger load time is a worthwhile tradeoff (or less
costly - such as loading from a DWP instead of a variety of DWOs
(possibly over a high-latency/distributed filesystem)) against object
file size, it can be reasonable to disable pubnames and corresponding
gdb-index creation in the linker.

A backend-flag version of this was implemented for NVPTX in
D44385/r327994 - which was fine for NVPTX which wouldn't mix-and-match
CUs. Now that it's going to be a user-facing option (likely powered by
"-gno-pubnames", the same as GCC) it should be encoded in the
DICompileUnit so it can vary per-CU.

After this, likely the NVPTX support should be migrated to the metadata
& the previous flag implementation should be removed.

Reviewers: aprantl

Differential Revision: https://reviews.llvm.org/D50213

llvm-svn: 339939

show more ...

[DebugInfoMetadata] Added DIFlags interface in DIBasicType.

Flags in DIBasicType will be used to pass attributes used in
DW_TAG_base_type, such as DW_AT_endianity.

Patch by Chirag Patel!

Different

[DebugInfoMetadata] Added DIFlags interface in DIBasicType.

Flags in DIBasicType will be used to pass attributes used in
DW_TAG_base_type, such as DW_AT_endianity.

Patch by Chirag Patel!

Differential Revision: https://reviews.llvm.org/D49610

llvm-svn: 339714

show more ...

Remove trailing space

sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h}

llvm-svn: 338293

[ThinLTO] Only emit referenced type id records in index files

Summary:
Currently all type ids are emitted into the index file when it is
written. For distributed ThinLTO, that meant that all type id

[ThinLTO] Only emit referenced type id records in index files

Summary:
Currently all type ids are emitted into the index file when it is
written. For distributed ThinLTO, that meant that all type ids were
being duplicated into every single distributed index file, regardless of
whether they were referenced, leading to huge amounts of unnecessary
duplication and size bloat.

Keep track of the type id GUIDs actually referenced by the GV summary
records being emitted, and only emit those type IDs.

Add a new test, and fix test/Assembler/thinlto-summary.ll so that all
type ids are referenced to prevent deletion in that test.

Reviewers: pcc

Subscribers: mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, vitalybuka, llvm-commits

Differential Revision: https://reviews.llvm.org/D49565

llvm-svn: 337503

show more ...

[ThinLTO] Parse module summary index from assembly

Summary:
Adds assembly parsing support for the module summary index (follow on
to r333335 which added the assembly writing support).

I added suppo

[ThinLTO] Parse module summary index from assembly

Summary:
Adds assembly parsing support for the module summary index (follow on
to r333335 which added the assembly writing support).

I added support to llvm-as to invoke the index parsing, so that it can
create either a bitcode file with a Module and a per-module index, or
a combined index without a Module.

I will send follow on patches soon to do the following:
- add support to tools such as llvm-lto2 to parse the per-module indexes
from assembly instead of bitcode when testing the thin link.
- verification support.

Depends on D47844 and D47842.

Reviewers: pcc, dexonsmith, mehdi_amini

Subscribers: inglorion, eraman, steven_wu, llvm-commits

Differential Revision: https://reviews.llvm.org/D47905

llvm-svn: 335602

show more ...

[ThinLTO] Compute GUID directly from GV when building per-module index

Summary:
I discovered when writing the summary parsing support that the
per-module index builder and writer are computing the G

[ThinLTO] Compute GUID directly from GV when building per-module index

Summary:
I discovered when writing the summary parsing support that the
per-module index builder and writer are computing the GUID from the
value name alone (ignoring the linkage type). This was ok since those
GUID were not emitted in the bitcode, and there are never multiple
conflicting names in a single module.

However, I don't see a reason for making the GUID computation different
for the per-module case. It also makes things simpler on the parsing
side to have the GUID computation consistent. So this patch changes the
summary analysis phase and the per-module summary writer to compute the
GUID using the facility on the GlobalValue.

Reviewers: pcc, dexonsmith

Subscribers: llvm-commits, inglorion

Differential Revision: https://reviews.llvm.org/D47844

llvm-svn: 335560

show more ...

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

In order to set breakpoints on labels and list source code around
labels, we need collect debug information for labels, i.e., label
nam

[DebugInfo] Add DILabel metadata and intrinsic llvm.dbg.label.

In order to set breakpoints on labels and list source code around
labels, we need collect debug information for labels, i.e., label
name, the function label belong, line number in the file, and the
address label located. In order to keep these information in LLVM
IR and to allow backend to generate debug information correctly.
We create a new kind of metadata for labels, DILabel. The format
of DILabel is

!DILabel(scope: !1, name: "foo", file: !2, line: 3)

We hope to keep debug information as much as possible even the
code is optimized. So, we create a new kind of intrinsic for label
metadata to avoid the metadata is eliminated with basic block.
The intrinsic will keep existing if we keep it from optimized out.
The format of the intrinsic is

llvm.dbg.label(metadata !1)

It has only one argument, that is the DILabel metadata. The
intrinsic will follow the label immediately. Backend could get the
label metadata through the intrinsic's parameter.

We also create DIBuilder API for labels to be used by Frontend.
Frontend could use createLabel() to allocate DILabel objects, and use
insertLabel() to insert llvm.dbg.label intrinsic in LLVM IR.

Differential Revision: https://reviews.llvm.org/D45024

Patch by Hsiangkai Wang.

llvm-svn: 331841

show more ...

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()

IWYU for llvm-config.h in llvm, additions.

See r331124 for how I made a list of files missing the include.
I then ran this Python script:

for f in open('filelist.txt'):
f = f.strip()
fl = open(f).readlines()

found = False
for i in xrange(len(fl)):
p = '#include "llvm/'
if not fl[i].startswith(p):
continue
if fl[i][len(p):] > 'Config':
fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
found = True
break
if not found:
print 'not found', f
else:
open(f, 'w').write(''.join(fl))

and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.

No intended behavior change.

llvm-svn: 331184

show more ...

[Bitcode] Change std::sort to llvm::sort in response to r327219

Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-det

[Bitcode] Change std::sort to llvm::sort in response to r327219

Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.

To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.

Note: This patch is one of a series of patches to replace *all* std::sort to llvm::sort.
Refer the comments section in D44363 for a list of all the required patches.

Reviewers: pcc, mehdi_amini, dexonsmith

Reviewed By: dexonsmith

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D45132

llvm-svn: 329334

show more ...

Add the ShadowCallStack attribute

Summary:
Introduce the ShadowCallStack function attribute. It's added to
functions compiled with -fsanitize=shadow-call-stack in order to mark
functions to be instr

Add the ShadowCallStack attribute

Summary:
Introduce the ShadowCallStack function attribute. It's added to
functions compiled with -fsanitize=shadow-call-stack in order to mark
functions to be instrumented by a ShadowCallStack pass to be submitted
in a separate change.

Reviewers: pcc, kcc, kubamracek

Reviewed By: pcc, kcc

Subscribers: cryptoad, mehdi_amini, javed.absar, llvm-commits, kcc

Differential Revision: https://reviews.llvm.org/D44800

llvm-svn: 329108

show more ...

[ThinLTO] Add an option to force summary call edges cold for debugging

Summary:
Useful to selectively disable importing into specific modules for
debugging/triaging/workarounds.

Reviewers: eraman

[ThinLTO] Add an option to force summary call edges cold for debugging

Summary:
Useful to selectively disable importing into specific modules for
debugging/triaging/workarounds.

Reviewers: eraman

Subscribers: inglorion, llvm-commits

Differential Revision: https://reviews.llvm.org/D45062

llvm-svn: 328909

show more ...

[SimplifyCFG] Create attribute for fuzzing-specific optimizations.

Summary:
When building with libFuzzer, converting control flow to selects or
obscuring the original operands of CMPs reduces the ef

[SimplifyCFG] Create attribute for fuzzing-specific optimizations.

Summary:
When building with libFuzzer, converting control flow to selects or
obscuring the original operands of CMPs reduces the effectiveness of
libFuzzer's heuristics.

This patch provides an attribute to disable or modify certain optimizations
for optimal fuzzing signal.

Provides a less aggressive alternative to https://reviews.llvm.org/D44057.

Reviewers: vitalybuka, davide, arsenm, hfinkel

Reviewed By: vitalybuka

Subscribers: junbuml, mehdi_amini, wdng, javed.absar, hiraditya, llvm-commits, kcc

Differential Revision: https://reviews.llvm.org/D44232

llvm-svn: 328214

show more ...

[X86] Added support for nocf_check attribute for indirect Branch Tracking

X86 Supports Indirect Branch Tracking (IBT) as part of Control-Flow Enforcement Technology (CET).
IBT instruments ENDBR inst

[X86] Added support for nocf_check attribute for indirect Branch Tracking

X86 Supports Indirect Branch Tracking (IBT) as part of Control-Flow Enforcement Technology (CET).
IBT instruments ENDBR instructions used to specify valid targets of indirect call / jmp.
The `nocf_check` attribute has two roles in the context of X86 IBT technology:
1. Appertains to a function - do not add ENDBR instruction at the beginning of the function.
2. Appertains to a function pointer - do not track the target function of this pointer by adding nocf_check prefix to the indirect-call instruction.

This patch implements `nocf_check` context for Indirect Branch Tracking.
It also auto generates `nocf_check` prefixes before indirect branchs to jump tables that are guarded by range checks.

Differential Revision: https://reviews.llvm.org/D41879

llvm-svn: 327767

show more ...

[DebugInfo] Support DWARF v5 source code embedding extension

In DWARF v5 the Line Number Program Header is extensible, allowing values with
new content types. In this extension a content type is add

[DebugInfo] Support DWARF v5 source code embedding extension

In DWARF v5 the Line Number Program Header is extensible, allowing values with
new content types. In this extension a content type is added,
DW_LNCT_LLVM_source, which contains the embedded source code of the file.

Add new optional attribute for !DIFile IR metadata called source which contains
source text. Use this to output the source to the DWARF line table of code
objects. Analogously extend METADATA_FILE in Bitcode and .file directive in ASM
to support optional source.

Teach llvm-dwarfdump and llvm-objdump about the new values. Update the output
format of llvm-dwarfdump to make room for the new attribute on file_names
entries, and support embedded sources for the -source option in llvm-objdump.

Differential Revision: https://reviews.llvm.org/D42765

llvm-svn: 325970

show more ...

bitcode support change for fast flags compatibility

Summary: The discussion and as per need, each vendor needs a way to keep the old fast flags and the new fast flags in the auto upgrade path of the

bitcode support change for fast flags compatibility

Summary: The discussion and as per need, each vendor needs a way to keep the old fast flags and the new fast flags in the auto upgrade path of the IR upgrader. This revision addresses that issue.

Patched by Michael Berg

Reviewers: qcolombet, hans, steven_wu

Reviewed By: qcolombet, steven_wu

Subscribers: dexonsmith, vsk, mehdi_amini, andrewrk, MatzeB, wristow, spatel

Differential Revision: https://reviews.llvm.org/D43253

llvm-svn: 325525

show more ...

[ThinLTO] Allow indexing to request backend to ignore the module

Summary:
Gold plugin does not add pass to ThinLTO modules without useful symbols.
In this case ThinLTO can't create corresponding ind

[ThinLTO] Allow indexing to request backend to ignore the module

Summary:
Gold plugin does not add pass to ThinLTO modules without useful symbols.
In this case ThinLTO can't create corresponding index file and some features, like CFI,
cannot be processes by backed correctly without index.
Given that we don't need the backed output we can request it to avoid
processing the module. This is implemented by this patch using new
"SkipModuleByDistributedBackend" flag.

Reviewers: pcc, tejohnson

Subscribers: mehdi_amini, inglorion, eraman, cfe-commits

Differential Revision: https://reviews.llvm.org/D42995

llvm-svn: 325411

show more ...

[ThinLTO/CFI] Include TYPE_ID summaries into GLOBALVAL_SUMMARY_BLOCK

Summary:
TypeID summaries are used by CFI and need to be serialized by ThinLTO
indexing for later use by LTO Backend.

Reviewers:

[ThinLTO/CFI] Include TYPE_ID summaries into GLOBALVAL_SUMMARY_BLOCK

Summary:
TypeID summaries are used by CFI and need to be serialized by ThinLTO
indexing for later use by LTO Backend.

Reviewers: tejohnson, pcc

Subscribers: mehdi_amini, inglorion, eraman, hiraditya, llvm-commits

Differential Revision: https://reviews.llvm.org/D42611

llvm-svn: 325182

show more ...

Pass a reference to a module to the bitcode writer.

This simplifies most callers as they are already using references or
std::unique_ptr.

llvm-svn: 325155

[DebugInfo] Unify ChecksumKind and Checksum value in DIFile

Rather than encode the absence of a checksum with a Kind variant, instead put
both the kind and value in a struct and wrap it in an Option

[DebugInfo] Unify ChecksumKind and Checksum value in DIFile

Rather than encode the absence of a checksum with a Kind variant, instead put
both the kind and value in a struct and wrap it in an Optional.

Differential Revision: http://reviews.llvm.org/D43043

llvm-svn: 324928

show more ...