ThinLTO: add early "dead-stripping" on the Index

Summary:
Using the linker-supplied list of "preserved" symbols, we can compute
the list of "dead" symbols, i.e. the one that are not reachable from
a

ThinLTO: add early "dead-stripping" on the Index

Summary:
Using the linker-supplied list of "preserved" symbols, we can compute
the list of "dead" symbols, i.e. the one that are not reachable from
a "preserved" symbol transitively on the reference graph.
Right now we are using this information to mark these functions as
non-eligible for import.

The impact is two folds:
- Reduction of compile time: we don't import these functions anywhere
or import the function these symbols are calling.
- The limited number of import/export leads to better internalization.

Patch originally by Mehdi Amini.

Reviewers: mehdi_amini, pcc

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D23488

llvm-svn: 291177

show more ...

[ThinLTO] Subsume all importing checks into a single flag

Summary:
This adds a new summary flag NotEligibleToImport that subsumes
several existing flags (NoRename, HasInlineAsmMaybeReferencingIntern

[ThinLTO] Subsume all importing checks into a single flag

Summary:
This adds a new summary flag NotEligibleToImport that subsumes
several existing flags (NoRename, HasInlineAsmMaybeReferencingInternal
and IsNotViableToInline). It also subsumes the checking of references
on the summary that was being done during the thin link by
eligibleForImport() for each candidate. It is much more efficient to
do that checking once during the per-module summary build and record
it in the summary.

Reviewers: mehdi_amini

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D28169

llvm-svn: 291108

show more ...

Clear the PendingTypeTests vector after moving from it.

This is to put the vector into a well defined state. Apparently the state of a
vector after being moved from is valid but unspecified. Found w

Clear the PendingTypeTests vector after moving from it.

This is to put the vector into a well defined state. Apparently the state of a
vector after being moved from is valid but unspecified. Found with clang-tidy.

llvm-svn: 290298

show more ...

IR: Function summary representation for type tests.

Each function summary has an attached list of type identifier GUIDs. The
idea is that during the regular LTO phase we would match these GUIDs to t

IR: Function summary representation for type tests.

Each function summary has an attached list of type identifier GUIDs. The
idea is that during the regular LTO phase we would match these GUIDs to type
identifiers defined by the regular LTO module and store the resolutions in
a top-level "type identifier summary" (which will be implemented separately).

Differential Revision: https://reviews.llvm.org/D27967

llvm-svn: 290280

show more ...

IR: Eliminate non-determinism in the module summary analysis.

Also make the summary ref and call graph vectors immutable. This means
a smaller API surface and fewer places to audit for non-determini

IR: Eliminate non-determinism in the module summary analysis.

Also make the summary ref and call graph vectors immutable. This means
a smaller API surface and fewer places to audit for non-determinism.

Differential Revision: https://reviews.llvm.org/D27875

llvm-svn: 290200

show more ...

[ThinLTO] Import composite types as declarations

Summary:
When reading the metadata bitcode, create a type declaration when
possible for composite types when we are importing. Doing this in
the bitc

[ThinLTO] Import composite types as declarations

Summary:
When reading the metadata bitcode, create a type declaration when
possible for composite types when we are importing. Doing this in
the bitcode reader saves memory. Also it works naturally in the case
when the type ODR map contains a definition for the same composite type
because it was used in the importing module (buildODRType will
automatically use the existing definition and not create a type
declaration).

For Chromium built with -g2, this reduces the aggregate size of the
generated native object files by 66% (from 31G to 10G). It reduced
the time through the ThinLTO link and backend phases by about 20% on
my machine.

Reviewers: mehdi_amini, dblaikie, aprantl

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D27775

llvm-svn: 289993

show more ...

Strip invalid TBAA when reading bitcode

This ensures backward compatibility on bitcode loading.

Differential Revision: https://reviews.llvm.org/D27839

llvm-svn: 289977

Replace APFloatBase static fltSemantics data members with getter functions

At least the plugin used by the LibreOffice build
(<https://wiki.documentfoundation.org/Development/Clang_plugins>) indirec

Replace APFloatBase static fltSemantics data members with getter functions

At least the plugin used by the LibreOffice build
(<https://wiki.documentfoundation.org/Development/Clang_plugins>) indirectly
uses those members (through inline functions in LLVM/Clang include files in turn
using them), but they are not exported by utils/extract_symbols.py on Windows,
and accessing data across DLL/EXE boundaries on Windows is generally
problematic.

Differential Revision: https://reviews.llvm.org/D26671

llvm-svn: 289647

show more ...

Refactor BitcodeReader: move Metadata and ValueId handling in their own class/file

Summary:
I'm planning on changing the way we load metadata to enable laziness.
I'm getting lost in this gigantic fi

Refactor BitcodeReader: move Metadata and ValueId handling in their own class/file

Summary:
I'm planning on changing the way we load metadata to enable laziness.
I'm getting lost in this gigantic files, and gigantic class that is the bitcode
reader. This is a first toward splitting it in a few coarse components that
are more easily understandable.

Reviewers: pcc, tejohnson

Subscribers: mgorny, llvm-commits, dexonsmith

Differential Revision: https://reviews.llvm.org/D27646

llvm-svn: 289461

show more ...

Remove IsMetadataMaterialized from BitcodeReader (NFC)

Summary: It does not seem useful.

Reviewers: pcc, dexonsmith

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D2766

Remove IsMetadataMaterialized from BitcodeReader (NFC)

Summary: It does not seem useful.

Reviewers: pcc, dexonsmith

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D27668

llvm-svn: 289457

show more ...

[DIExpression] Introduce a dedicated DW_OP_LLVM_fragment operation
so we can stop using DW_OP_bit_piece with the wrong semantics.

The entire back story can be found here:
http://lists.llvm.org/piper

[DIExpression] Introduce a dedicated DW_OP_LLVM_fragment operation
so we can stop using DW_OP_bit_piece with the wrong semantics.

The entire back story can be found here:
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20161114/405934.html

The gist is that in LLVM we've been misinterpreting DW_OP_bit_piece's
offset field to mean the offset into the source variable rather than
the offset into the location at the top the DWARF expression stack. In
order to be able to fix this in a subsequent patch, this patch
introduces a dedicated DW_OP_LLVM_fragment operation with the
semantics that we used to apply to DW_OP_bit_piece, which is what we
actually need while inside of LLVM. This patch is complete with a
bitcode upgrade for expressions using the old format. It does not yet
fix the DWARF backend to use DW_OP_bit_piece correctly.

Implementation note: We discussed several options for implementing
this, including reserving a dedicated field in DIExpression for the
fragment size and offset, but using an custom operator at the end of
the expression works just fine and is more efficient because we then
only pay for it when we need it.

Differential Revision: https://reviews.llvm.org/D27361
rdar://problem/29335809

llvm-svn: 288683

show more ...

Bitcode: The index used by ModuleSummaryIndexBitcodeReader is now required, so make it a reference. NFCI.

llvm-svn: 288318

Bitcode: Introduce BitcodeModule::{has,get}Summary().

These are equivalent to hasGlobalValueSummary() and getModuleSummaryIndex().

Differential Revision: https://reviews.llvm.org/D27242

llvm-svn:

Bitcode: Introduce BitcodeModule::{has,get}Summary().

These are equivalent to hasGlobalValueSummary() and getModuleSummaryIndex().

Differential Revision: https://reviews.llvm.org/D27242

llvm-svn: 288317

show more ...

Bitcode: Change expected layout of module blocks.

We now expect each module's identification block to appear immediately before
the module block. Any module block that appears without an identificat

Bitcode: Change expected layout of module blocks.

We now expect each module's identification block to appear immediately before
the module block. Any module block that appears without an identification block
immediately before it is interpreted as if it does not have a module block.

Also change the interpretation of VST and function offsets in bitcode.
The offset is always taken as relative to the start of the identification
(or module if not present) block, minus one word. This corresponds to the
historical interpretation of offsets, i.e. relative to the start of the file.

These changes allow for bitcode modules to be concatenated by copying bytes.

Differential Revision: https://reviews.llvm.org/D27184

llvm-svn: 288098

show more ...

Bitcode: Introduce initial multi-module reader API.

Implement getLazyBitcodeModule() and parseBitcodeFile() in terms of it.

Differential Revision: https://reviews.llvm.org/D26719

llvm-svn: 287156

[ThinLTO] Make inline assembly handling more efficient in summary

Summary:
The change in r285513 to prevent exporting of locals used in
inline asm added all locals in the llvm.used set to the refere

[ThinLTO] Make inline assembly handling more efficient in summary

Summary:
The change in r285513 to prevent exporting of locals used in
inline asm added all locals in the llvm.used set to the reference
set of functions containing inline asm. Since these locals were marked
NoRename, this automatically prevented importing of the function.

Unfortunately, this caused an explosion in the summary reference lists
in some cases. In my particular example, it happened for a large protocol
buffer generated C++ file, where many of the generated functions
contained an inline asm call. It was exacerbated when doing a ThinLTO
PGO instrumentation build, where the PGO instrumentation included
thousands of private __profd_* values that were added to llvm.used.

We really only need to include a single llvm.used local (NoRename) value
in the reference list of a function containing inline asm to block it
being imported. However, it seems cleaner to add a flag to the summary
that explicitly describes this situation, which is what this patch does.

Reviewers: mehdi_amini

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D26402

llvm-svn: 286840

show more ...

Bitcode: Change module reader functions to return an llvm::Expected.

Differential Revision: https://reviews.llvm.org/D26562

llvm-svn: 286752

Bitcode: More precise casting. NFCI.

llvm-svn: 286750

Bitcode: Change getModuleSummaryIndex() to return an llvm::Expected.

Differential Revision: https://reviews.llvm.org/D26539

llvm-svn: 286624

Bitcode: Clean up error handling for certain bitcode query functions.

The functions getBitcodeTargetTriple(), isBitcodeContainingObjCCategory(),
getBitcodeProducerString() and hasGlobalValueSummary(

Bitcode: Clean up error handling for certain bitcode query functions.

The functions getBitcodeTargetTriple(), isBitcodeContainingObjCCategory(),
getBitcodeProducerString() and hasGlobalValueSummary() now return errors
via their return value rather than via the diagnostic handler.

To make this work, re-implement these functions using non-member functions
so that they can be used without the LLVMContext required by BitcodeReader.

Differential Revision: https://reviews.llvm.org/D26532

llvm-svn: 286623

show more ...

Bitcode: Prepare to move bitcode readers to free functions.

Make initStream() a free function, and change BitcodeReaderBase ctor to take
a BitstreamCursor.

llvm-svn: 286622

Split Bitcode/ReaderWriter.h into separate reader and writer headers

Summary:
Split ReaderWriter.h which contains the APIs into both the BitReader and
BitWriter libraries into BitcodeReader.h and Bi

Split Bitcode/ReaderWriter.h into separate reader and writer headers

Summary:
Split ReaderWriter.h which contains the APIs into both the BitReader and
BitWriter libraries into BitcodeReader.h and BitcodeWriter.h.

This is to address Chandler's concern about sharing the same API header
between multiple libraries (BitReader and BitWriter). That concern is
why we create a single bitcode library in our downstream build of clang,
which led to r286297 being reverted as it added a dependency that
created a cycle only when there is a single bitcode library (not two as
in upstream).

Reviewers: mehdi_amini

Subscribers: dlj, mehdi_amini, llvm-commits

Differential Revision: https://reviews.llvm.org/D26502

llvm-svn: 286566

show more ...

Prevent at compile time converting from Error::success() to Expected<T>

This would trigger an assertion at runtime otherwise.

Differential Revision: https://reviews.llvm.org/D26482

llvm-svn: 286562

IR: Introduce inrange attribute on getelementptr indices.

If the inrange keyword is present before any index, loading from or
storing to any pointer derived from the getelementptr has undefined
beha

IR: Introduce inrange attribute on getelementptr indices.

If the inrange keyword is present before any index, loading from or
storing to any pointer derived from the getelementptr has undefined
behavior if the load or store would access memory outside of the bounds of
the element selected by the index marked as inrange.

This can be used, e.g. for alias analysis or to split globals at element
boundaries where beneficial.

As previously proposed on llvm-dev:
http://lists.llvm.org/pipermail/llvm-dev/2016-July/102472.html

Differential Revision: https://reviews.llvm.org/D22793

llvm-svn: 286514

show more ...

Make BitcodeReader::parseIdentificationBlock() robust to EOF

This method is particular: it iterates at the top-level and does
not have an enclosing block.

llvm-svn: 286394