ThinLTO: fix another non-determinism in bitcode writing

GlobalVars Refs are initialized from a DenseSet. We can sort them
using the value id to recover some determinism during serialization.

From:

ThinLTO: fix another non-determinism in bitcode writing

GlobalVars Refs are initialized from a DenseSet. We can sort them
using the value id to recover some determinism during serialization.

From: Mehdi Amini <[email protected]>
llvm-svn: 269635

show more ...

ThinLTO: fix non-determinism in bitcode writing

Calls are initialized from a DenseMap. We can sort them using the
value id to recover some determinism during serialization.

From: Mehdi Amini <mehdi

ThinLTO: fix non-determinism in bitcode writing

Calls are initialized from a DenseMap. We can sort them using the
value id to recover some determinism during serialization.

From: Mehdi Amini <[email protected]>
llvm-svn: 269634

show more ...

ThinLTO: fix non-determinism in bitcode writing

Refs are initialized from a DenseSet. We can sort them using the
value id to recover some determinism during serialization.

From: Mehdi Amini <mehdi.

ThinLTO: fix non-determinism in bitcode writing

Refs are initialized from a DenseSet. We can sort them using the
value id to recover some determinism during serialization.

From: Mehdi Amini <[email protected]>
llvm-svn: 269629

show more ...

Move helper classes into anonymous namespaces. NFC.

llvm-svn: 269591

[ThinLTO] Fix Windows debug failure in new iterator

This fixes a debug assert on Windows from the new iterator
implementation added in r269059. The Windows std::vector iterator
operator== checks in

[ThinLTO] Fix Windows debug failure in new iterator

This fixes a debug assert on Windows from the new iterator
implementation added in r269059. The Windows std::vector iterator
operator== checks in debug mode that the containers being iterated over
are the same, which they may not be.

Fixed by checking that we are iterating over the same container before
comparing the container iterators.

llvm-svn: 269232

show more ...

Restore "[ThinLTO] Emit individual index files for distributed backends"

This restores commit r268627:
Summary:
When launching ThinLTO backends in a distributed build (currently
supporte

Restore "[ThinLTO] Emit individual index files for distributed backends"

This restores commit r268627:
Summary:
When launching ThinLTO backends in a distributed build (currently
supported in gold via the thinlto-index-only plugin option), emit
an individual index file for each backend process as described here:
http://lists.llvm.org/pipermail/llvm-dev/2016-April/098272.html

...

Differential Revision: http://reviews.llvm.org/D19556

Address msan failures by avoiding std::prev on map.end(), the
theory is that this is causing issues due to some known UB problems
in __tree.

llvm-svn: 269059

show more ...

Implement a safer bitcode upgrade for DISubprogram.

The bitcode upgrade I added for DISubprogram in r266446 was based on the
assumption that the CU node for the subprogram was already materialized b

Implement a safer bitcode upgrade for DISubprogram.

The bitcode upgrade I added for DISubprogram in r266446 was based on the
assumption that the CU node for the subprogram was already materialized by the
time the DISubprogram is visited. This assumption may not hold true as future
versions of LLVM may decide to write out bitcode in a different order. This
patch corrects this by introducing a versioning bit next to the distinct flag to
unambiguously differentiate the new from the old record layouts.

Note for people stabilizing LLVM out-of-tree: This patch introduces a bitcode
incompatibility with llvm trunk revisions from r266446 — this commit. (But
D19987 will ensure that it degrades gracefully).

http://reviews.llvm.org/D20004
rdar://problem/26074194

llvm-svn: 268816

show more ...

BitcodeWriter: Simplify. NFC.

llvm-svn: 268707

Revert "[ThinLTO] Emit individual index files for distributed backends"

MemorySanitizer: use-of-uninitialized-value in lib/Bitcode/Writer/BitcodeWriter.cpp:364:70
http://lab.llvm.org:8011/builders/s

Revert "[ThinLTO] Emit individual index files for distributed backends"

MemorySanitizer: use-of-uninitialized-value in lib/Bitcode/Writer/BitcodeWriter.cpp:364:70
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/12544/steps/check-llvm%20msan/logs/stdio

This reverts commit 0c4a898ea550699d1b2f4fe3767251c8f9a48d52.

llvm-svn: 268660

show more ...

[ThinLTO] Emit individual index files for distributed backends

Summary:
When launching ThinLTO backends in a distributed build (currently
supported in gold via the thinlto-index-only plugin option),

[ThinLTO] Emit individual index files for distributed backends

Summary:
When launching ThinLTO backends in a distributed build (currently
supported in gold via the thinlto-index-only plugin option), emit
an individual index file for each backend process as described here:
http://lists.llvm.org/pipermail/llvm-dev/2016-April/098272.html

The individual index file encodes the summary and module information
required for implementing the importing/exporting decisions made
for a given module in the thin link step.
This is in place of the current mechanism that uses the combined index
to make importing decisions in each back end independently. It is an
enabler for doing global summary based optimizations in the thin link
step (which will be recorded in the individual index files), and reduces
the size of the index that must be sent to each backend process, and
the amount of work to scan it in the backends.

Rather than create entirely new ModuleSummaryIndex structures (and all
the included unique_ptrs) for each backend index file, a map is created
to record all of the GUID and summary pointers needed for a particular
index file. The IndexBitcodeWriter walks this map instead of the full
index (hiding the details of managing the appropriate summary iteration
in a new iterator subclass). This is more efficient than walking the
entire combined index and filtering out just the needed summaries during
each backend bitcode index write.

Depends on D19481.

Reviewers: joker.eph

Subscribers: llvm-commits, joker.eph

Differential Revision: http://reviews.llvm.org/D19556

llvm-svn: 268627

show more ...

Clean up to avoid compiler warnings for casting away const qualifiers.

Differential Revision: http://reviews.llvm.org/D19598

llvm-svn: 267753

[ThinLTO] Use valueid instead of bitcode offsets in combined index file

Summary:
With the removal of support for lazy parsing of combined index summary
records (e.g. r267344), we no longer need to i

[ThinLTO] Use valueid instead of bitcode offsets in combined index file

Summary:
With the removal of support for lazy parsing of combined index summary
records (e.g. r267344), we no longer need to include the summary record
bitcode offset in the VST entries for definitions. Change the combined
index format to be similar to the per-module index format in using value
ids to cross-reference from the summary record to the VST entry (rather
than the summary record bitcode offset to cross-reference in the other
direction).

The visible changes are:
1) Add the value id to the combined summary records
2) Remove the summary offset from the combined VST records, which has
the following effects:
- No longer need the VST_CODE_COMBINED_GVDEFENTRY record, as all
combined index VST entries now only contain the value id and
corresponding GUID.
- No longer have duplicate VST entries in the case where there are
multiple definitions of a symbol (e.g. weak/linkonce), as they all
have the same value id and GUID.

An implication of #2 above is that in order to hook up an alias to the
correct aliasee based on the value id of the aliasee recorded in the
combined index alias record, we need to scan the entries in the index
for that GUID to find the one from the same module (i.e. the case where
there are multiple entries for the aliasee). But the reader no longer
has to maintain a special map to hook up the alias/aliasee.

Reviewers: joker.eph

Subscribers: joker.eph, llvm-commits

Differential Revision: http://reviews.llvm.org/D19481

llvm-svn: 267712

show more ...

[ThinLTO] Remove GlobalValueInfo class from index

Summary:
Remove the GlobalValueInfo and change the ModuleSummaryIndex to directly
reference summary objects. The info structure was there to support

[ThinLTO] Remove GlobalValueInfo class from index

Summary:
Remove the GlobalValueInfo and change the ModuleSummaryIndex to directly
reference summary objects. The info structure was there to support lazy
parsing of the combined index summary objects, which is no longer
needed and not supported.

Reviewers: joker.eph

Subscribers: joker.eph, llvm-commits

Differential Revision: http://reviews.llvm.org/D19462

llvm-svn: 267344

show more ...

Silence two C4806 warnings ('|': unsafe operation: no value of type 'bool' promoted to type 'const unsigned int' can equal the given constant). The fact that they trigger with this code seems like it

Silence two C4806 warnings ('|': unsafe operation: no value of type 'bool' promoted to type 'const unsigned int' can equal the given constant). The fact that they trigger with this code seems like it may be a bug, but the warning itself is still generally useful enough to retain it for now.

llvm-svn: 267337

show more ...

Add "hasSection" flag in the Summary

Reviewers: tejohnson

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D19405

From: Mehdi Amini <[email protected]>
llvm-svn: 267329

Reorganize GlobalValueSummary with a "Flags" bitfield.

Right now it only contains the LinkageType, but will be extended
with "hasSection", "isOptSize", "hasInlineAssembly", etc.

Differential Revisi

Reorganize GlobalValueSummary with a "Flags" bitfield.

Right now it only contains the LinkageType, but will be extended
with "hasSection", "isOptSize", "hasInlineAssembly", etc.

Differential Revision: http://reviews.llvm.org/D19404

From: Mehdi Amini <[email protected]>
llvm-svn: 267319

show more ...

Add a version field in the bitcode for the summary

Differential Revision: http://reviews.llvm.org/D19456

From: Mehdi Amini <[email protected]>
llvm-svn: 267318

Store and emit original name in combined index

Summary:
As discussed in D18298, some local globals can't
be renamed/promoted (because they have a section, or because
they are referenced from inline

Store and emit original name in combined index

Summary:
As discussed in D18298, some local globals can't
be renamed/promoted (because they have a section, or because
they are referenced from inline assembly).
To be able to detect naming collision, we need to keep around
the "GUID" using their original name without taking the linkage
into account.

Reviewers: tejohnson

Subscribers: joker.eph, llvm-commits

Differential Revision: http://reviews.llvm.org/D19454

From: Mehdi Amini <[email protected]>
llvm-svn: 267304

show more ...

DebugInfo: Remove MDString-based type references

Eliminate DITypeIdentifierMap and make DITypeRef a thin wrapper around
DIType*. It is no longer legal to refer to a DICompositeType by its
'identifi

DebugInfo: Remove MDString-based type references

Eliminate DITypeIdentifierMap and make DITypeRef a thin wrapper around
DIType*. It is no longer legal to refer to a DICompositeType by its
'identifier:', and DIBuilder no longer retains all types with an
'identifier:' automatically.

Aside from the bitcode upgrade, this is mainly removing logic to resolve
an MDString-based reference to an actualy DIType. The commits leading
up to this have made the implicit type map in DICompileUnit's
'retainedTypes:' field superfluous.

This does not remove DITypeRef, DIScopeRef, DINodeRef, and
DITypeRefArray, or stop using them in DI-related metadata. Although as
of this commit they aren't serving a useful purpose, there are patchces
under review to reuse them for CodeView support.

The tests in LLVM were updated with deref-typerefs.sh, which is attached
to the thread "[RFC] Lazy-loading of debug info metadata":

http://lists.llvm.org/pipermail/llvm-dev/2016-April/098318.html

llvm-svn: 267296

show more ...

Address comments.

llvm-svn: 267274

Refactor bitcode writer into classes (NFC)

Summary:
As discussed in on the mailing list yesterday, I have refactored
BitcodeWriter.cpp to use classes to manage the bitcode writing process,
instead o

Refactor bitcode writer into classes (NFC)

Summary:
As discussed in on the mailing list yesterday, I have refactored
BitcodeWriter.cpp to use classes to manage the bitcode writing process,
instead of passing around long lists of parameters between static
functions. See:
http://lists.llvm.org/pipermail/llvm-dev/2016-April/098610.html

I created a parent BitcodeWriter class to own the BitstreamWriter,
write the header, and contain the main entry point into the writing
process. There are two derived classes, one for writing a module and one
for writing a combined index file (for ThinLTO), which manage the
writing process specific to those bitcode file types.

I also changed the functions to conform to LLVM coding standards
(lowercase function name first letter). The only two routines that still
start with an uppercase letter are the two external interfaces, which
can be fixed as a follow-on (I wanted to keep this round just within
BitcodeWriter.cpp).

Reviewers: dexonsmith, joker.eph

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D19447

llvm-svn: 267273

show more ...

[ThinLTO] Prevent importing of "llvm.used" values

Summary:
This patch prevents importing from (and therefore exporting from) any
module with a "llvm.used" local value. Local values need to be promot

[ThinLTO] Prevent importing of "llvm.used" values

Summary:
This patch prevents importing from (and therefore exporting from) any
module with a "llvm.used" local value. Local values need to be promoted
and renamed when importing, and their presense on the llvm.used variable
indicates that there are opaque uses that won't see the rename. One such
example is a use in inline assembly.

See also the discussion at:
http://lists.llvm.org/pipermail/llvm-dev/2016-April/098047.html

As part of this, move collectUsedGlobalVariables out of Transforms/Utils
and into IR/Module so that it can be used more widely. There are several
other places in LLVM that used copies of this code that can be cleaned
up as a follow on NFC patch.

Reviewers: joker.eph

Subscribers: pcc, llvm-commits, joker.eph

Differential Revision: http://reviews.llvm.org/D18986

llvm-svn: 266877

show more ...

[NFC] Header cleanup

Removed some unused headers, replaced some headers with forward class declarations.

Found using simple scripts like this one:
clear && ack --cpp -l '#include "llvm/ADT/IndexedM

[NFC] Header cleanup

Removed some unused headers, replaced some headers with forward class declarations.

Found using simple scripts like this one:
clear && ack --cpp -l '#include "llvm/ADT/IndexedMap.h"' | xargs grep -L 'IndexedMap[<]' | xargs grep -n --color=auto 'IndexedMap'

Patch by Eugene Kosov <[email protected]>

Differential Revision: http://reviews.llvm.org/D19219

From: Mehdi Amini <[email protected]>
llvm-svn: 266595

show more ...

ThinLTO: Make aliases explicit in the summary

To be able to work accurately on the reference graph when taking
decision about internalizing, promoting, renaming, etc. We need
to have the alias infor

ThinLTO: Make aliases explicit in the summary

To be able to work accurately on the reference graph when taking
decision about internalizing, promoting, renaming, etc. We need
to have the alias information explicit.

Differential Revision: http://reviews.llvm.org/D18836

From: Mehdi Amini <[email protected]>
llvm-svn: 266517

show more ...

[PR27284] Reverse the ownership between DICompileUnit and DISubprogram.

Currently each Function points to a DISubprogram and DISubprogram has a
scope field. For member functions the scope is a DICom

[PR27284] Reverse the ownership between DICompileUnit and DISubprogram.

Currently each Function points to a DISubprogram and DISubprogram has a
scope field. For member functions the scope is a DICompositeType. DIScopes
point to the DICompileUnit to facilitate type uniquing.

Distinct DISubprograms (with isDefinition: true) are not part of the type
hierarchy and cannot be uniqued. This change removes the subprograms
list from DICompileUnit and instead adds a pointer to the owning compile
unit to distinct DISubprograms. This would make it easy for ThinLTO to
strip unneeded DISubprograms and their transitively referenced debug info.

Motivation
----------

Materializing DISubprograms is currently the most expensive operation when
doing a ThinLTO build of clang.

We want the DISubprogram to be stored in a separate Bitcode block (or the
same block as the function body) so we can avoid having to expensively
deserialize all DISubprograms together with the global metadata. If a
function has been inlined into another subprogram we need to store a
reference the block containing the inlined subprogram.

Attached to https://llvm.org/bugs/show_bug.cgi?id=27284 is a python script
that updates LLVM IR testcases to the new format.

http://reviews.llvm.org/D19034
<rdar://problem/25256815>

llvm-svn: 266446

show more ...