Check module signature when the module has already been loaded

We may need to verify the signature on subsequent imports as well, just
like we verify the size/modtime:
@import A;
@import B; // impor

Check module signature when the module has already been loaded

We may need to verify the signature on subsequent imports as well, just
like we verify the size/modtime:
@import A;
@import B; // imports A
@import C; // imports A

llvm-svn: 221569

show more ...

Make VFS and FileManager match the current MemoryBuffer API.

This eliminates converting back and forth between the 3 formats and
gives us a more homogeneous interface.

llvm-svn: 220657

[Modules] Free modules that failed signature verification.

The control flow and ownership is weird enough so unique_ptr doesn't help here :(

llvm-svn: 220569

Add a "signature" to AST files to verify that they haven't changed

Since the order of the IDs in the AST file (e.g. DeclIDs, SelectorIDs)
is not stable, it is not safe to load an AST file that depen

Add a "signature" to AST files to verify that they haven't changed

Since the order of the IDs in the AST file (e.g. DeclIDs, SelectorIDs)
is not stable, it is not safe to load an AST file that depends on
another AST file that has been rebuilt since the importer was built,
even if "nothing changed". We previously used size and modtime to check
this, but I've seen cases where a module rebuilt quickly enough to foil
this check and caused very hard to debug build errors.

To save cycles when we're loading the AST, we just generate a random
nonce value and check that it hasn't changed when we load an imported
module, rather than actually hash the whole file.

This is slightly complicated by the fact that we need to verify the
signature inside addModule, since we might otherwise consider that a
mdoule is "OutOfDate" when really it is the importer that is out of
date. I didn't see any regressions in module load time after this
change.

llvm-svn: 220493

show more ...

[modules] Initial support for explicitly loading .pcm files.

Implicit module builds are not well-suited to a lot of build systems. In
particular, they fare badly in distributed build systems, and th

[modules] Initial support for explicitly loading .pcm files.

Implicit module builds are not well-suited to a lot of build systems. In
particular, they fare badly in distributed build systems, and they lead to
build artifacts that are not tracked as part of the usual dependency management
process. This change allows explicitly-built module files (which are already
supported through the -emit-module flag) to be explicitly loaded into a build,
allowing build systems to opt to manage module builds and dependencies
themselves.

This is only the first step in supporting such configurations, and it should
be considered experimental and subject to change or removal for now.

llvm-svn: 220359

show more ...

Return a std::unique_ptr from getBufferForFile. NFC.

llvm-svn: 216476

Store std::unique_ptr in InMemoryBuffers. NFC.

llvm-svn: 215928

Repace SmallPtrSet with SmallPtrSetImpl in function arguments to avoid needing to mention the size.

llvm-svn: 215869

Update for llvm api change.

llvm-svn: 212408

Avoid invalidating successfully loaded module files

Successfully loaded module files may be referenced in other
ModuleManagers, so don't invalidate them. Two related things are fixed:

1) I thought

Avoid invalidating successfully loaded module files

Successfully loaded module files may be referenced in other
ModuleManagers, so don't invalidate them. Two related things are fixed:

1) I thought the last module in the manager was always the one that
failed, but it isn't. So check explicitly against the list of
vetted modules from ReadASTCore.

2) We now keep the file descriptor of pcm file open, which avoids the
possibility of having two different pcms for the same module loaded when
building in parallel with headers being modified during a build.

<rdar://problem/16835846>

llvm-svn: 211330

show more ...

Include system_error directly.

llvm-svn: 210802

Replace llvm::error_code with std::error_code.

llvm-svn: 210780

Invalidate the file system cache entries for files that may rebuild

This reapplies r209910 with a fix for the assertion failures hit on the
buildbots.

original commit message:
I thought we could ge

Invalidate the file system cache entries for files that may rebuild

This reapplies r209910 with a fix for the assertion failures hit on the
buildbots.

original commit message:
I thought we could get away without this, but it means that the
FileEntry objects actually refer to the wrong files, since pcms are not
updated inplace, they are atomically renamed into place after compiling
a module.

So we are close to the original behaviour of invalidating the cache for
all modules being removed, but now we should only invalidate the ones
that depend on whichever module failed to load.

Unfortunately I haven't come up with a new test that didn't require
a race between parallel invocations of clang.

<rdar://problem/17038180>

llvm-svn: 209922

show more ...

Revert "Invalidate the file system cache entries for files that may rebuild"

This reverts commit r209910, which is breaking some of the bots.

llvm-svn: 209911

Invalidate the file system cache entries for files that may rebuild

I thought we could get away without this, but it means that the
FileEntry objects actually refer to the wrong files, since pcms ar

Invalidate the file system cache entries for files that may rebuild

I thought we could get away without this, but it means that the
FileEntry objects actually refer to the wrong files, since pcms are not
updated inplace, they are atomically renamed into place after compiling
a module.

So we are close to the original behaviour of invalidating the cache for
all modules being removed, but now we should only invalidate the ones
that depend on whichever module failed to load.

Unfortunately I haven't come up with a new test that didn't require
a race between parallel invocations of clang.

<rdar://problem/17038180>

llvm-svn: 209910

show more ...

[C++11] Use 'nullptr'. Serialization edition.

llvm-svn: 209392

Speculative fix for Windows buildbot after r209138

It appears that Windows doesn't like renaming over open files, which we
do in clearOutputFiles. The file being compiled should be safe to
removed,

Speculative fix for Windows buildbot after r209138

It appears that Windows doesn't like renaming over open files, which we
do in clearOutputFiles. The file being compiled should be safe to
removed, but this isn't very satisfying - we don't want to manually
manage the lifetime of files we cannot prove have no references.

llvm-svn: 209195

show more ...

Don't refresh stat() info for pcm files

Follow-up fix for 209138. Actually, since we already have this file
open, we don't want to refresh the stat() info, since that might be
newer than what we ha

Don't refresh stat() info for pcm files

Follow-up fix for 209138. Actually, since we already have this file
open, we don't want to refresh the stat() info, since that might be
newer than what we have open (bad!).

llvm-svn: 209143

show more ...

Fix use-after-free and spurious error during module load

FileManager::invalidateCache is not safe to call when there may be
existing references to the file. What module load failure needs is
to refr

Fix use-after-free and spurious error during module load

FileManager::invalidateCache is not safe to call when there may be
existing references to the file. What module load failure needs is
to refresh so stale stat() info isn't stored.

This may be the last user of invalidateCache; I'll take a look and
remove it if possible in a future commit.

This caused a use-after-free error as well as a spurious error message
that a module was "found in both 'X.pcm' and 'X.pcm'" in some cases.

llvm-svn: 209138

show more ...

Fix a use-after-free bug I recently introduced in lookupModuleFile

llvm-svn: 207932

Avoid a potential race between stat() and open() of ASTFile

We need to open an ASTFile while checking its expected size and
modification time, or another clang instance can modify the file between
t

Avoid a potential race between stat() and open() of ASTFile

We need to open an ASTFile while checking its expected size and
modification time, or another clang instance can modify the file between
the stat() and the open().

llvm-svn: 207735

show more ...

Allow multiple modules with the same name to coexist in the module cache

To differentiate between two modules with the same name, we will
consider the path the module map file that they are defined

Allow multiple modules with the same name to coexist in the module cache

To differentiate between two modules with the same name, we will
consider the path the module map file that they are defined by* part of
the ‘key’ for looking up the precompiled module (pcm file).
Specifically, this patch renames the precompiled module (pcm) files from
cache-path/<module hash>/Foo.pcm
to
cache-path/<module hash>/Foo-<hash of module map path>.pcm

In addition, I’ve taught the ASTReader to re-resolve the names of
imported modules during module loading so that if the header search
context changes between when a module was originally built and when it
is loaded we can rebuild it if necessary. For example, if module A
imports module B

first time:
clang -I /path/to/A -I /path/to/B ...

second time:
clang -I /path/to/A -I /different/path/to/B ...

will now rebuild A as expected.

* in the case of inferred modules, we use the module map file that
allowed the inference, not the __inferred_module.map file, since the
inferred file path is the same for every inferred module.

llvm-svn: 206201

show more ...

[C++11] Remove a now unnecessary use of std::function for a remove_if
predicate. The wrapper used by SetVector was erroneously requiring an
adaptable predicate. It has been fixed and we really don't

[C++11] Remove a now unnecessary use of std::function for a remove_if
predicate. The wrapper used by SetVector was erroneously requiring an
adaptable predicate. It has been fixed and we really don't want to
require an indirect call for every predicate evaluation.

llvm-svn: 202744

show more ...

[C++11] Replace verbose functors with succinct lambdas

No functionality change.

llvm-svn: 202590

Recommit virtual file system

Previously reverted in r201755 due to causing an assertion failure.

I've removed the offending assertion, and taught the CompilerInstance to
create a default virtual fi

Recommit virtual file system

Previously reverted in r201755 due to causing an assertion failure.

I've removed the offending assertion, and taught the CompilerInstance to
create a default virtual file system inside createFileManager. In the
future, we should be able to reach into the CompilerInvocation to
customize this behaviour without breaking clients that don't care.

llvm-svn: 201818

show more ...