[clang-repl] Implement code undo.

In interactive C++ it is convenient to roll back to a previous state of the
compiler. For example:
clang-repl> int x = 42;
clang-repl> %undo
clang-repl> float x = 2

[clang-repl] Implement code undo.

In interactive C++ it is convenient to roll back to a previous state of the
compiler. For example:
clang-repl> int x = 42;
clang-repl> %undo
clang-repl> float x = 24 // not an error

To support this, the patch extends the functionality used to recover from
errors and adds functionality to recover the low-level execution infrastructure.

The current implementation is based on watermarks. It exploits the fact that
at each incremental input the underlying compiler infrastructure is in a valid
state. We can only go N incremental inputs back to a previous valid state. We do
not need and do not do any further dependency tracking.

This patch was co-developed with V. Vassilev, relies on the past work of Purva
Chaudhari in clang-repl and is inspired by the past work on the same feature
in the Cling interpreter.

Co-authored-by: Purva-Chaudhari <[email protected]>
Co-authored-by: Vassil Vassilev <[email protected]>
Signed-off-by: Jun Zhang <[email protected]>

show more ...

Implement soft reset of the diagnostics engine.

This patch implements soft reset and adds tests for soft reset success of the
diagnostics engine. This allows us to recover from errors in clang-repl

Implement soft reset of the diagnostics engine.

This patch implements soft reset and adds tests for soft reset success of the
diagnostics engine. This allows us to recover from errors in clang-repl without
resetting the pragma handlers' state.

Differential revision: https://reviews.llvm.org/D126183

show more ...

Reland "[clang-repl] Recover the lookup tables of the primary context."

The asan issue was fixed in llvm/llvm-project@7bc00ce5cd41

This reverts commit 575e297fcb289f0a9b0ac4b01d1d0fa051f5cc29.

Dif

Reland "[clang-repl] Recover the lookup tables of the primary context."

The asan issue was fixed in llvm/llvm-project@7bc00ce5cd41

This reverts commit 575e297fcb289f0a9b0ac4b01d1d0fa051f5cc29.

Differential revision: https://reviews.llvm.org/D123674

show more ...

[clang-repl] Remove memory leak of ASTContext/TargetMachine.

Removes memory leak of ASTContext and TargetMachine. When DisableFree is turned on, it intentionally leaks these instances as they can be

[clang-repl] Remove memory leak of ASTContext/TargetMachine.

Removes memory leak of ASTContext and TargetMachine. When DisableFree is turned on, it intentionally leaks these instances as they can be trivially deallocated. This patch turns this off and delete Parser instance early so that they will not reference dangling pargma headers.

Asan shouldn't detect these as leaks normally, since burypointer is called for them. But, every invocation of incremental parser createa an additional leak of TargetMachine. If there are many invocations within a single test case, we easily reach number of leaks exceeding kGraveYardMaxSize (which is 12) and leaks start to get reported by asan buildbots.

Reviewed By: v.g.vassilev

Differential Revision: https://reviews.llvm.org/D127991

show more ...

Revert "[clang-repl] Recover the lookup tables of the primary context."

This reverts commit 5ff27fe1ff03d5aeaf8567c97618170f0cef8f58.

This patch caused failures in asan: https://lab.llvm.org/buildb

Revert "[clang-repl] Recover the lookup tables of the primary context."

This reverts commit 5ff27fe1ff03d5aeaf8567c97618170f0cef8f58.

This patch caused failures in asan: https://lab.llvm.org/buildbot/#/builders/5/builds/24221

show more ...

[clang-repl] Recover the lookup tables of the primary context.

Before this patch, there was re-declaration error if error was encountered in
the same line. The recovery support acted only if this ty

[clang-repl] Recover the lookup tables of the primary context.

Before this patch, there was re-declaration error if error was encountered in
the same line. The recovery support acted only if this type of error was
encountered in the first line of the program and not in subsequent lines.

For example:

```
clang-repl> int i=9;
clang-repl> int j=9; err;
input_line_3:1:5: error: redefinition of 'j'
int j = 9;
```

Differential revision: https://reviews.llvm.org/D123674

show more ...

[C++20][Modules][1/8] Track valid import state.

In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the

[C++20][Modules][1/8] Track valid import state.

In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.

We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).

Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:

import module A;

int some_decl();

import module B;

where being in a module purview is insufficient to identify them.

Differential Revision: https://reviews.llvm.org/D118893

show more ...

Revert "[C++20][Modules][1/8] Track valid import state."

This reverts commit 8a3f9a584ad43369cf6a034dc875ebfca76d9033.

need to investigate build failures that do not show on CI or local
testing.

[C++20][Modules][1/8] Track valid import state.

In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the

[C++20][Modules][1/8] Track valid import state.

In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.

We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).

Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:

import module A;

int some_decl();

import module B;

where being in a module purview is insufficient to identify them.

Differential Revision: https://reviews.llvm.org/D118893

show more ...

[clang] Use nullptr instead of 0 or NULL (NFC)

Identified with modernize-use-nullptr.

[clang-repl] Allow Interpreter::getSymbolAddress to take a mangled name.

Reland "[clang-repl] Allow loading of plugins in clang-repl."

Differential revision: https://reviews.llvm.org/D110484

Revert "[clang-repl] Allow loading of plugins in clang-repl."

This reverts commit 81fb640f83b6a5d099f9124739ab3049be79ea56 due to bot failures:
https://lab.llvm.org/buildbot#builders/57/builds/10807

[clang-repl] Allow loading of plugins in clang-repl.

Differential revision: https://reviews.llvm.org/D110484

Reland "[clang-repl] Implement partial translation units and error recovery."

Original commit message:

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org

Reland "[clang-repl] Implement partial translation units and error recovery."

Original commit message:

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:

Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.

The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.

The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.

Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.

It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.

We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.

We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.

This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:

```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```

Differential revision: https://reviews.llvm.org/D104918

show more ...

Revert "[clang-repl] Implement partial translation units and error recovery."

This reverts commit 6775fc6ffa3ca1c36b20c25fa4e7f48f81213cf2.

It also reverts "[lldb] Fix compilation by adjusting to t

Revert "[clang-repl] Implement partial translation units and error recovery."

This reverts commit 6775fc6ffa3ca1c36b20c25fa4e7f48f81213cf2.

It also reverts "[lldb] Fix compilation by adjusting to the new ASTContext signature."

This reverts commit 03a3f86071c10a1f6cbbf7375aa6fe9d94168972.

We see some failures on the lldb infrastructure, these changes might play a role
in it. Let's revert it now and see if the bots will become green.

Ref: https://reviews.llvm.org/D104918

show more ...

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the

[clang-repl] Implement partial translation units and error recovery.

https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:

Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.

The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.

The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.

Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.

It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.

We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.

We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.

This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:

```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```

Differential revision: https://reviews.llvm.org/D104918

show more ...

[clang-repl] Better match the underlying architecture.

In cases where -fno-integrated-as is specified we should overwrite the
EmitAssembly action as well.

We also should rely on the target triple f

[clang-repl] Better match the underlying architecture.

In cases where -fno-integrated-as is specified we should overwrite the
EmitAssembly action as well.

We also should rely on the target triple from the process at least until we
implement out-of-process execution.

This patch should improve clang-repl on AIX.

Discussion available at: https://reviews.llvm.org/D96033

Differential revision: https://reviews.llvm.org/D102688

show more ...

[clang-repl] Recommit "Land initial infrastructure for incremental parsing"

Original commit message:

In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have
mentioned our plan

[clang-repl] Recommit "Land initial infrastructure for incremental parsing"

Original commit message:

In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have
mentioned our plans to make some of the incremental compilation facilities
available in llvm mainline.

This patch proposes a minimal version of a repl, clang-repl, which enables
interpreter-like interaction for C++. For instance:

./bin/clang-repl
clang-repl> int i = 42;
clang-repl> extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=42
clang-repl> quit

The patch allows very limited functionality, for example, it crashes on invalid
C++. The design of the proposed patch follows closely the design of cling. The
idea is to gather feedback and gradually evolve both clang-repl and cling to
what the community agrees upon.

The IncrementalParser class is responsible for driving the clang parser and
codegen and allows the compiler infrastructure to process more than one input.
Every input adds to the “ever-growing” translation unit. That model is enabled
by an IncrementalAction which prevents teardown when HandleTranslationUnit.

The IncrementalExecutor class hides some of the underlying implementation
details of the concrete JIT infrastructure. It exposes the minimal set of
functionality required by our incremental compiler/interpreter.

The Transaction class keeps track of the AST and the LLVM IR for each
incremental input. That tracking information will be later used to implement
error recovery.

The Interpreter class orchestrates the IncrementalParser and the
IncrementalExecutor to model interpreter-like behavior. It provides the public
API which can be used (in future) when using the interpreter library.

Differential revision: https://reviews.llvm.org/D96033

show more ...

[clang-repl] Land initial infrastructure for incremental parsing

In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have
mentioned our plans to make some of the incremental compila

[clang-repl] Land initial infrastructure for incremental parsing

In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have
mentioned our plans to make some of the incremental compilation facilities
available in llvm mainline.

This patch proposes a minimal version of a repl, clang-repl, which enables
interpreter-like interaction for C++. For instance:

./bin/clang-repl
clang-repl> int i = 42;
clang-repl> extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=42
clang-repl> quit

The patch allows very limited functionality, for example, it crashes on invalid
C++. The design of the proposed patch follows closely the design of cling. The
idea is to gather feedback and gradually evolve both clang-repl and cling to
what the community agrees upon.

The IncrementalParser class is responsible for driving the clang parser and
codegen and allows the compiler infrastructure to process more than one input.
Every input adds to the “ever-growing” translation unit. That model is enabled
by an IncrementalAction which prevents teardown when HandleTranslationUnit.

The IncrementalExecutor class hides some of the underlying implementation
details of the concrete JIT infrastructure. It exposes the minimal set of
functionality required by our incremental compiler/interpreter.

The Transaction class keeps track of the AST and the LLVM IR for each
incremental input. That tracking information will be later used to implement
error recovery.

The Interpreter class orchestrates the IncrementalParser and the
IncrementalExecutor to model interpreter-like behavior. It provides the public
API which can be used (in future) when using the interpreter library.

Differential revision: https://reviews.llvm.org/D96033

show more ...