[llvm] Call *set::insert without checking membership first (NFC)

[ORC] Check for errors when materializing absolute symbols.

This code previously used cantFail, but both steps (resolution and emission)
can fail if the resource tracker associated with the
Absolute

[ORC] Check for errors when materializing absolute symbols.

This code previously used cantFail, but both steps (resolution and emission)
can fail if the resource tracker associated with the
AbsoluteSymbolsMaterializationUnit is removed. Checking these errors is
necessary for correct error propagation.

show more ...

[ORC] Allow FailedToMaterialize errors to outlive ExecutionSessions.

Idiomatic llvm::Error usage can result in a FailedToMaterialize error tearing
down an ExecutionSession instance. Since the Failed

[ORC] Allow FailedToMaterialize errors to outlive ExecutionSessions.

Idiomatic llvm::Error usage can result in a FailedToMaterialize error tearing
down an ExecutionSession instance. Since the FailedToMaterialize error holds
SymbolStringPtrs and JITDylib references this leads to crashes when accessing
or logging the error.

This patch modifies FailedToMaterialize to retain the SymbolStringPool and
JITDylibs involved in the failure so that we can safely report an error message
to the client, even if the error tears down the session.

The contract for JITDylibs allows the getName method to be used even after the
session has been torn down, but no other JITDylib fields should be accessed via
the FailedToMaterialize error if the ssesion has been torn down. Logging the
error is guaranteed to be safe in all cases.

show more ...

[ORC] Add a ~ExectionSession destructor to verify that endSession was called.

Clients are required to call ExecutionSession::endSession before destroying the
ExecutionSession. Failure to do so can l

[ORC] Add a ~ExectionSession destructor to verify that endSession was called.

Clients are required to call ExecutionSession::endSession before destroying the
ExecutionSession. Failure to do so can lead to memory leaks and other difficult
to debug issues. Enforcing this requirement by assertion makes it easy to spot
or debug situations where the contract was not followed.

show more ...

[ORC] Add a FIXME.

[ORC] Add missing std::moves, pass SymbolLookupSet by value.

Avoids some unnecessary SymbolStringPtr copies.

[ORC][ORC-RT] Rewrite the MachO platform to use allocation actions.

This patch updates the MachO platform (both the ORC MachOPlatform class and the
ORC-Runtime macho_platform.* files) to use allocat

[ORC][ORC-RT] Rewrite the MachO platform to use allocation actions.

This patch updates the MachO platform (both the ORC MachOPlatform class and the
ORC-Runtime macho_platform.* files) to use allocation actions, rather than EPC
calls, to transfer the initializer information scraped from each linked object.
Interactions between the ORC and ORC-Runtime sides of the platform are
substantially redesigned to accomodate the change.

The high-level changes in this patch are:

1. The MachOPlatform::setupJITDylib method now calls into the runtime to set up
a dylib name <-> header mapping, and a dylib state object (JITDylibState).

2. The MachOPlatformPlugin builds an allocation action that calls the
__orc_rt_macho_register_object_platform_sections and
__orc_rt_macho_deregister_object_platform_sections functions in the runtime
to register the address ranges for all "interesting" sections in the object
being allocated (TLS data sections, initializers, language runtime metadata
sections, etc.).

3. The MachOPlatform::rt_getInitializers method (the entry point in the
controller for requests from the runtime for initializer information) is
replaced by MachOPlatform::rt_pushInitializers. The former returned a data
structure containing the "interesting" section address ranges, but these are
now handled by __orc_rt_macho_register_object_platform_sections. The new
rt_pushInitializers method first issues a lookup to trigger materialization
of the "interesting" sections, then returns the dylib dependence tree rooted
at the requested dylib for dlopen to consume. (The dylib dependence tree is
returned by rt_pushInitializers, rather than being handled by some dedicated
call, because rt_pushInitializers can alter the dependence tree).

The advantage of these changes (beyond the performance advantages of using
allocation actions) is that it moves more information about the materialized
portions of the JITDylib into the executor. This tends to make the runtime
easier to reason about, e.g. the implementation of dlopen in the runtime is now
recursive, rather than relying on recursive calls in the controller to build a
linear data structure for consumption by the runtime. This change can also make
some operations more efficient, e.g. JITDylibs can be dlclosed and then
re-dlopened without having to pull all initializers over from the controller
again.

In addition to the high-level changes, there are some low-level changes to ORC
and the runtime:

* In ORC, at ExecutionSession teardown time JITDylibs are now destroyed in
reverse creation order. This is on the assumption that the ORC runtime will be
loaded into an earlier dylib that will be used by later JITDylibs. This is a
short-term solution to crashes that arose during testing when the runtime was
torn down before its users. Longer term we will likely destroy dylibs in
dependence order.

* toSPSSerializable(Expected<T> E) is updated to explicitly initialize the T
value, allowing it to be used by Ts that have explicit constructors.

* The ORC runtime now (1) attempts to track ref-counts, and (2) distinguishes
not-yet-processed "interesting" sections from previously processed ones. (1)
is necessary for standard dlopen/dlclose emulation. (2) is intended as a step
towards better REPL support -- it should enable future runtime calls that
run only newly registered initializers ("dlopen_more", "dlopen_additions",
...?).

show more ...

[llvm] Use = default (NFC)

[ORC] Fix JITDylib debug output: don't output symbol table entry flags twice.

[ORC] Allow JITDylib::getDFSLinkOrder and friends to fail for defunct JITDylibs.

Calls to JITDylib's getDFSLinkOrder and getReverseDFSLinkOrder methods (both
static an non-static versions) are now v

[ORC] Allow JITDylib::getDFSLinkOrder and friends to fail for defunct JITDylibs.

Calls to JITDylib's getDFSLinkOrder and getReverseDFSLinkOrder methods (both
static an non-static versions) are now valid to make on defunct JITDylibs, but
will return an error if any JITDylib in the link order is defunct.

This means that platforms can safely lookup link orders by name in response to
jit-dlopen calls from the ORC runtime, even if the call names a defunct
JITDylib -- the call will just fail with an error.

show more ...

[ORC] Add Platform::teardownJITDylib method.

This is a counterpart to Platform::setupJITDylib, and is called when JITDylib
instances are removed (via ExecutionSession::removeJITDylib).

Upcoming Mac

[ORC] Add Platform::teardownJITDylib method.

This is a counterpart to Platform::setupJITDylib, and is called when JITDylib
instances are removed (via ExecutionSession::removeJITDylib).

Upcoming MachOPlatform patches will use this to clear per-JITDylib data when
JITDylibs are removed.

show more ...

[llvm] Use true/false instead of 1/0 (NFC)

Identified with modernize-use-bool-literals.

[ORC] Add early-out to OL_applyQueryPhase1.

If all symbols in a lookup match before we reach the end of the search order
then bail out of the search-order loop early.

This should reduce unnecessary

[ORC] Add early-out to OL_applyQueryPhase1.

If all symbols in a lookup match before we reach the end of the search order
then bail out of the search-order loop early.

This should reduce unnecessary contention on the session lock and improve
readability of the debug logs.

show more ...

[ORC] Add a MaterializationUnit::Interface struct.

MaterializationUnit::Interface holds the values that make up the interface
(for ORC's purposes) of a materialization unit: the symbol flags map and

[ORC] Add a MaterializationUnit::Interface struct.

MaterializationUnit::Interface holds the values that make up the interface
(for ORC's purposes) of a materialization unit: the symbol flags map and
initializer symbol.

Having a type for this will make functions that build materializer interfaces
more readable and maintainable.

show more ...

[llvm] Fix "unused variable" warnings

[ORC] Add support for removing JITDylibs.

This allows JITDylibs to be removed from the ExecutionSession. Calling
ExecutionSession::removeJITDylib will disconnect the JITDylib from the
ExecutionSessi

[ORC] Add support for removing JITDylibs.

This allows JITDylibs to be removed from the ExecutionSession. Calling
ExecutionSession::removeJITDylib will disconnect the JITDylib from the
ExecutionSession and clear it (removing all trackers associated with it). The
JITDylib object will then be destroyed as soon as the last JITDylibSP pointing
at it is destroyed.

show more ...

[ORC] Only use JITDylib::GeneratorsMutex while running generators.

GeneratorsMutex should prevent lookups from proceeding through the
generators of a single JITDylib concurrently (since this could
r

[ORC] Only use JITDylib::GeneratorsMutex while running generators.

GeneratorsMutex should prevent lookups from proceeding through the
generators of a single JITDylib concurrently (since this could
result in redundant attempts to generate definitions). Mutation of
the generators list itself should be done under the session lock.

show more ...

[ORC] Hold ResourceTracker in MaterializationResponsibility.

This keeps the tracker alive for the lifetime of the MR. This is needed so that
we can check whether the tracker has become defunct befor

[ORC] Hold ResourceTracker in MaterializationResponsibility.

This keeps the tracker alive for the lifetime of the MR. This is needed so that
we can check whether the tracker has become defunct before posting results (or
failure) for the MR.

show more ...

[ORC][JITLink] Move JITDylib name into JITLinkDylib base class.

This will enable better error messages and debug logs in JITLink.

[ORC] Add optional RunPolicy to ExecutorProcessControl::callWrapperAsync.

The callWrapperAsync and callSPSWrapperAsync methods take a handler object
that is run on the return value of the call when

[ORC] Add optional RunPolicy to ExecutorProcessControl::callWrapperAsync.

The callWrapperAsync and callSPSWrapperAsync methods take a handler object
that is run on the return value of the call when it is ready. The new RunPolicy
parameters allow clients to control how these handlers are run. If no policy is
specified then the handler will be packaged as a GenericNamedTask and dispatched
using the ExecutorProcessControl's TaskDispatch member. Callers can use the
ExecutorProcessControl::RunInPlace policy to cause the handler to be run
directly instead, which may be preferrable for simple handlers, or they can
write their own policy object (e.g. to dispatch as some other kind of Task,
rather than GenericNamedTask).

show more ...

[ORC] Add TaskDispatch API and thread it through ExecutorProcessControl.

ExecutorProcessControl objects will now have a TaskDispatcher member which
should be used to dispatch work (in particular, ha

[ORC] Add TaskDispatch API and thread it through ExecutorProcessControl.

ExecutorProcessControl objects will now have a TaskDispatcher member which
should be used to dispatch work (in particular, handling incoming packets in
the implementation of remote EPC implementations like SimpleRemoteEPC).

The GenericNamedTask template can be used to wrap function objects that are
callable as 'void()' (along with an optional name to describe the task).
The makeGenericNamedTask functions can be used to create GenericNamedTask
instances without having to name the function object type.

In a future patch ExecutionSession will be updated to use the
ExecutorProcessControl's dispatcher, instead of its DispatchTaskFunction.

show more ...

[ORC] Hold shared_ptr<SymbolStringPool> in errors containing SymbolStringPtrs.

This allows these error values to remain valid, even if they tear down the JIT
itself.

[ORC] Fix missing newline in debugging output.

[ORC] Fix typo in debugging output

[ORC] Require ExecutorProcessControl when constructing an ExecutionSession.

Wrapper function call and dispatch handler helpers are moved to
ExecutionSession, and existing EPC-based tools are re-writ

[ORC] Require ExecutorProcessControl when constructing an ExecutionSession.

Wrapper function call and dispatch handler helpers are moved to
ExecutionSession, and existing EPC-based tools are re-written to take an
ExecutionSession argument instead.

Requiring an ExecutorProcessControl instance simplifies existing EPC based
utilities (which only need to take an ES now), and should encourage more
utilities to use the EPC interface. It also simplifies process termination,
since the session can automatically call ExecutorProcessControl::disconnect
(previously this had to be done manually, and carefully ordered with the
rest of JIT tear-down to work correctly).

show more ...