Re-re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper...

...with more fixes.

The original patch was reverted in 3e9cc543f22 due to bot failures caused by
a missing dependence on lib

Re-re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper...

...with more fixes.

The original patch was reverted in 3e9cc543f22 due to bot failures caused by
a missing dependence on librt. That issue was fixed in 32d8d23cd0, but that
commit also broke sanitizer bots due to a bug in SimplePackedSerialization:
empty ArrayRef<char>s triggered a zero-byte memcpy from a null source. The
ArrayRef<char> serialization issue was fixed in 67220c2ad7, and this patch has
also been updated with a new custom SharedMemorySegFinalizeRequest message that
should avoid serializing empty ArrayRefs in the first place.

https://reviews.llvm.org/D128544

show more ...

Revert "Re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper, with fixes."

This reverts commit 32d8d23cd0b2d4d010eb112dfe5216f11b2681f9.

Reason: Broke the UBSan buildbots. See more de

Revert "Re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper, with fixes."

This reverts commit 32d8d23cd0b2d4d010eb112dfe5216f11b2681f9.

Reason: Broke the UBSan buildbots. See more details on Phabricator:
https://reviews.llvm.org/D128544

show more ...

Re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper, with fixes.

The original commit was reverted in 3e9cc543f223 due to buildbot failures, which
should be fixed by the addition of de

Re-apply 5acd47169884, Add a shared-memory based orc::MemoryMapper, with fixes.

The original commit was reverted in 3e9cc543f223 due to buildbot failures, which
should be fixed by the addition of dependencies on librt.

Differential Revision: https://reviews.llvm.org/D128544

show more ...

Revert "[ORC] Add a shared-memory based orc::MemoryMapper."

This reverts commit 5acd471698849d9e322a29e6ca08791e8d447b7b.

Breaks shared library build with:

ld.lld-12: error: undefined symbol: sh

Revert "[ORC] Add a shared-memory based orc::MemoryMapper."

This reverts commit 5acd471698849d9e322a29e6ca08791e8d447b7b.

Breaks shared library build with:

ld.lld-12: error: undefined symbol: shm_open
>>> referenced by ExecutorSharedMemoryMapperService.cpp:68
(/home/culrho01/llvm-project/llvm/lib/ExecutionEngine/Orc/TargetProcess/ExecutorSharedMemoryMapperService.cpp:68)
>>>
lib/ExecutionEngine/Orc/TargetProcess/CMakeFiles/LLVMOrcTargetProcess.dir/ExecutorSharedMemoryMapperService.cpp.o:(llvm::orc::rt_bootstrap::ExecutorSharedMemoryMapperService::reserve[abi:cxx11](unsigned
long))
>>> did you mean: sem_open
>>> defined in:
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../aarch64-linux-gnu/libpthread.so

show more ...

[ORC] Add a shared-memory based orc::MemoryMapper.

This is an implementation of orc::MemoryMapper that maps shared memory
pages in both executor and controller process and writes directly to
them av

[ORC] Add a shared-memory based orc::MemoryMapper.

This is an implementation of orc::MemoryMapper that maps shared memory
pages in both executor and controller process and writes directly to
them avoiding transferring content over EPC. All allocations are properly
deinitialized automatically on the executor side at shutdown by the
ExecutorSharedMemoryMapperService.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D128544

show more ...

[ORC][JITLink] Merge JITLink AllocActionCall and ORC WrapperFunctionCall.

These types performed identical roles. Merging them simplifies interoperability
between JITLink and ORC APIs (allowing us to

[ORC][JITLink] Merge JITLink AllocActionCall and ORC WrapperFunctionCall.

These types performed identical roles. Merging them simplifies interoperability
between JITLink and ORC APIs (allowing us to address a few FIXMEs).

show more ...

Reintroduce "[ORC] Introduce EPCGenericRTDyldMemoryManager."

This reintroduces "[ORC] Introduce EPCGenericRTDyldMemoryManager."
(bef55a2b47a938ef35cbd7b61a1e5fa74e68c9ed) and "[lli] Add ChildTarget

Reintroduce "[ORC] Introduce EPCGenericRTDyldMemoryManager."

This reintroduces "[ORC] Introduce EPCGenericRTDyldMemoryManager."
(bef55a2b47a938ef35cbd7b61a1e5fa74e68c9ed) and "[lli] Add ChildTarget dependence
on OrcTargetProcess library." (7a219d801bf2c3006482cf3cbd3170b3b4ea2e1b) which were
reverted in 99951a56842d8e4cd0706cd17a04f77b5d0f6dd0 due to bot failures.

The root cause of the bot failures should be fixed by "[ORC] Fix uninitialized
variable." (0371049277912afc201da721fa659ecef7ab7fba) and "[ORC] Wait for
handleDisconnect to complete in SimpleRemoteEPC::disconnect."
(320832cc9b7e7fea5fc8afbed75c34c4a43287ba).

show more ...

Revert "[ORC] Introduce EPCGenericRTDyldMemoryManager."

This reverts commit bef55a2b47a938ef35cbd7b61a1e5fa74e68c9ed while I investigate
failures on some bots. Also reverts "[lli] Add ChildTarget de

Revert "[ORC] Introduce EPCGenericRTDyldMemoryManager."

This reverts commit bef55a2b47a938ef35cbd7b61a1e5fa74e68c9ed while I investigate
failures on some bots. Also reverts "[lli] Add ChildTarget dependence on
OrcTargetProcess library." (7a219d801bf2c3006482cf3cbd3170b3b4ea2e1b) which was
a fallow-up to bef55a2b47a.

show more ...

[ORC] Introduce EPCGenericRTDyldMemoryManager.

EPCGenericRTDyldMemoryMnaager is an EPC-based implementation of the
RuntimeDyld::MemoryManager interface. It enables remote-JITing via EPC (backed
by a

[ORC] Introduce EPCGenericRTDyldMemoryManager.

EPCGenericRTDyldMemoryMnaager is an EPC-based implementation of the
RuntimeDyld::MemoryManager interface. It enables remote-JITing via EPC (backed
by a SimpleExecutorMemoryManager instance on the executor side) for RuntimeDyld
clients.

The lli and lli-child-target tools are updated to use SimpleRemoteEPC and
SimpleRemoteEPCServer (rather than OrcRemoteTargetClient/Server), and
EPCGenericRTDyldMemoryManager for MCJIT tests.

By enabling remote-JITing for MCJIT and RuntimeDyld-based ORC clients,
EPCGenericRTDyldMemoryManager allows us to deprecate older remote-JITing
support, including OrcTargetClient/Server, OrcRPCExecutorProcessControl, and the
Orc RPC system itself. These will be removed in future patches.

show more ...

[ORC] Introduce EPCGenericDylibManager / SimpleExecutorDylibManager.

EPCGenericDylibManager provides an interface for loading dylibs and looking up
symbols in the executor, implemented using EPC-cal

[ORC] Introduce EPCGenericDylibManager / SimpleExecutorDylibManager.

EPCGenericDylibManager provides an interface for loading dylibs and looking up
symbols in the executor, implemented using EPC-calls to functions in the
executor.

SimpleExecutorDylibManager is an executor-side service that provides the
functions used by EPCGenericDylibManager.

SimpleRemoteEPC is updated to use an EPCGenericDylibManager instance to
implement the ExecutorProcessControl loadDylib and lookup methods. In a future
commit these methods will be removed, and clients updated to use
EPCGenericDylibManagers directly.

show more ...

[ORC] Add finalization & deallocation actions, SimpleExecutorMemoryManager class

Finalization and deallocation actions are a key part of the upcoming
JITLinkMemoryManager redesign: They generalize t

[ORC] Add finalization & deallocation actions, SimpleExecutorMemoryManager class

Finalization and deallocation actions are a key part of the upcoming
JITLinkMemoryManager redesign: They generalize the existing finalization and
deallocate concepts (basically "copy-and-mprotect", and "munmap") to include
support for arbitrary registration and deregistration of parts of JIT linked
code. This allows us to register and deregister eh-frames, TLV sections,
language metadata, etc. using regular memory management calls with no additional
IPC/RPC overhead, which should both improve JIT performance and simplify
interactions between ORC and the ORC runtime.

The SimpleExecutorMemoryManager class provides executor-side support for memory
management operations, including finalization and deallocation actions.

This support is being added in advance of the rest of the memory manager
redesign as it will simplify the introduction of an EPC based
RuntimeDyld::MemoryManager (since eh-frame registration/deregistration will be
expressible as actions). The new RuntimeDyld::MemoryManager will in turn allow
us to remove older remote allocators that are blocking the rest of the memory
manager changes.

show more ...

[ORC] Add Shared/OrcRTBridge, and TargetProcess/OrcRTBootstrap.

This is a small first step towards reorganization of the ORC libraries:

Declarations for types and function names (as strings) to be

[ORC] Add Shared/OrcRTBridge, and TargetProcess/OrcRTBootstrap.

This is a small first step towards reorganization of the ORC libraries:

Declarations for types and function names (as strings) to be found in the
"ORC runtime bootstrap" set are moved into OrcRTBridge.h / OrcRTBridge.cpp.

The current implementation of the "ORC runtime bootstrap" functions is moved
into OrcRTBootstrap.h and OrcRTBootstrap.cpp. It is likely that this code will
eventually be moved into ORT-RT proper (in compiler RT).

The immediate goal of this change is to make these bootstrap functions usable
for clients other than SimpleRemoteEPC/SimpleRemoteEPCServer. The first planned
client is a new RuntimeDyld::MemoryManager that will run over EPC, which will
allow us to remove the old OrcRemoteTarget code.

show more ...