[BPF] Handle anon record for CO-RE relocations

When doing experiment in kernel, for kernel data structure sockptr_t
in CO-RE operation, I hit an assertion error. The sockptr_t definition
and usage l

[BPF] Handle anon record for CO-RE relocations

When doing experiment in kernel, for kernel data structure sockptr_t
in CO-RE operation, I hit an assertion error. The sockptr_t definition
and usage look like below:
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} sockptr_t;
#pragma clang attribute pop
int test(sockptr_t *arg) {
return arg->is_kernel;
}
The assertion error looks like
clang: ../lib/Target/BPF/BPFAbstractMemberAccess.cpp:878: llvm::Value*
{anonymous}::BPFAbstractMemberAccess::computeBaseAndAccessKey(llvm::CallInst*,
{anonymous}::BPFAbstractMemberAccess::CallInfo&, std::__cxx11::string&,
llvm::MDNode*&): Assertion `TypeName.size()' failed.

In this particular, the clang frontend attach the debuginfo metadata associated
with anon structure with the preserve_access_info IR intrinsic. But the first
debuginfo type has to be a named type so libbpf can have a sound start to
do CO-RE relocation.

Besides the above approach using pragma to push attribute, the below typedef/struct
definition can have preserve_access_index directly applying to the anon struct.
typedef struct {
union {
void *kernel;
void *user;
};
unsigned is_kernel : 1;
} __attribute__((preserve_access_index) sockptr_t;

This patch fixed the issue by preprocessing function argument/return types
and local variable types used by other CO-RE intrinsics. For any
typedef struct/union { ... } typedef_name
an association of <anon struct/union, typedef> is recorded to replace
the IR intrinsic metadata 'anon struct/union' to 'typedef'.
It is possible that two different 'typedef' types may have identical
anon struct/union type. For such a case, the association will be
<anon struct/union, nullptr> to indicate the invalid case.

Differential Revision: https://reviews.llvm.org/D129621

show more ...

[BPF] Introduce support for type match relocations

Among others, BPF currently supports the type-exists CO-RE relocation
(e.g., see D83878 & D83242). Its intention, as the name tries to convey,
is t

[BPF] Introduce support for type match relocations

Among others, BPF currently supports the type-exists CO-RE relocation
(e.g., see D83878 & D83242). Its intention, as the name tries to convey,
is to be used for checking existence of a type in a target.
While that check is useful and has its place, we would also like to
be able to perform stricter type queries: instead of just checking mere
existence, we want to make sure that members match up in composite
types, that enum variants are present, etc. We refer to this as "type
match".

This change proposes the addition of a new relocation variant/value that
we intend to use for establishing this match relation.

Differential Revision: https://reviews.llvm.org/D126838

show more ...

[NFC] Rename Instrinsic to Intrinsic

[BPF] handle opaque-pointer for __builtin_preserve_enum_value

Opaque pointer [1] is enabled as the default with commit [2].
Andrii found that current __builtin_preserve_enum_value() can only handle

[BPF] handle opaque-pointer for __builtin_preserve_enum_value

Opaque pointer [1] is enabled as the default with commit [2].
Andrii found that current __builtin_preserve_enum_value() can only handle
non opaque pointer code pattern and will segfault with latest
llvm main branch where opaque-pointer is enabled by default.

This patch added the opaque pointer support.
Besides llvm selftests, also verified with bpf-next bpf selftests.

[1] https://llvm.org/docs/OpaquePointers.html
[2] https://reviews.llvm.org/D123122

Differential Revision: https://reviews.llvm.org/D123800

show more ...

[BPF] Don't fetch alignment of llvm.preserve.union.access.index argument

The way the alignment is determined here is not compatible with
opaque pointers -- we'd have to thread it through using eithe

[BPF] Don't fetch alignment of llvm.preserve.union.access.index argument

The way the alignment is determined here is not compatible with
opaque pointers -- we'd have to thread it through using either
align or elementtype attributes.

However, as far as I can tell this alignment is actually never
used for this particular intrinsic, so I've dropped the assignment
entirely and converted RecordAlignment to MaybeAlign, so we get an
assertion failure if it does end up being used.

Differential Revision: https://reviews.llvm.org/D122379

show more ...

[BPF] fix a CO-RE bitfield relocation error with >8 record alignment

Jussi Maki reported a fatal error like below for a bitfield
CO-RE relocation:
fatal error: error in backend: Unsupported field

[BPF] fix a CO-RE bitfield relocation error with >8 record alignment

Jussi Maki reported a fatal error like below for a bitfield
CO-RE relocation:
fatal error: error in backend: Unsupported field expression for
llvm.bpf.preserve.field.info, requiring too big alignment
The failure is related to kernel struct thread_struct. The following
is a simplied example.

Suppose we have below structure:
struct t2 {
int a[8];
} __attribute__((aligned(64))) __attribute__((preserve_access_index));
struct t1 {
int f1:1;
int f2:2;
struct t2 f3;
} __attribute__((preserve_access_index));

Note that struct t2 has aligned 64, which is used sometimes in the
kernel to enforce cache line alignment.

The above struct will be encoded into BTF and the following is what
C code looks like and the struct will appear in the file like vmlinux.h.
struct t2 {
int a[8];
long: 64;
long: 64;
long: 64;
long: 64;
} __attribute__((preserve_access_index));
struct t1 {
int f1: 1;
int f2: 2;
long: 61;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
long: 64;
struct t2 f3;
} __attribute__((preserve_access_index));

Note that after
origin_source -> BTF -> new_source
transition, the new source has the same memory layout as the old one
but the alignment interpretation inside the compiler could be different.
The bpf program will use the later explicitly padded structure as in
vmlinux.h.

In the above case, the compiler internal ABI alignment for new struct t1
is 16 while it is 4 for old struct t1. I didn't do a thorough investigation
why the ABI alignment is 16 and I suspect it is related to anonymous padding
in the above.

Current BPF bitfield CO-RE handling requires alignment <= 8 so proper
bitfield operatin can be performed. Therefore, alignment 16 will cause
a compiler fatal error.

To fix the ABI alignment >=16, let us check whether the bitfield
can be held within a 8-byte-aligned range. If this is the case,
we can use alignment 8. Otherwise, a fatal error will be reported.

Differential Revision: https://reviews.llvm.org/D121821

show more ...

[IR] Use CallBase::getParamElementType() (NFC)

As this method now exists on CallBase, use it rather than the
one on AttributeList.

Reduce dependencies on llvm/BinaryFormat/Dwarf.h

This header is very large (3M Lines once expended) and was included in location
where dwarf-specific information were not needed.

More specifically,

Reduce dependencies on llvm/BinaryFormat/Dwarf.h

This header is very large (3M Lines once expended) and was included in location
where dwarf-specific information were not needed.

More specifically, this commit suppresses the dependencies on
llvm/BinaryFormat/Dwarf.h in two headers: llvm/IR/IRBuilder.h and
llvm/IR/DebugInfoMetadata.h. As these headers (esp. the former) are widely used,
this has a decent impact on number of preprocessed lines generated during
compilation of LLVM, as showcased below.

This is achieved by moving some definitions back to the .cpp file, no
performance impact implied[0].

As a consequence of that patch, downstream user may need to manually some extra
files:

llvm/IR/IRBuilder.h no longer includes llvm/BinaryFormat/Dwarf.h
llvm/IR/DebugInfoMetadata.h no longer includes llvm/BinaryFormat/Dwarf.h

In some situations, codes maybe relying on the fact that
llvm/BinaryFormat/Dwarf.h was including llvm/ADT/Triple.h, this hidden
dependency now needs to be explicit.

$ clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Transforms/Scalar/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l
after: 10978519
before: 11245451

Related Discourse thread: https://llvm.discourse.group/t/include-what-you-use-include-cleanup
[0] https://llvm-compile-time-tracker.com/compare.php?from=fa7145dfbf94cb93b1c3e610582c495cb806569b&to=995d3e326ee1d9489145e20762c65465a9caeab4&stat=instructions

Differential Revision: https://reviews.llvm.org/D118781

show more ...

[llvm] Use nullptr instead of 0 (NFC)

Identified with modernize-use-nullptr.

[BPF] Use elementtype attribute for preserve.array/struct.index intrinsics

Use the elementtype attribute introduced in D105407 for the
llvm.preserve.array/struct.index intrinsics. It carries the
ele

[BPF] Use elementtype attribute for preserve.array/struct.index intrinsics

Use the elementtype attribute introduced in D105407 for the
llvm.preserve.array/struct.index intrinsics. It carries the
element type of the GEP these intrinsics effectively encode.

This patch:

* Adds a verifier check that the attribute is required.
* Adds it in the IRBuilder methods for these intrinsics.
* Autoupgrades old bitcode without the attribute.
* Updates the lowering code to use the attribute rather than
the pointer element type.
* Updates lots of tests to specify the attribute.
* Adds -force-opaque-pointers to the intrinsic-array.ll test
to demonstrate they work now.

https://reviews.llvm.org/D106184

show more ...

[IR] Deprecate GetElementPtrInst::CreateInBounds without element type

This API is not compatible with opaque pointers, the method
accepting an explicit pointer element type should be used instead.

[IR] Deprecate GetElementPtrInst::CreateInBounds without element type

This API is not compatible with opaque pointers, the method
accepting an explicit pointer element type should be used instead.

Thankfully there were few in-tree users. The BPF case still ends
up using the pointer element type for now and needs something like
D105407 to avoid doing so.

show more ...

BPF: fix FIELD_EXISTS relocation with array subscripts

Lorenz Bauer reported an issue in bpf mailing list ([1]) where
for FIELD_EXISTS relocation, if the object is an array subscript,
the patched im

BPF: fix FIELD_EXISTS relocation with array subscripts

Lorenz Bauer reported an issue in bpf mailing list ([1]) where
for FIELD_EXISTS relocation, if the object is an array subscript,
the patched immediate is the object offset from the base address,
instead of 1.

Currently in BPF AbstractMemberAccess pass, the final offset
from the base address is the patched offset except FIELD_EXISTS
which is 1 unconditionally. In this particular case, the last
data structure access is not a field (struct/union offset)
so it didn't hit the place to set patched immediate to be 1.

This patch fixed the issue by checking the relocation type.
If the type is FIELD_EXISTS, just set to 1.
Tested by modifying some bpf selftests, libbpf is okay with
such types with FIELD_EXISTS relocation.

[1] https://lore.kernel.org/bpf/CACAyw99n-cMEtVst7aK-3BfHb99GMEChmRLCvhrjsRpHhPrtvA@mail.gmail.com/

Differential Revision: https://reviews.llvm.org/D102036

show more ...

BPF: avoid duplicated globals for CORE relocations

This patch fixed two issues related with relocation globals.
In LLVM, if a global, e.g. with name "g", is created and
conflict with another global

BPF: avoid duplicated globals for CORE relocations

This patch fixed two issues related with relocation globals.
In LLVM, if a global, e.g. with name "g", is created and
conflict with another global with the same name, LLVM will
rename the global, e.g., with a new name "g.2". Since
relocation global name has special meaning, we do not want
llvm to change it, so internally we have logic to check
whether duplication happens or not. If happens, just reuse
the previous global.

The first bug is related to non-btf-id relocation
(BPFAbstractMemberAccess.cpp). Commit 54d9f743c8b0
("BPF: move AbstractMemberAccess and PreserveDIType passes
to EP_EarlyAsPossible") changed ModulePass to FunctionPass,
i.e., handling each function at a time. But still just
one BPFAbstractMemberAccess object is created so module
level de-duplication still possible. Commit 40251fee0084
("[BPF][NewPM] Make BPFTargetMachine properly adjust NPM optimizer
pipeline") made a change to create a BPFAbstractMemberAccess
object per function so module level de-duplication is not
possible any more without going through all module globals.
This patch simply changed the map which holds reloc globals
as class static, so it will be available to all
BPFAbstractMemberAccess objects for different functions.

The second bug is related to btf-id relocation
(BPFPreserveDIType.cpp). Before Commit 54d9f743c8b0, the pass
is a ModulePass, so we have a local variable, incremented for
each instance, and works fine. But after Commit 54d9f743c8b0,
the pass becomes a FunctionPass. Local variable won't work
properly since different functions will start with the same
initial value. Fix the issue by change the local count variable
as static, so it will be truely unique across the whole module
compilation.

Differential Revision: https://reviews.llvm.org/D88942

show more ...

[BPF][NewPM] Make BPFTargetMachine properly adjust NPM optimizer pipeline

This involves porting BPFAbstractMemberAccess and BPFPreserveDIType to
NPM, then adding them BPFTargetMachine::registerPassB

[BPF][NewPM] Make BPFTargetMachine properly adjust NPM optimizer pipeline

This involves porting BPFAbstractMemberAccess and BPFPreserveDIType to
NPM, then adding them BPFTargetMachine::registerPassBuilderCallbacks
(the NPM equivalent of adjustPassManager()).

Reviewed By: yonghong-song, asbirlea

Differential Revision: https://reviews.llvm.org/D88855

show more ...

BPF: move AbstractMemberAccess and PreserveDIType passes to EP_EarlyAsPossible

Move abstractMemberAccess and PreserveDIType passes as early as
possible, right after clang code generation.

Currently

BPF: move AbstractMemberAccess and PreserveDIType passes to EP_EarlyAsPossible

Move abstractMemberAccess and PreserveDIType passes as early as
possible, right after clang code generation.

Currently, compiler may transform the above code
p1 = llvm.bpf.builtin.preserve.struct.access(base, 0, 0);
p2 = llvm.bpf.builtin.preserve.struct.access(p1, 1, 2);
a = llvm.bpf.builtin.preserve_field_info(p2, EXIST);
if (a) {
p1 = llvm.bpf.builtin.preserve.struct.access(base, 0, 0);
p2 = llvm.bpf.builtin.preserve.struct.access(p1, 1, 2);
bpf_probe_read(buf, buf_size, p2);
}
to
p1 = llvm.bpf.builtin.preserve.struct.access(base, 0, 0);
p2 = llvm.bpf.builtin.preserve.struct.access(p1, 1, 2);
a = llvm.bpf.builtin.preserve_field_info(p2, EXIST);
if (a) {
bpf_probe_read(buf, buf_size, p2);
}
and eventually assembly code looks like
reloc_exist = 1;
reloc_member_offset = 10; //calculate member offset from base
p2 = base + reloc_member_offset;
if (reloc_exist) {
bpf_probe_read(bpf, buf_size, p2);
}
if during libbpf relocation resolution, reloc_exist is actually
resolved to 0 (not exist), reloc_member_offset relocation cannot
be resolved and will be patched with illegal instruction.
This will cause verifier failure.

This patch attempts to address this issue by do chaining
analysis and replace chains with special globals right
after clang code gen. This will remove the cse possibility
described in the above. The IR typically looks like
%6 = load @llvm.sk_buff:0:50$0:0:0:2:0
%7 = bitcast %struct.sk_buff* %2 to i8*
%8 = getelementptr i8, i8* %7, %6
for a particular address computation relocation.

But this transformation has another consequence, code sinking
may happen like below:
PHI = <possibly different @preserve_*_access_globals>
%7 = bitcast %struct.sk_buff* %2 to i8*
%8 = getelementptr i8, i8* %7, %6

For such cases, we will not able to generate relocations since
multiple relocations are merged into one.

This patch introduced a passthrough builtin
to prevent such optimization. Looks like inline assembly has more
impact for optimizaiton, e.g., inlining. Using passthrough has
less impact on optimizations.

A new IR pass is introduced at the beginning of target-dependent
IR optimization, which does:
- report fatal error if any reloc global in PHI nodes
- remove all bpf passthrough builtin functions

Changes for existing CORE tests:
- for clang tests, add "-Xclang -disable-llvm-passes" flags to
avoid builtin->reloc_global transformation so the test is still
able to check correctness for clang generated IR.
- for llvm CodeGen/BPF tests, add "opt -O2 <ir_file> | llvm-dis" command
before "llc" command since "opt" is needed to call newly-placed
builtin->reloc_global transformation. Add target triple in the IR
file since "opt" requires it.
- Since target triple is added in IR file, if a test may produce
different results for different endianness, two tests will be
created, one for bpfeb and another for bpfel, e.g., some tests
for relocation of lshift/rshift of bitfields.
- field-reloc-bitfield-1.ll has different relocations compared to
old codes. This is because for the structure in the test,
new code returns struct layout alignment 4 while old code
is 8. Align 8 is more precise and permits double load. With align 4,
the new mechanism uses 4-byte load, so generating different
relocations.
- test intrinsic-transforms.ll is removed. This is used to test
cse on intrinsics so we do not lose metadata. Now metadata is attached
to global and not instruction, it won't get lost with cse.

Differential Revision: https://reviews.llvm.org/D87153

show more ...

BPF: support type exist/size and enum exist/value relocations

Four new CO-RE relocations are introduced:
- TYPE_EXISTENCE: whether a typedef/record/enum type exists
- TYPE_SIZE: the size of a ty

BPF: support type exist/size and enum exist/value relocations

Four new CO-RE relocations are introduced:
- TYPE_EXISTENCE: whether a typedef/record/enum type exists
- TYPE_SIZE: the size of a typedef/record/enum type
- ENUM_VALUE_EXISTENCE: whether an enum value of an enum type exists
- ENUM_VALUE: the enum value of an enum type

These additional relocations will make CO-RE bpf programs
more adaptive for potential kernel internal data structure
changes.

Differential Revision: https://reviews.llvm.org/D83878

show more ...

[Alignment][NFC] Use Align for BPFAbstractMemberAccess::RecordAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/ll

[Alignment][NFC] Use Align for BPFAbstractMemberAccess::RecordAlignment

This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Differential Revision: https://reviews.llvm.org/D82962

show more ...

[BPF] Fix a bug for __builtin_preserve_field_info() processing

Andrii discovered a problem where a simple case similar to below
will generate wrong relocation kind:
enum { FIELD_EXISTENCE = 2, };

[BPF] Fix a bug for __builtin_preserve_field_info() processing

Andrii discovered a problem where a simple case similar to below
will generate wrong relocation kind:
enum { FIELD_EXISTENCE = 2, };
struct s1 { int a1; };
int test() {
struct s1 *v = 0;
return __builtin_preserve_field_info(v[0], FIELD_EXISTENCE);
}
The expected relocation kind should be FIELD_EXISTENCE, but
recorded reloc kind in the final object file is FIELD_BYTE_OFFSET,
which is incorrect.

This exposed a bug in generating access strings from intrinsics.
The current access string generation has two steps:
step 1: find the base struct/union type,
step 2: traverse members in the base type.

The current implementation relies on at lease one member access
in step 2 to get the correct relocation kind, which is true
in typical cases. But if there is no member accesses, the current
implementation falls to the default info kind FIELD_BYTE_OFFSET.
This is incorrect, we should still record the reloc kind
based on the user input. This patch fixed this issue by properly
recording the reloc kind in such cases.

Differential Revision: https://reviews.llvm.org/D82932

show more ...

Remove global std::strings. NFCI.

[IR] Replace all uses of CallBase::getCalledValue() with getCalledOperand().

This method has been commented as deprecated for a while. Remove
it and replace all uses with the equivalent getCalledOpe

[IR] Replace all uses of CallBase::getCalledValue() with getCalledOperand().

This method has been commented as deprecated for a while. Remove
it and replace all uses with the equivalent getCalledOperand().

I also made a few cleanups in here. For example, to removes use
of getElementType on a pointer when we could just use getFunctionType
from the call.

Differential Revision: https://reviews.llvm.org/D78882

show more ...

[NFC] Clean up uses of LoadInst constructor.

[BPF] handle typedef of struct/union for CO-RE relocations

Linux commit
https://github.com/torvalds/linux/commit/1cf5b23988ea0086a252a5c8b005b075f1e9b030#diff-289313b9fec99c6f0acfea19d9cfd949
uses

[BPF] handle typedef of struct/union for CO-RE relocations

Linux commit
https://github.com/torvalds/linux/commit/1cf5b23988ea0086a252a5c8b005b075f1e9b030#diff-289313b9fec99c6f0acfea19d9cfd949
uses "#pragma clang attribute push (__attribute__((preserve_access_index)),
apply_to = record)"
to apply CO-RE relocations to all records including the following pattern:
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
typedef struct {
int a;
} __t;
#pragma clang attribute pop
int test(__t *arg) { return arg->a; }

The current approach to use struct/union type in the relocation record will
result in an anonymous struct, which make later type matching difficult
in bpf loader. In fact, current BPF backend will fail the above program
with assertion:
clang: ../lib/Target/BPF/BPFAbstractMemberAccess.cpp:796: ...
Assertion `TypeName.size()' failed.

clang will change to use the type of the base of the member access
which will preserve the typedef modifier for the
preserve_{struct,union}_access_index intrinsics in the above example.
Here we adjust BPF backend to accept that the debuginfo
type metadata may be 'typedef' and handle them properly.

Differential Revision: https://reviews.llvm.org/D73902

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[BPF] add "llvm." prefix to BPF internally created globals

Currently, BPF backend creates some global variables with name like
<type_name>:<reloc_type>:<patch_imm>$<access_str>
to carry certain in

[BPF] add "llvm." prefix to BPF internally created globals

Currently, BPF backend creates some global variables with name like
<type_name>:<reloc_type>:<patch_imm>$<access_str>
to carry certain information to BPF backend.

With direct clang compilation, the following code in
llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
is triggered and the above globals are emitted to the ELF file.
(clang enabled this as opt flag -faddrsig is on by default.)
if (TM.Options.EmitAddrsig) {
// Emit address-significance attributes for all globals.
OutStreamer->EmitAddrsig();
for (const GlobalValue &GV : M.global_values())
if (!GV.use_empty() && !GV.isThreadLocal() &&
!GV.hasDLLImportStorageClass() && !GV.getName().startswith("llvm.") &&
!GV.hasAtLeastLocalUnnamedAddr())
OutStreamer->EmitAddrsigSym(getSymbol(&GV));
}
...
10162: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND tcp_sock:0:2048$0:117
10163: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND tcp_sock:0:2112$0:126:0
10164: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND tcp_sock:1:8$0:31:6
...
While in llc, those globals are not emited since EmitAddrsig
default option is false for llc. The llc flag "-addrsig" can be used to
enable the above code.

This patch added "llvm." prefix to these internal globals so that
they can be ignored in the above codes and possible other
places.

Differential Revision: https://reviews.llvm.org/D70703

show more ...

Fix uninitialized variable warning. NFCI.