Changes in BPFISelLowering.cpp

50f5cef3 - [BPF] Implement mod operation

Wed, 11 May 2022 06:13:35 +0000

[BPF] Implement mod operationImplement BPF_MOD instruction to fix lack of assembly parser support mentioned in https://github.com/llvm/llvm-project/issues/55192.Reviewed By: astDifferential Revision: https://reviews.llvm.org/D125207 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

58989793 - BPF: support inlining __builtin_memcmp intrinsic call

Tue, 29 Mar 2022 06:30:10 +0000

BPF: support inlining __builtin_memcmp intrinsic callDelyan Kratunov reported an issue where __builtin_memcmp isnot inlined into simple load/compare instructions.This is a known issue. In the current state, __builtin_memcmpwill be converted to memcmp call which won't work forbpf programs.This patch added support for expanding __builtin_memcmp withactual loads and compares up to currently maximum 128 total loads.The implementation is identical to PowerPC.Differential Revision: https://reviews.llvm.org/D122676 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

5a667c0e - [llvm] Use nullptr instead of 0 (NFC)

Tue, 28 Dec 2021 16:52:25 +0000

[llvm] Use nullptr instead of 0 (NFC)Identified with modernize-use-nullptr. List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

5b8bbbec - [NFC][llvm] Inclusive language: reword and remove uses of sanity in llvm/lib/Target

Thu, 18 Nov 2021 02:59:00 +0000

[NFC][llvm] Inclusive language: reword and remove uses of sanity in llvm/lib/TargetReworded removed code comments that contain `sanity check` and `sanitytest`. List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

3562ad3e - BPF: implement isLegalAddressingMode() properly

Mon, 27 Sep 2021 15:21:48 +0000

BPF: implement isLegalAddressingMode() properlyLatest upstream llvm caused the kernel bpf selftest emitting thefollowing warnings: In file included from progs/profiler3.c:6: progs/profiler.inc.h:489:2: warning: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering [-Wpass-failed=transform-warning] for (int i = 0; i < MAX_PATH_DEPTH; i++) { ^Further bisecting shows this SimplifyCFG patch ([1]) changedthe condition on how to fold branch to common dest. This causedsome unroll pragma is not honored in selftests/bpf.The patch [1] test getUserCost() as the condition toperform the certain basic block folding transformation.For the above example, before the loop unroll pass, the control flowlooks like: cond_block: branch target: body_block, cleanup_block body_block: branch target: cleanup_block, end_block end_block: branch target: cleanup_block, end10_block end10_block: %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2 %inc = add nuw nsw i32 %i.0, 1 branch target: cond_blockIn the above, %call2 is an unknown scalar.Before patch [1], end10_block will be folded into end_block, formingthe code like cond_block: branch target: body_block, cleanup_block body_block: branch target: cleanup_block, end_block end_block: branch target: cleanup_block, cond_blockand the compiler is happy to perform unrolling.With patch [1], getUserCost(), which calls getGEPCost(), which callsisLegalAddressingMode() in TargetLoweringBase.cpp, considers IR %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2is free, so the above basic block folding transformation is not performedand unrolling does not happen.For BPF target, the IR %add.ptr = getelementptr i8, i8* %payload.addr.0, i64 %call2is not free and we don't have ld/st instruction address with 'r+r' mode.This patch implemented a BPF hook for isLegalAddressingMode(), which isidentical to Mips isLegalAddressingMode() implementation wherethe address pattern like 'r+r', 'r+r+i' or '2*r' are not allowed.With testing kernel bpf selftests, all loop not unrolled warningsare gone and all selftests run successfully. [1] https://reviews.llvm.org/D108837Differential Revision: https://reviews.llvm.org/D110789 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

833e9b2e - [BPF] add support for 32 bit registers in inline asm

Sun, 16 May 2021 16:32:36 +0000

[BPF] add support for 32 bit registers in inline asmAdd "w" constraint type which allows selecting 32 bit registers.32 bit registers were added in https://reviews.llvm.org/rGca31c3bb3ff149850b664838fbbc7d40ce571879.Differential Revision: https://reviews.llvm.org/D102118 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

379d9088 - BPF: provide better error message for unsupported atomic operations

Thu, 11 Mar 2021 20:28:04 +0000

BPF: provide better error message for unsupported atomic operationsCurrently, BPF backend does not support all variants of atomic_load_{add,and,or,xor}, atomic_swap and atomic_cmp_swapFor example, it only supports 32bit (with alu32 mode) and 64bitoperations for atomic_load_{and,or,xor}, atomic_swap andatomic_cmp_swap. Due to historical reason, atomic_load_add isalways supported with 32bit and 64bit.If user used an unsupported atomic operation, currently,codegen selectiondag cannot find bpf support and will issuea fatal error. This is not user friendly as user may mistakenlythink this is a compiler bug.This patch added Custom rule for unsupported atomic operationsand will emit better error message during ReplaceNodeResults()callback. The following is an example output. $ cat t.c short sync(short *p) { return __sync_val_compare_and_swap (p, 2, 3); } $ clang -target bpf -O2 -g -c t.c t.c:2:11: error: Unsupported atomic operations, please use 64 bit version return __sync_val_compare_and_swap (p, 2, 3); ^ fatal error: error in backend: Cannot select: t19: i64,ch = AtomicCmpSwap<(load store seq_cst seq_cst 2 on %ir.p)> t0, t2, Constant:i64<2>, Constant:i64<3>, t.c:2:11 t2: i64,ch = CopyFromReg t0, Register:i64 %0 t1: i64 = Register %0 t11: i64 = Constant<2> t10: i64 = Constant<3> In function: sync PLEASE submit a bug report ...Fatal error will still happen since we did not really do properlowering for these unsupported atomic operations. But we do geta much better error message.Differential Revision: https://reviews.llvm.org/D98471 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

ca1e69a6 - [NFC] remove unused includes of SelectionDAGISel.h

Sat, 11 Jul 2020 02:47:12 +0000

[NFC] remove unused includes of SelectionDAGISel.h List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

13f6c81c - [BPF] simplify zero extension with MOV_32_64

Wed, 27 May 2020 18:18:16 +0000

[BPF] simplify zero extension with MOV_32_64The current pattern matching for zext results in the following code snippetbeing produced, w1 = w0 r1 <<= 32 r1 >>= 32Because BPF implementations require zero extension on 32bit loads thisboth adds a few extra unneeded instructions but also makes it a bitharder for the verifier to track the r1 register bounds. For example inthis verifier trace we see at the end of the snippet R2 offset is unknown.However, if we track this correctly we see w1 should have the same boundsas r8. R8 smax is less than U32 max value so a zero extend load should keepthe same value. Adding a max value of 800 (R8=inv(id=0,smax_value=800)) toan off=0, as seen in R7 should create a max offset of 800. However at theend of the snippet we note the R2 max offset is 0xffffFFFF. R0=inv(id=0,smax_value=800) R1_w=inv(id=0,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R6=ctx(id=0,off=0,imm=0) R7=map_value(id=0,off=0,ks=4,vs=1600,imm=0) R8_w=inv(id=0,smax_value=800,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R9=inv800 R10=fp0 fp-8=mmmm???? 58: (1c) w9 -= w8 59: (bc) w1 = w8 60: (67) r1 <<= 32 61: (77) r1 >>= 32 62: (bf) r2 = r7 63: (0f) r2 += r1 64: (bf) r1 = r6 65: (bc) w3 = w9 66: (b7) r4 = 0 67: (85) call bpf_get_stack#67 R0=inv(id=0,smax_value=800) R1_w=ctx(id=0,off=0,imm=0) R2_w=map_value(id=0,off=0,ks=4,vs=1600,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R3_w=inv(id=0,umax_value=800,var_off=(0x0; 0x3ff)) R4_w=inv0 R6=ctx(id=0,off=0,imm=0) R7=map_value(id=0,off=0,ks=4,vs=1600,imm=0) R8_w=inv(id=0,smax_value=800,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R9_w=inv(id=0,umax_value=800,var_off=(0x0; 0x3ff)) R10=fp0 fp-8=mmmm????After this patch R1 bounds are not smashed by the <<=32 >>=32 shift and weget correct bounds on R2 umax_value=800.Further it reduces 3 insns to 1.Signed-off-by: John Fastabend Differential Revision: https://reviews.llvm.org/D73985 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

61bd33e3 - [BPF] explicit warning of not supporting dynamic stack allocation

Thu, 13 Feb 2020 04:00:20 +0000

[BPF] explicit warning of not supporting dynamic stack allocationCurrently, BPF does not support dynamic static allocation.For a program like below: extern void bar(int *); void foo(int n) { int a[n]; bar(a); }The current error message looks like: unimplemented operand UNREACHABLE executed at /.../llvm/lib/Target/BPF/BPFISelLowering.cpp:199!Let us make error message explicit so it will be clear to the userwhat is the problem. With this patch, the error message looks like: fatal error: error in backend: Unsupported dynamic stack allocation ...Differential Revision: https://reviews.llvm.org/D74521 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

29bc5dd1 - [BPF] implement isTruncateFree and isZExtFree in BPFTargetLowering

Wed, 05 Feb 2020 18:27:43 +0000

[BPF] implement isTruncateFree and isZExtFree in BPFTargetLoweringCurrently, isTruncateFree() and isZExtFree() callbacks return falseas they are not implemented in BPF backend. This may cause suboptimalcode generation. For example, if the load in the context of zero extensionhas more than one use, the pattern zextload{i8,i16,i32} willnot be generated. Rather, the load will be matched first andthen the result is zero extended.For example, in the test together with this commit, we have I1: %0 = load i32, i32* %data_end1, align 4, !tbaa !2 I2: %conv = zext i32 %0 to i64 ... I3: %2 = load i32, i32* %data, align 4, !tbaa !7 I4: %conv2 = zext i32 %2 to i64 ... I5: %4 = trunc i64 %sub.ptr.lhs.cast to i32 I6: %conv13 = sub i32 %4, %2 ...The I1 and I2 will match to one zextloadi32 DAG node, where SUBREG_TO_REG isused to convert a 32bit register to 64bit one. During code generation,SUBREG_TO_REG is a noop.The %2 in I3 is used in both I4 and I6. If isTruncateFree() is false,the current implementation will generate a SLL_ri and SRL_rifor the zext part during lowering.This patch implement isTruncateFree() in the BPF backend, so for theabove example, I3 and I4 will generate a zextloadi32 DAG node withSUBREG_TO_REG is generated during lowering to Machine IR.isZExtFree() is also implemented as it should help code gen as well.This patch also enables the change in https://reviews.llvm.org/D73985since it won't kick in generates MOV_32_64 machine instruction.Differential Revision: https://reviews.llvm.org/D74101 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

18f805a7 - [Alignment][NFC] Remove unneeded llvm:: scoping on Align types

Fri, 27 Sep 2019 12:54:21 +0000

[Alignment][NFC] Remove unneeded llvm:: scoping on Align typesllvm-svn: 373081 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

ad1cea0d - [Alignment][NFC] Use Align with TargetLowering::setPrefFunctionAlignment

Fri, 06 Sep 2019 15:03:49 +0000

[Alignment][NFC] Use Align with TargetLowering::setPrefFunctionAlignmentSummary:This is patch is part of a series to introduce an Alignment type.See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.htmlSee this patch for the introduction of the type: https://reviews.llvm.org/D64790Reviewers: courbetSubscribers: nemanjai, javed.absar, hiraditya, kbarton, asb, rbar, johnrusso, simoncook, apazos, sabuasal, niosHD, jrtc27, MaskRay, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, jsji, s.egerton, pzheng, ychen, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D67267llvm-svn: 371212 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

4fc3ad9e - [Alignment][NFC] Use Align with TargetLowering::setMinFunctionAlignment

Fri, 06 Sep 2019 12:48:34 +0000

[Alignment][NFC] Use Align with TargetLowering::setMinFunctionAlignmentSummary:This is patch is part of a series to introduce an Alignment type.See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.htmlSee this patch for the introduction of the type: https://reviews.llvm.org/D64790Reviewers: courbetSubscribers: jyknight, sdardis, nemanjai, javed.absar, hiraditya, kbarton, fedor.sergeev, asb, rbar, johnrusso, simoncook, apazos, sabuasal, niosHD, jrtc27, MaskRay, zzheng, edward-jones, atanasyan, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, jsji, s.egerton, pzheng, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D67229llvm-svn: 371200 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

aff45e4b - [LLVM][Alignment] Make functions using log of alignment explicit

Thu, 05 Sep 2019 10:00:22 +0000

[LLVM][Alignment] Make functions using log of alignment explicitSummary:This patch renames functions that takes or returns alignment as log2, this patch will help with the transition to llvm::Align.The renaming makes it explicit that we deal with log(alignment) instead of a power of two alignment.A few renames uncovered dubious assignments: - `MirParser`/`MirPrinter` was expecting powers of two but `MachineFunction` and `MachineBasicBlock` were using deal with log2(align). This patch fixes it and updates the documentation. - `MachineBlockPlacement` exposes two flags (`align-all-blocks` and `align-all-nofallthru-blocks`) supposedly interpreted as power of two alignments, internally these values are interpreted as log2(align). This patch updates the documentation, - `MachineFunctionexposes` exposes `align-all-functions` also interpreted as power of two alignment, internally this value is interpreted as log2(align). This patch updates the documentation,Reviewers: lattner, thegameg, courbetSubscribers: dschuff, arsenm, jyknight, dylanmckay, sdardis, nemanjai, jvesely, nhaehnle, javed.absar, hiraditya, kbarton, fedor.sergeev, asb, rbar, johnrusso, simoncook, apazos, sabuasal, niosHD, jrtc27, MaskRay, zzheng, edward-jones, atanasyan, rogfer01, MartinMosbeck, brucehoult, the_o, dexonsmith, PkmX, jocewei, jsji, Jim, s.egerton, llvm-commits, courbetTags: #llvmDifferential Revision: https://reviews.llvm.org/D65945llvm-svn: 371045 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

0c476111 - Apply llvm-prefer-register-over-unsigned from clang-tidy to LLVM

Thu, 15 Aug 2019 19:22:08 +0000

Apply llvm-prefer-register-over-unsigned from clang-tidy to LLVMSummary:This clang-tidy check is looking for unsigned integer variables whose initializerstarts with an implicit cast from llvm::Register and changes the type of thevariable to llvm::Register (dropping the llvm:: where possible).Partial reverts in:X86FrameLowering.cpp - Some functions return unsigned and arguably should be MCRegisterX86FixupLEAs.cpp - Some functions return unsigned and arguably should be MCRegisterX86FrameLowering.cpp - Some functions return unsigned and arguably should be MCRegisterHexagonBitSimplify.cpp - Function takes BitTracker::RegisterRef which appears to be unsigned&MachineVerifier.cpp - Ambiguous operator==() given MCRegister and const RegisterPPCFastISel.cpp - No Register::operator-=()PeepholeOptimizer.cpp - TargetInstrInfo::optimizeLoadInstr() takes an unsigned&MachineTraceMetrics.cpp - MachineTraceMetrics lacks a suitable constructorManual fixups in:ARMFastISel.cpp - ARMEmitLoad() now takes a Register& instead of unsigned&HexagonSplitDouble.cpp - Ternary operator was ambiguous between unsigned/RegisterHexagonConstExtenders.cpp - Has a local class named Register, used llvm::Register instead of Register.PPCFastISel.cpp - PPCEmitLoad() now takes a Register& instead of unsigned&Depends on D65919Reviewers: arsenm, bogner, craig.topper, RKSimonReviewed By: arsenmSubscribers: RKSimon, craig.topper, lenary, aemerson, wuzish, jholewinski, MatzeB, qcolombet, dschuff, jyknight, dylanmckay, sdardis, nemanjai, jvesely, wdng, nhaehnle, sbc100, jgravelle-google, kristof.beyls, hiraditya, aheejin, kbarton, fedor.sergeev, javed.absar, asb, rbar, johnrusso, simoncook, apazos, sabuasal, niosHD, jrtc27, MaskRay, zzheng, edward-jones, atanasyan, rogfer01, MartinMosbeck, brucehoult, the_o, tpr, PkmX, jocewei, jsji, Petar.Avramovic, asbirlea, Jim, s.egerton, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D65962llvm-svn: 369041 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

66b18e57 - [BPF] add code-gen support for JMP32 instructions

Thu, 07 Feb 2019 10:43:09 +0000

[BPF] add code-gen support for JMP32 instructionsJMP32 instructions has been added to eBPF ISA. They are 32-bit variants ofexisting BPF conditional jump instructions, but the comparison happens onlow 32-bit sub-register only, therefore some unnecessary extensions couldbe saved.JMP32 instructions will only be available for -mcpu=v3. Host probe hook hasbeen updated accordingly.JMP32 instructions will only be enabled in code-gen when -mattr=+alu32enabled, meaning compiling the program using sub-register mode.For JMP32 encoding, it is a new instruction class, and is using thereserved eBPF class number 0x6.This patch has been tested by compiling and running kernel bpf selftestswith JMP32 enabled.Acked-by: Yonghong Song Signed-off-by: Jiong Wang llvm-svn: 353384 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

2946cd70 - Update the file headers across all of the LLVM projects in the monorepo

Sat, 19 Jan 2019 08:50:56 +0000

Update the file headers across all of the LLVM projects in the monorepoto reflect the new license.We understand that people may be surprised that we're moving the headerentirely to discuss the new license. We checked this carefully with theFoundation's lawyer and we believe this is the correct approach.Essentially, all code in the project is now made available by the LLVMproject under our new license, so you will see that the license headersinclude that license only. Some of our contributors have contributedcode under our old license, and accordingly, we have retained a copy ofour old license notice in the top-level files in each project andrepository.llvm-svn: 351636 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

71d81e5c - bpf: new option -bpf-expand-memcpy-in-order to expand memcpy in order

Wed, 25 Jul 2018 22:40:02 +0000

bpf: new option -bpf-expand-memcpy-in-order to expand memcpy in orderSome BPF JIT backends would want to optimize memcpy in their ownarchitecture specific way.However, at the moment, there is no way for JIT backends to see memcpysemantics in a reliable way. This is due to LLVM BPF backend is expandingmemcpy into load/store sequences and could possibly schedule them apart fromeach other further. So, BPF JIT backends inside kernel can't reliablyrecognize memcpy semantics by peephole BPF sequence.This patch introduce new intrinsic expand infrastructure to memcpy.To get stable in-order load/store sequence from memcpy, we first lowermemcpy into BPF::MEMCPY node which then expanded into in-order load/storesequences in expandPostRAPseudo pass which will happen after instructionscheduling. By this way, kernel JIT backends could reliably recognizememcpy through scanning BPF sequence.This new memcpy expand infrastructure is gated by a new option: -bpf-expand-memcpy-in-orderAcked-by: Jakub Kicinski Signed-off-by: Jiong Wang Signed-off-by: Yonghong Song llvm-svn: 337977 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp

8467411d - Set ADDE/ADDC/SUBE/SUBC to expand by default

Fri, 01 Jun 2018 13:21:33 +0000

Set ADDE/ADDC/SUBE/SUBC to expand by defaultSummary:They've been deprecated in favor of UADDO/ADDCARRY or USUBO/SUBCARRY for a while.Target that uses these opcodes are changed in order to ensure their behavior doesn't change.Reviewers: efriedma, craig.topper, dblaikie, bkramerSubscribers: jholewinski, arsenm, jyknight, sdardis, nemanjai, nhaehnle, kbarton, fedor.sergeev, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, niosHD, jrtc27, zzheng, edward-jones, mgrang, atanasyan, llvm-commitsDifferential Revision: https://reviews.llvm.org/D47422llvm-svn: 333748 List of files: /llvm-project-15.0.7/llvm/lib/Target/BPF/BPFISelLowering.cpp