en Copyright 2015 Java http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#5f620d00 [LV] Update handling of scalable pointer inductions after b73d2c8.The dependent code has been changed quite a lot since 151c144 whichb73d2c8 effectively reverts. Now we run into a case where loweringdidn't expect/support the behavior pre 151c144 any longer.Update the code dealing with scalable pointer inductions to also checkfor uniformity in combination with isScalarAfterVectorization. Thisshould ensure scalable pointer inductions are handled properly duringepilogue vectorization.Fixes #57912.(cherry picked from commit 2c692d891ed639779b1c4b504ca63037bbacc0e8) List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Fri, 23 Sep 2022 17:23:01 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#b73d2c8c [LV] Keep track of cost-based ScalarAfterVec in VPWidenPointerInd.Epilogue vectorization uses isScalarAfterVectorization to check ifwidened versions for inductions need to be generated and bails out inthose cases.At the moment, there are scenarios where isScalarAfterVectorizationreturns true but VPWidenPointerInduction::onlyScalarsGenerated wouldreturn false, causing widening.This can lead to widened phis with incorrect start values being createdin the epilogue vector body.This patch addresses the issue by storing the cost-model decision inVPWidenPointerInductionRecipe and restoring the behavior before 151c144.This effectively reverts 151c144, but the long-term fix is to properlysupport widened inductions during epilogue vectorizationFixes #57712. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Mon, 19 Sep 2022 17:14:34 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#03fee671 [LoopVectorize] Add option to use active lane mask for loop control flowCurrently, for vectorised loops that use the get.active.lane.maskintrinsic we only use the mask for predicated vector operations,such as masked loads and stores, etc. The loop itself is stillcontrolled by comparing the canonical induction variable with thetrip count. However, for some targets this is inefficient when it'scheap to use the mask itself to control the loop.This patch adds support for using the active lane mask for controlflow by:1. Generating the active lane mask for the next iteration of thevector loop, rather than the current one. If there are still anyremaining iterations then at least the first bit of the mask willbe set.2. Extract the first bit of this mask and use this bit for theconditional branch.I did this by creating a new VPActiveLaneMaskPHIRecipe that setsup the initial PHI values in the vector loop pre-header. I've alsomade use of the new BranchOnCond VPInstruction for the finalinstruction in the loop region.Differential Revision: https://reviews.llvm.org/D125301 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Tue, 10 May 2022 09:49:43 +0000 David Sherwood <david.sherwood@arm.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#02d6950d [LoopVectorize][NFC] Add optional Name parameter to VPInstructionThis patch is a simple piece of refactoring that now permits usersto create VPInstructions and specify the name of the value beinggenerated. This is useful for creating more readable/meaningfulnames in IR.Differential Revision: https://reviews.llvm.org/D128982 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Fri, 01 Jul 2022 09:19:45 +0000 David Sherwood <david.sherwood@arm.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#b0da3c6f [VPlan] Move setDebugLocFromInst to VPTransformState (NFC).The moved helpers are only used for codegen. It will allow moving theremaining ::execute implementations out of LoopVectorize.cpp.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D128657 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Sat, 02 Jul 2022 14:18:16 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#0dddf04c [LV] Don't optimize exit cond during epilogue vectorization.At the moment, the same VPlan can be used code generation of both themain vector and epilogue vector loop. This can lead to wrong results, ifthe plan is optimized based on the VF of the main vector loop and thenre-used for the epilogue loop.One example where this is problematic is if the scalar loops need toexecute at least one iteration, e.g. due to interleave groups.To prevent mis-compiles in the short-term, disable optimizing exitconditions for VPlans when using epilogue vectorization. The proper fixis to avoid re-using the same plan for both loops, which will requiresupport for cloning plans first.Fixes #56319. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Fri, 01 Jul 2022 12:48:38 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#583abd0e [VPlan] Move addMetadata to VPTransformState (NFC).The moved helpers are only used for codegen. It will allow moving theremaining ::execute implementations out of LoopVectorize.cpp.Depends on D127966.Depends on D127965.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D127968 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Fri, 01 Jul 2022 11:03:24 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#85983ca4 [VPlan] Replace remaining use of needsScalarIV.All information is already available in VPlan. Note that there are sometest changes, because we now can correctly look through instructionslike truncates to analyze the actual users.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D123541 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Thu, 09 Jun 2022 11:05:37 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#cedfd7a2 Recommit "[VPlan] Remove uneeded needsVectorIV check."This reverts commit 266ea446ab747671eb6c736569c3c9c5f3c53d11.The reasons for the revert have been addressed by cleaning up conditionhandling in VPlan and properly marking VPBranchOnMaskRecipe as usingscalars.The test case for the revert from D123720 has been added in 3d663308a5d. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Wed, 08 Jun 2022 13:06:45 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#a5bb4a3b [VPlan] Replace CondBit with BranchOnCond VPInstruction.This patch removes CondBit and Predicate from VPBasicBlock. To do so,the patch introduces a new branch-on-cond VPInstruction opcode to modela branch on a condition explicitly.This addresses a long-standing TODO/FIXME that blocks shouldn't be usersof VPValues. Those extra users can cause issues for VPValue-basedanalyses that don't expect blocks. Addressing this fixme should allow usto re-introduce 266ea446ab7476.The generic branch opcode can also be used in follow-up patches.Depends on D123005.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D126618 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Fri, 03 Jun 2022 10:47:16 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#05776122 [VPlan] Use region for each loop in native path.This patch updates the VPlan native path to use VPRegionBlocks for allloops in a loop nest. Up to now, only the outermost loop used a region.This is a step towards unifying both paths and keep things consistentbetween them. It also prepares various code-gen parts for modeling thepre-header in the inner loop vectorizer (D121624).Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D123005 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Wed, 01 Jun 2022 09:41:04 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#d1570194 [VPlan] Remove unused native utilities incompatible with nested regions.The implementations of VPlanDominatorTree, VPlanLoopInfo and VPlanPredicatorare all incompatible with modeling loops in VPlans as region withoutexplicit back-edges.Those pieces are not actively used and only exercised by a few gtestunit tests. They are at the moment blocking progress towards unifyingthe native and inner-loop vectorizer paths in D121624 and D123005.I think we should not block forward progress on unused pieces of code,so this patch removes the utilities for now. The plan is to re-introducethem as needed in a way that is compatible with the unified VPlan schemeused in both the inner loop vectorizer and the native path.Reviewed By: sguggillDifferential Revision: https://reviews.llvm.org/D123017 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Wed, 01 Jun 2022 08:32:59 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#6abce17f [VPlan] Use Exiting-block instead of Exit-block terminology (NFC).In LLVM's common loop terminology, an exit block is a block outside aloop with a predecessor inside the loop. An exiting block is a blockinside the loop which branches to an exit block outside the loop.This patch updates a few places where VPlan was using ExitBlock for ablock exiting a region. Those instances have been updated to useExitingBlock.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D126173 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Sat, 28 May 2022 20:16:05 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#1ba42dd0 [VPlan] Use MapVector for LiveOuts for deterministic iteration.During code-gen, we iterate over the LiveOuts and the differences initeration order can cause slightly different outputs. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Wed, 25 May 2022 08:29:52 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#97590bae [LV] Widen ptr-inductions with scalar uses for scalable VFs.Current codegen only supports scalarization of pointer inductions forscalable VFs if they are uniform. After 3bebec659 we now may enter thescalarization code path in VPWidenPointerInductionRecipe::execute forscalable vectors.Fall back to widening for scalable vectors if necessary.This should fix a build failure when bootstrapping LLVM with SVE, e.g.https://lab.llvm.org/buildbot/#/builders/176/builds/1723 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Sun, 22 May 2022 15:24:13 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#3bebec65 [VPlan] Model first exit values using VPLiveOut.This patch introduces a new VPLiveOut subclass of VPUser to model exit values explicitly. The initial version handles exit values thatare neither part of induction or reduction chains nor first orderrecurrence phis.Fixes #51366, #54867, #55167, #55459Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D123537 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Sat, 21 May 2022 15:01:38 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#df56fb44 [VPlan] Update VPWidenMemoryInstruction to not inherit from VPValue.VPWidenMemoryInstruction also models stores which may not produce a value.This can trip over analyses. Improve the modeling by only addingVPValues for VPWidenMemoryInstructionRecipes modeling loads. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Thu, 19 May 2022 15:24:38 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#c1a9d149 [VPlan] Move usesScalars/onlyFirstLaneUsed to VPUser.Those helpers model properties of a user and they should also beavailable to non-recipe users. This will be used in D123537 for a newexit value user.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D124936 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Tue, 17 May 2022 10:20:06 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#635b7522 [VPlan] VPInterleaveRecipe only uses first lane if op not stored.With opaque pointers, both the stored value and the address can be thesame. Only consider the recipe using the first lane only *if* theaddress is not stored.Fixes #55375. List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Wed, 11 May 2022 10:24:56 +0000 Florian Hahn <flo@fhahn.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h#a65f2730 [VPlan] Expand induction step in VPlan pre-header.This patch moves SCEV expansion of steps used byVPWidenIntOrFpInductionRecipes to the pre-header usingVPExpandSCEVRecipe. This ensures that those steps are expanded while theCFG is in a valid state. Previously, SCEV expansion may happen duringvector body code-generation, during which the CFG may be invalid,causing issues with SCEV expansion.Depends on D122095.Reviewed By: AyalDifferential Revision: https://reviews.llvm.org/D122096 List of files: /llvm-project-15.0.7/llvm/lib/Transforms/Vectorize/VPlan.h Tue, 19 Apr 2022 11:06:39 +0000 Florian Hahn <flo@fhahn.com>