en Copyright 2015 Java http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#78eaff2e [llvm-exegesis] Loop unrolling for loop snippet repetitor modeI really needed this, like, factually, yesterday,when verifying dependency breaking idioms for AMD Zen 3 scheduler model.Consider the following example:```$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=duplicateCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-4a7e50.o---mode: inverse_throughputkey: instructions: - 'VPXORYrr YMM0 YMM0 YMM0' config: '' register_initial_values: []cpu_name: znver3llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 1000000measurements: - { key: inverse_throughput, value: 0.31025, per_snippet_value: 0.31025 }error: ''info: ''assembled_snippet: C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C5FDEFC0C3...```What does it tell us?So wait, it can only execute ~3 x86 AVX YMM PXOR zero-idioms per cycle?That doesn't seem right. That's even less than there are pipes supporting this type of op.Now, second example:```$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=loopCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-2418b5.o---mode: inverse_throughputkey: instructions: - 'VPXORYrr YMM0 YMM0 YMM0' config: '' register_initial_values: []cpu_name: znver3llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 1000000measurements: - { key: inverse_throughput, value: 1.00011, per_snippet_value: 1.00011 }error: ''info: ''assembled_snippet: 49B80800000000000000C5FDEFC0C5FDEFC04983C0FF75F2C3...```Now that's just worse. Due to the looping, the throughput completely plummeted,and now we can only do a single instruction/cycle!?That's not great.And final example:```$ ./bin/llvm-exegesis --mode=inverse_throughput --snippets-file=/tmp/snippet.s --num-repetitions=1000000 --repetition-mode=loop --loop-body-size=1000Check generated assembly with: /usr/bin/objdump -d /tmp/snippet-c402e2.o---mode: inverse_throughputkey: instructions: - 'VPXORYrr YMM0 YMM0 YMM0' config: '' register_initial_values: []cpu_name: znver3llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 1000000measurements: - { key: inverse_throughput, value: 0.167087, per_snippet_value: 0.167087 }error: ''info: ''assembled_snippet: 49B80800000000000000C5FDEFC0C5FDEFC04983C0FF75F2C3...```So if we merge the previous two approaches, do duplicate this single-instruction snippet 1000x(loop-body-size/instruction count in snippet), and run a loop with 1000 iterationsover that duplicated/unrolled snippet, the measured throughput goes through the roof,up to 5.9 instructions/cycle, which finally tells us that this idiom is zero-cycle!Reviewed By: courbetDifferential Revision: https://reviews.llvm.org/D102522 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Tue, 25 May 2021 08:48:43 +0000 Roman Lebedev <lebedev.ri@gmail.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#de22d715 [llvm-exegesis] 'Min' repetition modeSummary:As noted in documentation, different repetition modes have different trade-offs:> .. option:: -repetition-mode=[duplicate|loop]>> Specify the repetition mode. `duplicate` will create a large, straight line> basic block with `num-repetitions` copies of the snippet. `loop` will wrap> the snippet in a loop which will be run `num-repetitions` times. The `loop`> mode tends to better hide the effects of the CPU frontend on architectures> that cache decoded instructions, but consumes a register for counting> iterations.Indeed. Example:>>! In D74156#1873657, @lebedev.ri wrote:> At least for `CMOV`, i'm seeing wildly different results> | | Latency | RThroughput |> | duplicate | 1 | 0.8 |> | loop | 2 | 0.6 |> where latency=1 seems correct, and i'd expect the througput to be close to 1/2 (since there are two execution units).This isn't great for analysis, at least for schedule model development.As discussed in excruciating detail in>>! In D74156#1924514, @gchatelet wrote:>>>! In D74156#1920632, @lebedev.ri wrote:>> ... did that explanation of the question i'm having made any sense?>> Thx for digging in the conversation !> Ok it makes more sense now.>> I discussed it a bit with @courbet:> - We want the analysis tool to stay simple so we'd rather not make it knowledgeable of the repetition mode.> - We'd like to still be able to select either repetition mode to dig into special cases>> So we could add a third `min` repetition mode that would run both and take the minimum. It could be the default option.> Would you have some time to look what it would take to add this third mode?there appears to be an agreement that it is indeed sub-par,and that we should provide an optional, measurement (not analysis!) -timeway to rectify the situation.However, the solutions isn't entirely straight-forward.We can just add an actual 'multiplexer' `MinSnippetRepetitor`, becauseif we just concatenate snippets produced by `DuplicateSnippetRepetitor`and `LoopSnippetRepetitor` and run+measure that, the measurement willnaturally be different from what we'd get by running+measuringthem separately and taking the min.([[ https://www.wolframalpha.com/input/?i=%28x%2By%29%2F2+%21%3D+min%28x%2C+y%29 | `time(D+L)/2 != min(time(D), time(L))` ]])Also, it seems best to me to have a single snippet instead of generatinga snippet per repetition mode, since the only difference here is that theloop repetition mode reserves one register for loop counter.As far as i can tell, we can either teach `BenchmarkRunner::runConfiguration()`to produce a single report given multiple repetitors (as in the patch),or do that one layer higher - don't modify `BenchmarkRunner::runConfiguration()`,produce multiple reports, don't actually print each one, but aggregate them somehowand only print the final one.Initially i've gone ahead with the latter approach, but it didn't look like a natural fit;the former (as in the diff) does seem like a better fit to me.There's also a question of the test coverage. It sure currently does work here:```$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=duplicateCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-8fb949.o---mode: inverse_throughputkey: instructions: - 'CMOV64rr RAX RAX R11 i_0x0' - 'CMOV64rr RBP RBP R15 i_0x0' - 'CMOV64rr RBX RBX RBX i_0x0' - 'CMOV64rr RCX RCX RBX i_0x0' - 'CMOV64rr RDI RDI R10 i_0x0' - 'CMOV64rr RDX RDX RAX i_0x0' - 'CMOV64rr RSI RSI RAX i_0x0' - 'CMOV64rr R8 R8 R8 i_0x0' - 'CMOV64rr R9 R9 RDX i_0x0' - 'CMOV64rr R10 R10 RBX i_0x0' - 'CMOV64rr R11 R11 R14 i_0x0' - 'CMOV64rr R12 R12 R9 i_0x0' - 'CMOV64rr R13 R13 R12 i_0x0' - 'CMOV64rr R14 R14 R15 i_0x0' - 'CMOV64rr R15 R15 R13 i_0x0' config: '' register_initial_values: - 'RAX=0x0' - 'R11=0x0' - 'EFLAGS=0x0' - 'RBP=0x0' - 'R15=0x0' - 'RBX=0x0' - 'RCX=0x0' - 'RDI=0x0' - 'R10=0x0' - 'RDX=0x0' - 'RSI=0x0' - 'R8=0x0' - 'R9=0x0' - 'R14=0x0' - 'R12=0x0' - 'R13=0x0'cpu_name: bdver2llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 10000measurements: - { key: inverse_throughput, value: 0.819, per_snippet_value: 12.285 }error: ''info: instruction has tied variables, using static renaming.assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BF000000000000000048BB000000000000000048B9000000000000000048BF000000000000000049BA000000000000000048BA000000000000000048BE000000000000000049B8000000000000000049B9000000000000000049BE000000000000000049BC000000000000000049BD0000000000000000490F40C3490F40EF480F40DB480F40CB490F40FA480F40D0480F40F04D0F40C04C0F40CA4C0F40D34D0F40DE4D0F40E14D0F40EC4D0F40F74D0F40FD490F40C35B415C415D415E415F5DC3...$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=loopCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-051eb3.o---mode: inverse_throughputkey: instructions: - 'CMOV64rr RAX RAX R11 i_0x0' - 'CMOV64rr RBP RBP RSI i_0x0' - 'CMOV64rr RBX RBX R9 i_0x0' - 'CMOV64rr RCX RCX RSI i_0x0' - 'CMOV64rr RDI RDI RBP i_0x0' - 'CMOV64rr RDX RDX R9 i_0x0' - 'CMOV64rr RSI RSI RDI i_0x0' - 'CMOV64rr R9 R9 R12 i_0x0' - 'CMOV64rr R10 R10 R11 i_0x0' - 'CMOV64rr R11 R11 R9 i_0x0' - 'CMOV64rr R12 R12 RBP i_0x0' - 'CMOV64rr R13 R13 RSI i_0x0' - 'CMOV64rr R14 R14 R14 i_0x0' - 'CMOV64rr R15 R15 R10 i_0x0' config: '' register_initial_values: - 'RAX=0x0' - 'R11=0x0' - 'EFLAGS=0x0' - 'RBP=0x0' - 'RSI=0x0' - 'RBX=0x0' - 'R9=0x0' - 'RCX=0x0' - 'RDI=0x0' - 'RDX=0x0' - 'R12=0x0' - 'R10=0x0' - 'R13=0x0' - 'R14=0x0' - 'R15=0x0'cpu_name: bdver2llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 10000measurements: - { key: inverse_throughput, value: 0.6083, per_snippet_value: 8.5162 }error: ''info: instruction has tied variables, using static renaming.assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000048BE000000000000000048BB000000000000000049B9000000000000000048B9000000000000000048BF000000000000000048BA000000000000000049BC000000000000000049BA000000000000000049BD000000000000000049BE000000000000000049BF000000000000000049B80200000000000000490F40C3480F40EE490F40D9480F40CE480F40FD490F40D1480F40F74D0F40CC4D0F40D34D0F40D94C0F40E54C0F40EE4D0F40F64D0F40FA4983C0FF75C25B415C415D415E415F5DC3...$ ./bin/llvm-exegesis --opcode-name=CMOV64rr --mode=inverse_throughput --repetition-mode=minCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-c7a47d.oCheck generated assembly with: /usr/bin/objdump -d /tmp/snippet-2581f1.o---mode: inverse_throughputkey: instructions: - 'CMOV64rr RAX RAX R11 i_0x0' - 'CMOV64rr RBP RBP R10 i_0x0' - 'CMOV64rr RBX RBX R10 i_0x0' - 'CMOV64rr RCX RCX RDX i_0x0' - 'CMOV64rr RDI RDI RAX i_0x0' - 'CMOV64rr RDX RDX R9 i_0x0' - 'CMOV64rr RSI RSI RAX i_0x0' - 'CMOV64rr R9 R9 RBX i_0x0' - 'CMOV64rr R10 R10 R12 i_0x0' - 'CMOV64rr R11 R11 RDI i_0x0' - 'CMOV64rr R12 R12 RDI i_0x0' - 'CMOV64rr R13 R13 RDI i_0x0' - 'CMOV64rr R14 R14 R9 i_0x0' - 'CMOV64rr R15 R15 RBP i_0x0' config: '' register_initial_values: - 'RAX=0x0' - 'R11=0x0' - 'EFLAGS=0x0' - 'RBP=0x0' - 'R10=0x0' - 'RBX=0x0' - 'RCX=0x0' - 'RDX=0x0' - 'RDI=0x0' - 'R9=0x0' - 'RSI=0x0' - 'R12=0x0' - 'R13=0x0' - 'R14=0x0' - 'R15=0x0'cpu_name: bdver2llvm_triple: x86_64-unknown-linux-gnunum_repetitions: 10000measurements: - { key: inverse_throughput, value: 0.6073, per_snippet_value: 8.5022 }error: ''info: instruction has tied variables, using static renaming.assembled_snippet: 5541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BA000000000000000048BB000000000000000048B9000000000000000048BA000000000000000048BF000000000000000049B9000000000000000048BE000000000000000049BC000000000000000049BD000000000000000049BE000000000000000049BF0000000000000000490F40C3490F40EA490F40DA480F40CA480F40F8490F40D1480F40F04C0F40CB4D0F40D44C0F40DF4C0F40E74C0F40EF4D0F40F14C0F40FD490F40C3490F40EA5B415C415D415E415F5DC35541574156415541545348B8000000000000000049BB00000000000000004883EC08C7042400000000C7442404000000009D48BD000000000000000049BA000000000000000048BB000000000000000048B9000000000000000048BA000000000000000048BF000000000000000049B9000000000000000048BE000000000000000049BC000000000000000049BD000000000000000049BE000000000000000049BF000000000000000049B80200000000000000490F40C3490F40EA490F40DA480F40CA480F40F8490F40D1480F40F04C0F40CB4D0F40D44C0F40DF4C0F40E74C0F40EF4D0F40F14C0F40FD4983C0FF75C25B415C415D415E415F5DC3...```but i open to suggestions as to how test that.I also have gone with the suggestion to default to this new mode.This was irking me for some time, so i'm happy to finally see progress here.Looking forward to feedback.Reviewers: courbet, gchateletReviewed By: courbet, gchateletSubscribers: mstojanovic, RKSimon, llvm-commits, courbet, gchateletTags: #llvmDifferential Revision: https://reviews.llvm.org/D76921 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Thu, 02 Apr 2020 06:28:35 +0000 Roman Lebedev <lebedev.ri@gmail.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#50cdd56b [llvm-exegesis][NFC] Remove extra `llvm::` qualifications.Summary: Second patch: in the lib.Reviewers: gchateletSubscribers: nemanjai, tschuett, MaskRay, mgrang, jsji, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D68692llvm-svn: 374158 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Wed, 09 Oct 2019 11:58:42 +0000 Clement Courbet <courbet@google.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#c0292744 [llvm-exegesis][NFC] Rename ExegesisTarget::decrementLoopCounterAndLoop()Summary: To decrementLoopCounterAndJump, and explicitely take the jump target.Reviewers: gchateletSubscribers: tschuett, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D68375llvm-svn: 373571 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Thu, 03 Oct 2019 07:56:56 +0000 Clement Courbet <courbet@google.com> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#1a55431a Fix MSVC "not all control paths return a value" warning. NFCI.llvm-svn: 373100 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Fri, 27 Sep 2019 16:56:07 +0000 Simon Pilgrim <llvm-dev@redking.me.uk> http://172.16.0.5:8080/history/llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp#9431b72c [llvm-exegesis] Add loop mode for repeating the snippet.Summary:Before this change the Executable function was made by duplicating thesnippet. This change adds a --repetion-mode={loop|duplicate} flag thatallows choosing between this behaviour and wrapping the snippet instructionsin a loop.The new mode can help measurements when the snippet fits in the DSB byshort-cirtcuiting decoding. The loop adds a dec + jmp to the measurements, butsince these are not part of the critical path, they execute in parallelwith the measured code and do not impact measurements in practice.Overview of the change: - New SnippetRepetitor abstraction that handles repeating the snippet. The assembler delegates repeating the instructions to this class. - ExegesisTarget learns how to decrement loop counter and jump. - Some refactoring of the assembler into FunctionFiller/BasicBlockFiller.Reviewers: gchateletSubscribers: mgorny, tschuett, llvm-commitsTags: #llvmDifferential Revision: https://reviews.llvm.org/D68125llvm-svn: 373083 List of files: /llvm-project-15.0.7/llvm/tools/llvm-exegesis/lib/SnippetRepetitor.cpp Fri, 27 Sep 2019 12:56:24 +0000 Clement Courbet <courbet@google.com>