Changes in Makefile

8c4fc9ce - crypto: x86/aes-ctr - rewrite AESNI+AVX optimized CTR and add VAES support

Mon, 10 Feb 2025 16:50:20 +0000

crypto: x86/aes-ctr - rewrite AESNI+AVX optimized CTR and add VAES supportDelete aes_ctrby8_avx-x86_64.S and add a new assembly fileaes-ctr-avx-x86_64.S which follows a similar approach toaes-xts-avx-x86_64.S in that it uses a "template" to provide AESNI+AVX,VAES+AVX2, VAES+AVX10/256, and VAES+AVX10/512 code, instead of justAESNI+AVX. Wire it up to the crypto API accordingly.This greatly improves the performance of AES-CTR and AES-XCTR onVAES-capable CPUs, with the best case being AMD Zen 5 where an over 230%increase in throughput is seen on long messages. Performance onnon-VAES-capable CPUs remains about the same, and the non-AVX AES-CTRcode (aesni_ctr_enc) is also kept as-is for now. There are some slightregressions (less than 10%) on some short message lengths on some CPUs;these are difficult to avoid, given how the previous code was so heavilyunrolled by message length, and they are not particularly important.Detailed performance results are given in the tables below.Both CTR and XCTR support is retained. The main loop remains8-vector-wide, which differs from the 4-vector-wide main loops that areused in the XTS and GCM code. A wider loop is appropriate for CTR andXCTR since they have fewer other instructions (such as vpclmulqdq) tointerleave with the AES instructions.Similar to what was the case for AES-GCM, the new assembly code also hasa much smaller binary size, as it fixes the excessive unrolling by datalength and key length present in the old code. Specifically, the newassembly file compiles to about 9 KB of text vs. 28 KB for the old file.This is despite 4x as many implementations being included.The tables below show the detailed performance results. The tables showpercentage improvement in single-threaded throughput for repeatedencryption of the given message length; an increase from 6000 MB/s to12000 MB/s would be listed as 100%. They were collected by directlymeasuring the Linux crypto API performance using a custom kernel module.The tested CPUs were all server processors from Google Compute Engineexcept for Zen 5 which was a Ryzen 9 9950X desktop processor.Table 1: AES-256-CTR throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |---------------------+-------+-------+-------+-------+-------+-------+AMD Zen 5 | 232% | 203% | 212% | 143% | 71% | 95% |Intel Emerald Rapids | 116% | 116% | 117% | 91% | 78% | 79% |Intel Ice Lake | 109% | 103% | 107% | 81% | 54% | 56% |AMD Zen 4 | 109% | 91% | 100% | 70% | 43% | 59% |AMD Zen 3 | 92% | 78% | 87% | 57% | 32% | 43% |AMD Zen 2 | 9% | 8% | 14% | 12% | 8% | 21% |Intel Skylake | 7% | 7% | 8% | 5% | 3% | 8% | | 300 | 200 | 64 | 63 | 16 |---------------------+-------+-------+-------+-------+-------+AMD Zen 5 | 57% | 39% | -9% | 7% | -7% |Intel Emerald Rapids | 37% | 42% | -0% | 13% | -8% |Intel Ice Lake | 39% | 30% | -1% | 14% | -9% |AMD Zen 4 | 42% | 38% | -0% | 18% | -3% |AMD Zen 3 | 38% | 35% | 6% | 31% | 5% |AMD Zen 2 | 24% | 23% | 5% | 30% | 3% |Intel Skylake | 9% | 1% | -4% | 10% | -7% |Table 2: AES-256-XCTR throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |---------------------+-------+-------+-------+-------+-------+-------+AMD Zen 5 | 240% | 201% | 216% | 151% | 75% | 108% |Intel Emerald Rapids | 100% | 99% | 102% | 91% | 94% | 104% |Intel Ice Lake | 93% | 89% | 92% | 74% | 50% | 64% |AMD Zen 4 | 86% | 75% | 83% | 60% | 41% | 52% |AMD Zen 3 | 73% | 63% | 69% | 45% | 21% | 33% |AMD Zen 2 | -2% | -2% | 2% | 3% | -1% | 11% |Intel Skylake | -1% | -1% | 1% | 2% | -1% | 9% | | 300 | 200 | 64 | 63 | 16 |---------------------+-------+-------+-------+-------+-------+AMD Zen 5 | 78% | 56% | -4% | 38% | -2% |Intel Emerald Rapids | 61% | 55% | 4% | 32% | -5% |Intel Ice Lake | 57% | 42% | 3% | 44% | -4% |AMD Zen 4 | 35% | 28% | -1% | 17% | -3% |AMD Zen 3 | 26% | 23% | -3% | 11% | -6% |AMD Zen 2 | 13% | 24% | -1% | 14% | -3% |Intel Skylake | 16% | 8% | -4% | 35% | -3% |Signed-off-by: Eric Biggers Reviewed-by: Ard Biesheuvel Tested-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

ed4bc981 - x86/crc-t10dif: expose CRC-T10DIF function through lib

Mon, 02 Dec 2024 01:20:48 +0000

x86/crc-t10dif: expose CRC-T10DIF function through libMove the x86 CRC-T10DIF assembly code into the lib directory and wire itup to the library interface. This allows it to be used without goingthrough the crypto API. It remains usable via the crypto API too viathe shash algorithms that use the library interface. Thus all thearch-specific "shash" code becomes unnecessary and is removed.Reviewed-by: Ard Biesheuvel Reviewed-by: Martin K. Petersen Link: https://lore.kernel.org/r/20241202012056.209768-5-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux-6.15/arch/x86/crypto/Makefile

55d1ecce - x86/crc32: expose CRC32 functions through lib

Mon, 02 Dec 2024 01:08:38 +0000

x86/crc32: expose CRC32 functions through libMove the x86 CRC32 assembly code into the lib directory and wire it upto the library interface. This allows it to be used without goingthrough the crypto API. It remains usable via the crypto API too viathe shash algorithms that use the library interface. Thus all thearch-specific "shash" code becomes unnecessary and is removed.Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20241202010844.144356-14-ebiggers@kernel.orgSigned-off-by: Eric Biggers List of files: /linux-6.15/arch/x86/crypto/Makefile

e6e758fa - crypto: x86/aes-gcm - rewrite the AES-NI optimized AES-GCM

Sun, 02 Jun 2024 22:22:20 +0000

crypto: x86/aes-gcm - rewrite the AES-NI optimized AES-GCMRewrite the AES-NI implementations of AES-GCM, taking advantage ofthings I learned while writing the VAES-AVX10 implementations. This isa complete rewrite that reduces the AES-NI GCM source code size by about70% and the binary code size by about 95%, while not regressingperformance and in fact improving it significantly in many cases.The following summarizes the state before this patch:- The aesni-intel module registered algorithms "generic-gcm-aesni" and "rfc4106-gcm-aesni" with the crypto API that actually delegated to one of three underlying implementations according to the CPU capabilities detected at runtime: AES-NI, AES-NI + AVX, or AES-NI + AVX2.- The AES-NI + AVX and AES-NI + AVX2 assembly code was in aesni-intel_avx-x86_64.S and consisted of 2804 lines of source and 257 KB of binary. This massive binary size was not really appropriate, and depending on the kconfig it could take up over 1% the size of the entire vmlinux. The main loops did 8 blocks per iteration. The AVX code minimized the use of carryless multiplication whereas the AVX2 code did not. The "AVX2" code did not actually use AVX2; the check for AVX2 was really a check for Intel Haswell or later to detect support for fast carryless multiplication. The long source length was caused by factors such as significant code duplication.- The AES-NI only assembly code was in aesni-intel_asm.S and consisted of 1501 lines of source and 15 KB of binary. The main loops did 4 blocks per iteration and minimized the use of carryless multiplication by using Karatsuba multiplication and a multiplication-less reduction.- The assembly code was contributed in 2010-2013. Maintenance has been sporadic and most design choices haven't been revisited.- The assembly function prototypes and the corresponding glue code were separate from and were not consistent with the new VAES-AVX10 code I recently added. The older code had several issues such as not precomputing the GHASH key powers, which hurt performance.This rewrite achieves the following goals:- Much shorter source and binary sizes. The assembly source shrinks from 4300 lines to 1130 lines, and it produces about 9 KB of binary instead of 272 KB. This is achieved via a better designed AES-GCM implementation that doesn't excessively unroll the code and instead prioritizes the parts that really matter. Sharing the C glue code with the VAES-AVX10 implementations also saves 250 lines of C source.- Improve performance on most (possibly all) CPUs on which this code runs, for most (possibly all) message lengths. Benchmark results are given in Tables 1 and 2 below.- Use the same function prototypes and glue code as the new VAES-AVX10 algorithms. This fixes some issues with the integration of the assembly and results in some significant performance improvements, primarily on short messages. Also, the AVX and non-AVX implementations are now registered as separate algorithms with the crypto API, which makes them both testable by the self-tests.- Keep support for AES-NI without AVX (for Westmere, Silvermont, Goldmont, and Tremont), but unify the source code with AES-NI + AVX. Since 256-bit vectors cannot be used without VAES anyway, this is made feasible by just using the non-VEX coded form of most instructions.- Use a unified approach where the main loop does 8 blocks per iteration and uses Karatsuba multiplication to save one pclmulqdq per block but does not use the multiplication-less reduction. This strikes a good balance across the range of CPUs on which this code runs.- Don't spam the kernel log with an informational message on every boot.The following tables summarize the improvement in AES-GCM throughput onvarious CPU microarchitectures as a result of this patch:Table 1: AES-256-GCM encryption throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |-------------------+-------+-------+-------+-------+-------+-------+Intel Broadwell | 2% | 8% | 11% | 18% | 31% | 26% |Intel Skylake | 1% | 4% | 7% | 12% | 26% | 19% |Intel Cascade Lake | 3% | 8% | 10% | 18% | 33% | 24% |AMD Zen 1 | 6% | 12% | 6% | 15% | 27% | 24% |AMD Zen 2 | 8% | 13% | 13% | 19% | 26% | 28% |AMD Zen 3 | 8% | 14% | 13% | 19% | 26% | 25% | | 300 | 200 | 64 | 63 | 16 |-------------------+-------+-------+-------+-------+-------+Intel Broadwell | 35% | 29% | 45% | 55% | 54% |Intel Skylake | 25% | 19% | 28% | 33% | 27% |Intel Cascade Lake | 36% | 28% | 39% | 49% | 54% |AMD Zen 1 | 27% | 22% | 23% | 29% | 26% |AMD Zen 2 | 32% | 24% | 22% | 25% | 31% |AMD Zen 3 | 30% | 24% | 22% | 23% | 26% |Table 2: AES-256-GCM decryption throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |-------------------+-------+-------+-------+-------+-------+-------+Intel Broadwell | 3% | 8% | 11% | 19% | 32% | 28% |Intel Skylake | 3% | 4% | 7% | 13% | 28% | 27% |Intel Cascade Lake | 3% | 9% | 11% | 19% | 33% | 28% |AMD Zen 1 | 15% | 18% | 14% | 20% | 36% | 33% |AMD Zen 2 | 9% | 16% | 13% | 21% | 26% | 27% |AMD Zen 3 | 8% | 15% | 12% | 18% | 23% | 23% | | 300 | 200 | 64 | 63 | 16 |-------------------+-------+-------+-------+-------+-------+Intel Broadwell | 36% | 31% | 40% | 51% | 53% |Intel Skylake | 28% | 21% | 23% | 30% | 30% |Intel Cascade Lake | 36% | 29% | 36% | 47% | 53% |AMD Zen 1 | 35% | 31% | 32% | 35% | 36% |AMD Zen 2 | 31% | 30% | 27% | 38% | 30% |AMD Zen 3 | 27% | 23% | 24% | 32% | 26% |The above numbers are percentage improvements in single-threadthroughput, so e.g. an increase from 3000 MB/s to 3300 MB/s would belisted as 10%. They were collected by directly measuring the Linuxcrypto API performance using a custom kernel module. Note that indirectbenchmarks (e.g. 'cryptsetup benchmark' or benchmarking dm-crypt I/O)include more overhead and won't see quite as much of a difference. Allthese benchmarks used an associated data length of 16 bytes. Note thatAES-GCM is almost always used with short associated data lengths.I didn't test Intel CPUs before Broadwell, AMD CPUs before Zen 1, orIntel low-power CPUs, as these weren't readily available to me.However, based on the design of the new code and the availableinformation about these other CPU microarchitectures, I wouldn't expectany significant regressions, and there's a good chance performance isimproved just as it is above.Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

b06affb1 - crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM

Sun, 02 Jun 2024 22:22:19 +0000

crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCMAdd implementations of AES-GCM for x86_64 CPUs that support VAES (vectorAES), VPCLMULQDQ (vector carryless multiplication), and either AVX512 orAVX10. There are two implementations, sharing most source code: oneusing 256-bit vectors and one using 512-bit vectors. This patchimproves AES-GCM performance by up to 162%; see Tables 1 and 2 below.I wrote the new AES-GCM assembly code from scratch, focusing oncorrectness, performance, code size (both source and binary), anddocumenting the source. The new assembly file aes-gcm-avx10-x86_64.S isabout 1200 lines including extensive comments, and it generates lessthan 8 KB of binary code. The main loop does 4 vectors at a time, withthe AES and GHASH instructions interleaved. Any remainder is handledusing a simple 1 vector at a time loop, with masking.Several VAES + AVX512 implementations of AES-GCM exist from Intel,including one in OpenSSL and one proposed for inclusion in Linux in 2021(https://lore.kernel.org/linux-crypto/1611386920-28579-6-git-send-email-megha.dey@intel.com/).These aren't really suitable to be used, though, due to the massiveamount of binary code generated (696 KB for OpenSSL, 200 KB for Linux)and well as the significantly larger amount of assembly source (4978lines for OpenSSL, 1788 lines for Linux). Also, Intel's code does notsupport 256-bit vectors, which makes it not usable on futureAVX10/256-only CPUs, and also not ideal for certain Intel CPUs that havedownclocking issues. So I ended up starting from scratch. Usually mymuch shorter code is actually slightly faster than Intel's AVX512 code,though it depends on message length and on which of Intel'simplementations is used; for details, see Tables 3 and 4 below.To facilitate potential integration into other projects, I'vedual-licensed aes-gcm-avx10-x86_64.S under Apache-2.0 OR BSD-2-Clause,the same as the recently added RISC-V crypto code.The following two tables summarize the performance improvement over theexisting AES-GCM code in Linux that uses AES-NI and AVX2:Table 1: AES-256-GCM encryption throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |----------------------+-------+-------+-------+-------+-------+-------+Intel Ice Lake | 42% | 48% | 60% | 62% | 70% | 69% |Intel Sapphire Rapids | 157% | 145% | 162% | 119% | 96% | 96% |Intel Emerald Rapids | 156% | 144% | 161% | 115% | 95% | 100% |AMD Zen 4 | 103% | 89% | 78% | 56% | 54% | 54% | | 300 | 200 | 64 | 63 | 16 |----------------------+-------+-------+-------+-------+-------+Intel Ice Lake | 66% | 48% | 49% | 70% | 53% |Intel Sapphire Rapids | 80% | 60% | 41% | 62% | 38% |Intel Emerald Rapids | 79% | 60% | 41% | 62% | 38% |AMD Zen 4 | 51% | 35% | 27% | 32% | 25% |Table 2: AES-256-GCM decryption throughput improvement, CPU microarchitecture vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |----------------------+-------+-------+-------+-------+-------+-------+Intel Ice Lake | 42% | 48% | 59% | 63% | 67% | 71% |Intel Sapphire Rapids | 159% | 145% | 161% | 125% | 102% | 100% |Intel Emerald Rapids | 158% | 144% | 161% | 124% | 100% | 103% |AMD Zen 4 | 110% | 95% | 80% | 59% | 56% | 54% | | 300 | 200 | 64 | 63 | 16 |----------------------+-------+-------+-------+-------+-------+Intel Ice Lake | 67% | 56% | 46% | 70% | 56% |Intel Sapphire Rapids | 79% | 62% | 39% | 61% | 39% |Intel Emerald Rapids | 80% | 62% | 40% | 58% | 40% |AMD Zen 4 | 49% | 36% | 30% | 35% | 28% |The above numbers are percentage improvements in single-threadthroughput, so e.g. an increase from 4000 MB/s to 6000 MB/s would belisted as 50%. They were collected by directly measuring the Linuxcrypto API performance using a custom kernel module. Note that indirectbenchmarks (e.g. 'cryptsetup benchmark' or benchmarking dm-crypt I/O)include more overhead and won't see quite as much of a difference. Allthese benchmarks used an associated data length of 16 bytes. Note thatAES-GCM is almost always used with short associated data lengths.The following two tables summarize how the performance of my codecompares with Intel's AVX512 AES-GCM code, both the version that is inOpenSSL and the version that was proposed for inclusion in Linux.Neither version exists in Linux currently, but these are alternativeAES-GCM implementations that could be chosen instead of mine. Icollected the following numbers on Emerald Rapids using a userspacebenchmark program that calls the assembly functions directly.I've also included a comparison with Cloudflare's AES-GCM implementationfrom https://boringssl-review.googlesource.com/c/boringssl/+/65987/3.Table 3: VAES-based AES-256-GCM encryption throughput in MB/s, implementation name vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |---------------------+-------+-------+-------+-------+-------+-------+This implementation | 14171 | 12956 | 12318 | 9588 | 7293 | 6449 |AVX512_Intel_OpenSSL | 14022 | 12467 | 11863 | 9107 | 5891 | 6472 |AVX512_Intel_Linux | 13954 | 12277 | 11530 | 8712 | 6627 | 5898 |AVX512_Cloudflare | 12564 | 11050 | 10905 | 8152 | 5345 | 5202 | | 300 | 200 | 64 | 63 | 16 |---------------------+-------+-------+-------+-------+-------+This implementation | 4939 | 3688 | 1846 | 1821 | 738 |AVX512_Intel_OpenSSL | 4629 | 4532 | 2734 | 2332 | 1131 |AVX512_Intel_Linux | 4035 | 2966 | 1567 | 1330 | 639 |AVX512_Cloudflare | 3344 | 2485 | 1141 | 1127 | 456 |Table 4: VAES-based AES-256-GCM decryption throughput in MB/s, implementation name vs. message length in bytes: | 16384 | 4096 | 4095 | 1420 | 512 | 500 |---------------------+-------+-------+-------+-------+-------+-------+This implementation | 14276 | 13311 | 13007 | 11086 | 8268 | 8086 |AVX512_Intel_OpenSSL | 14067 | 12620 | 12421 | 9587 | 5954 | 7060 |AVX512_Intel_Linux | 14116 | 12795 | 11778 | 9269 | 7735 | 6455 |AVX512_Cloudflare | 13301 | 12018 | 11919 | 9182 | 7189 | 6726 | | 300 | 200 | 64 | 63 | 16 |---------------------+-------+-------+-------+-------+-------+This implementation | 6454 | 5020 | 2635 | 2602 | 1079 |AVX512_Intel_OpenSSL | 5184 | 5799 | 2957 | 2545 | 1228 |AVX512_Intel_Linux | 4394 | 4247 | 2235 | 1635 | 922 |AVX512_Cloudflare | 4289 | 3851 | 1435 | 1417 | 574 |So, usually my code is actually slightly faster than Intel's code,though the OpenSSL implementation has a slight edge on messages shorterthan 256 bytes in this microbenchmark. (This also holds true when doingthe same tests on AMD Zen 4.) It can be seen that the large code size(up to 94x larger!) of the Intel implementations doesn't seem to bringmuch benefit, so starting from scratch with much smaller code, as I'vedone, seems appropriate. The performance of my code on messages shorterthan 256 bytes could be improved through a limited amount of unrolling,but it's unclear it would be worth it, given code size considerations(e.g. caches) that don't get measured in microbenchmarks.Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

d6371688 - crypto: x86/aes-xts - add AES-XTS assembly macro for modern CPUs

Fri, 29 Mar 2024 08:03:50 +0000

crypto: x86/aes-xts - add AES-XTS assembly macro for modern CPUsAdd an assembly file aes-xts-avx-x86_64.S which contains a macro thatexpands into AES-XTS implementations for x86_64 CPUs that support atleast AES-NI and AVX, optionally also taking advantage of VAES,VPCLMULQDQ, and AVX512 or AVX10.This patch doesn't expand the macro at all. Later patches will do so,adding each implementation individually so that the motivation and usecase for each individual implementation can be fully presented.The file also provides a function aes_xts_encrypt_iv() which handles theencryption of the IV (tweak), using AES-NI and AVX.Signed-off-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

c970d420 - crypto: x86/aria - implement aria-avx512

Sun, 01 Jan 2023 09:12:52 +0000

crypto: x86/aria - implement aria-avx512aria-avx512 implementation uses AVX512 and GFNI.It supports 64way parallel processing.So, byteslicing code is changed to support 64way parallel.And it exports some aria-avx2 functions such as encrypt() and decrypt().AVX and AVX2 have 16 registers.They should use memory to store/load state because of lack of registers.But AVX512 supports 32 registers.So, it doesn't require store/load in the s-box layer.It means that it can reduce overhead of store/load in the s-box layer.Also code become much simpler.Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:ARIA-AVX512(128bit and 256bit) testing speed of multibuffer ecb(aria) (ecb-aria-avx512) encryptiontcrypt: 1 operation in 1504 cycles (1024 bytes)tcrypt: 1 operation in 4595 cycles (4096 bytes)tcrypt: 1 operation in 1763 cycles (1024 bytes)tcrypt: 1 operation in 5540 cycles (4096 bytes) testing speed of multibuffer ecb(aria) (ecb-aria-avx512) decryptiontcrypt: 1 operation in 1502 cycles (1024 bytes)tcrypt: 1 operation in 4615 cycles (4096 bytes)tcrypt: 1 operation in 1759 cycles (1024 bytes)tcrypt: 1 operation in 5554 cycles (4096 bytes)ARIA-AVX2 with GFNI(128bit and 256bit) testing speed of multibuffer ecb(aria) (ecb-aria-avx2) encryptiontcrypt: 1 operation in 2003 cycles (1024 bytes)tcrypt: 1 operation in 5867 cycles (4096 bytes)tcrypt: 1 operation in 2358 cycles (1024 bytes)tcrypt: 1 operation in 7295 cycles (4096 bytes) testing speed of multibuffer ecb(aria) (ecb-aria-avx2) decryptiontcrypt: 1 operation in 2004 cycles (1024 bytes)tcrypt: 1 operation in 5956 cycles (4096 bytes)tcrypt: 1 operation in 2409 cycles (1024 bytes)tcrypt: 1 operation in 7564 cycles (4096 bytes)Signed-off-by: Taehee Yoo Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

37d8d3ae - crypto: x86/aria - implement aria-avx2

Sun, 01 Jan 2023 09:12:51 +0000

crypto: x86/aria - implement aria-avx2aria-avx2 implementation uses AVX2, AES-NI, and GFNI.It supports 32way parallel processing.So, byteslicing code is changed to support 32way parallel.And it exports some aria-avx functions such as encrypt() and decrypt().There are two main logics, s-box layer and diffusion layer.These codes are the same as aria-avx implementation.But some instruction are exchanged because they don't support 256bitregisters.Also, AES-NI doesn't support 256bit register.So, aesenclast and aesdeclast are used twice like below: vextracti128 $1, ymm0, xmm6; vaesenclast xmm7, xmm0, xmm0; vaesenclast xmm7, xmm6, xmm6; vinserti128 $1, xmm6, ymm0, ymm0;Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:ARIA-AVX2 with GFNI(128bit and 256bit) testing speed of multibuffer ecb(aria) (ecb-aria-avx2) encryptiontcrypt: 1 operation in 2003 cycles (1024 bytes)tcrypt: 1 operation in 5867 cycles (4096 bytes)tcrypt: 1 operation in 2358 cycles (1024 bytes)tcrypt: 1 operation in 7295 cycles (4096 bytes) testing speed of multibuffer ecb(aria) (ecb-aria-avx2) decryptiontcrypt: 1 operation in 2004 cycles (1024 bytes)tcrypt: 1 operation in 5956 cycles (4096 bytes)tcrypt: 1 operation in 2409 cycles (1024 bytes)tcrypt: 1 operation in 7564 cycles (4096 bytes)ARIA-AVX with GFNI(128bit and 256bit) testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryptiontcrypt: 1 operation in 2761 cycles (1024 bytes)tcrypt: 1 operation in 9390 cycles (4096 bytes)tcrypt: 1 operation in 3401 cycles (1024 bytes)tcrypt: 1 operation in 11876 cycles (4096 bytes) testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryptiontcrypt: 1 operation in 2735 cycles (1024 bytes)tcrypt: 1 operation in 9424 cycles (4096 bytes)tcrypt: 1 operation in 3369 cycles (1024 bytes)tcrypt: 1 operation in 11954 cycles (4096 bytes)Signed-off-by: Taehee Yoo Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

c390c452 - crypto: x86/curve25519 - disable gcov

Tue, 22 Nov 2022 22:54:49 +0000

crypto: x86/curve25519 - disable gcovcurve25519-x86_64.c fails to build when CONFIG_GCOV_KERNEL is enabled.The error is "inline assembly requires more registers than available"thrown from the `fsqr()` function. Therefore, excluding this file fromGCOV profiling until this issue is resolved. Thereby allowingCONFIG_GCOV_PROFILE_ALL to be enabled for x86.Signed-off-by: Joe Fradley Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

ba3579e6 - crypto: aria-avx - add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipher

Fri, 16 Sep 2022 12:57:35 +0000

crypto: aria-avx - add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipherThe implementation is based on the 32-bit implementation of the aria.Also, aria-avx process steps are the similar to the camellia-avx.1. Byteslice(16way)2. Add-round-key.3. Sbox4. Diffusion layer.Except for s-box, all steps are the same as the aria-genericimplementation. s-box step is very similar to camellia andsm4 implementation.There are 2 implementations for s-box step.One is to use AES-NI and affine transformation, which is the same asCamellia, sm4, and others.Another is to use GFNI.GFNI implementation is faster than AES-NI implementation.So, it uses GFNI implementation if the running CPU supports GFNI.There are 4 s-boxes in the ARIA and the 2 s-boxes are the same asAES's s-boxes.To calculate the first sbox, it just uses the aesenclast and theninverts shift_row.No more process is needed for this job because the first s-box isthe same as the AES encryption s-box.To calculate the second sbox(invert of s1), it just uses the aesdeclastand then inverts shift_row.No more process is needed for this job because the second s-box isthe same as the AES decryption s-box.To calculate the third s-box, it uses the aesenclast,then affine transformation, which is combined AES inverse affine andARIA S2.To calculate the last s-box, it uses the aesdeclast,then affine transformation, which is combined X2 and AES forward affine.The optimized third and last s-box logic and GFNI s-box logic areimplemented by Jussi Kivilinna.The aria-generic implementation is based on a 32-bit implementation,not an 8-bit implementation. the aria-avx Diffusion Layer implementationis based on aria-generic implementation because 8-bit implementation isnot fit for parallel implementation but 32-bit is enough to fit for this.Signed-off-by: Taehee Yoo Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

2d16803c - crypto: blake2s - remove shash module

Sat, 28 May 2022 19:44:07 +0000

crypto: blake2s - remove shash moduleBLAKE2s has no currently known use as an shash. Just remove all of thisunnecessary plumbing. Removing this shash was something we talked aboutback when we were making BLAKE2s a built-in, but I simply never gotaround to doing it. So this completes that project.Importantly, this fixs a bug in which the lib code depends oncrypto_simd_disabled_for_test, causing linker errors.Also add more alignment tests to the selftests and compare SIMD andnon-SIMD compression functions, to make up for what we lose fromtestmgr.c.Reported-by: gaochao Cc: Eric Biggers Cc: Ard Biesheuvel Cc: stable@vger.kernel.orgFixes: 6048fdcc5f26 ("lib/crypto: blake2s: include as built-in")Signed-off-by: Jason A. Donenfeld Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

34f7f6c3 - crypto: x86/polyval - Add PCLMULQDQ accelerated implementation of POLYVAL

Fri, 20 May 2022 18:14:59 +0000

crypto: x86/polyval - Add PCLMULQDQ accelerated implementation of POLYVALAdd hardware accelerated version of POLYVAL for x86-64 CPUs withPCLMULQDQ support.This implementation is accelerated using PCLMULQDQ instructions toperform the finite field computations. For added efficiency, 8 blocksof the message are processed simultaneously by precomputing the first8 powers of the key.Schoolbook multiplication is used instead of Karatsuba multiplicationbecause it was found to be slightly faster on x86-64 machines.Montgomery reduction must be used instead of Barrett reduction due tothe difference in modulus between POLYVAL's field and other finitefields.More information on POLYVAL can be found in the HCTR2 paper:"Length-preserving encryption with HCTR2":https://eprint.iacr.org/2021/1441.pdfSigned-off-by: Nathan Huckleberry Reviewed-by: Ard Biesheuvel Reviewed-by: Eric Biggers Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

930ab34d - crypto: x86/sm3 - add AVX assembly implementation

Fri, 07 Jan 2022 12:06:59 +0000

crypto: x86/sm3 - add AVX assembly implementationThis patch adds AVX assembly accelerated implementation of SM3 securehash algorithm. From the benchmark data, compared to pure softwareimplementation sm3-generic, the performance increase is up to 38%.The main algorithm implementation based on SM3 AES/BMI2 acceleratedwork by libgcrypt at:https://gnupg.org/software/libgcrypt/index.htmlBenchmark on Intel i5-6200U 2.30GHz, performance data of twoimplementations, pure software sm3-generic and sm3-avx acceleration.The data comes from the 326 mode and 422 mode of tcrypt. The abscissasare different lengths of per update. The data is tabulated and theunit is Mb/s:update-size | 16 64 256 1024 2048 4096 8192------------+-------------------------------------------------------sm3-generic | 105.97 129.60 182.12 189.62 188.06 193.66 194.88sm3-avx | 119.87 163.05 244.44 260.92 257.60 264.87 265.88Signed-off-by: Tianjia Zhang Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

6048fdcc - lib/crypto: blake2s: include as built-in

Wed, 22 Dec 2021 13:56:58 +0000

lib/crypto: blake2s: include as built-inIn preparation for using blake2s in the RNG, we change the way that itis wired-in to the build system. Instead of using ifdefs to select theright symbol, we use weak symbols. And because ARM doesn't need thegeneric implementation, we make the generic one default only if an archlibrary doesn't need it already, and then have arch libraries that doneed it opt-in. So that the arch libraries can remain tristate ratherthan bool, we then split the shash part from the glue code.Acked-by: Herbert Xu Acked-by: Ard Biesheuvel Acked-by: Greg Kroah-Hartman Cc: Masahiro Yamada Cc: linux-kbuild@vger.kernel.orgCc: linux-crypto@vger.kernel.orgSigned-off-by: Jason A. Donenfeld List of files: /linux-6.15/arch/x86/crypto/Makefile

5b2efa2b - crypto: x86/sm4 - add AES-NI/AVX2/x86_64 implementation

Wed, 18 Aug 2021 03:31:17 +0000

crypto: x86/sm4 - add AES-NI/AVX2/x86_64 implementationLike the implementation of AESNI/AVX, this patch adds an acceleratedimplementation of AESNI/AVX2. In terms of code implementation, byreusing AESNI/AVX mode-related codes, the amount of code is greatlyreduced. From the benchmark data, it can be seen that when the blocksize is 1024, compared to AVX acceleration, the performance achievedby AVX2 has increased by about 70%, it is also 7.7 times of the puresoftware implementation of sm4-generic.The main algorithm implementation comes from SM4 AES-NI work bylibgcrypt and Markku-Juhani O. Saarinen at:https://github.com/mjosaarinen/sm4niThis optimization supports the four modes of SM4, ECB, CBC, CFB,and CTR. Since CBC and CFB do not support multiple block parallelencryption, the optimization effect is not obvious.Benchmark on Intel i5-6200U 2.30GHz, performance data of threeimplementation methods, pure software sm4-generic, aesni/avxacceleration, and aesni/avx2 acceleration, the data comes fromthe 218 mode and 518 mode of tcrypt. The abscissas are blocks ofdifferent lengths. The data is tabulated and the unit is Mb/s:block-size | 16 64 128 256 1024 1420 4096sm4-generic ECB enc | 60.94 70.41 72.27 73.02 73.87 73.58 73.59 ECB dec | 61.87 70.53 72.15 73.09 73.89 73.92 73.86 CBC enc | 56.71 66.31 68.05 69.84 70.02 70.12 70.24 CBC dec | 54.54 65.91 68.22 69.51 70.63 70.79 70.82 CFB enc | 57.21 67.24 69.10 70.25 70.73 70.52 71.42 CFB dec | 57.22 64.74 66.31 67.24 67.40 67.64 67.58 CTR enc | 59.47 68.64 69.91 71.02 71.86 71.61 71.95 CTR dec | 59.94 68.77 69.95 71.00 71.84 71.55 71.95sm4-aesni-avx ECB enc | 44.95 177.35 292.06 316.98 339.48 322.27 330.59 ECB dec | 45.28 178.66 292.31 317.52 339.59 322.52 331.16 CBC enc | 57.75 67.68 69.72 70.60 71.48 71.63 71.74 CBC dec | 44.32 176.83 284.32 307.24 328.61 312.61 325.82 CFB enc | 57.81 67.64 69.63 70.55 71.40 71.35 71.70 CFB dec | 43.14 167.78 282.03 307.20 328.35 318.24 325.95 CTR enc | 42.35 163.32 279.11 302.93 320.86 310.56 317.93 CTR dec | 42.39 162.81 278.49 302.37 321.11 310.33 318.37sm4-aesni-avx2 ECB enc | 45.19 177.41 292.42 316.12 339.90 322.53 330.54 ECB dec | 44.83 178.90 291.45 317.31 339.85 322.55 331.07 CBC enc | 57.66 67.62 69.73 70.55 71.58 71.66 71.77 CBC dec | 44.34 176.86 286.10 501.68 559.58 483.87 527.46 CFB enc | 57.43 67.60 69.61 70.52 71.43 71.28 71.65 CFB dec | 43.12 167.75 268.09 499.33 558.35 490.36 524.73 CTR enc | 42.42 163.39 256.17 493.95 552.45 481.58 517.19 CTR dec | 42.49 163.11 256.36 493.34 552.62 481.49 516.83Signed-off-by: Tianjia Zhang Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

a7ee22ee - crypto: x86/sm4 - add AES-NI/AVX/x86_64 implementation

Tue, 20 Jul 2021 03:46:41 +0000

crypto: x86/sm4 - add AES-NI/AVX/x86_64 implementationThis patch adds AES-NI/AVX/x86_64 assembler implementation of SM4block cipher. Through two affine transforms, we can use the AES S-Boxto simulate the SM4 S-Box to achieve the effect of instructionacceleration.The main algorithm implementation comes from SM4 AES-NI work bylibgcrypt and Markku-Juhani O. Saarinen at:https://github.com/mjosaarinen/sm4niThis optimization supports the four modes of SM4, ECB, CBC, CFB, andCTR. Since CBC and CFB do not support multiple block parallelencryption, the optimization effect is not obvious.Benchmark on Intel Xeon Cascadelake, the data comes from the 218 modeand 518 mode of tcrypt. The abscissas are blocks of different lengths.The data is tabulated and the unit is Mb/s:sm4-generic | 16 64 128 256 1024 1420 4096 ECB enc | 40.99 46.50 48.05 48.41 49.20 49.25 49.28 ECB dec | 41.07 46.99 48.15 48.67 49.20 49.25 49.29 CBC enc | 37.71 45.28 46.77 47.60 48.32 48.37 48.40 CBC dec | 36.48 44.82 46.43 47.45 48.23 48.30 48.36 CFB enc | 37.94 44.84 46.12 46.94 47.57 47.46 47.68 CFB dec | 37.50 42.84 43.74 44.37 44.85 44.80 44.96 CTR enc | 39.20 45.63 46.75 47.49 48.09 47.85 48.08 CTR dec | 39.64 45.70 46.72 47.47 47.98 47.88 48.06sm4-aesni-avx ECB enc | 33.75 134.47 221.64 243.43 264.05 251.58 258.13 ECB dec | 34.02 134.92 223.11 245.14 264.12 251.04 258.33 CBC enc | 38.85 46.18 47.67 48.34 49.00 48.96 49.14 CBC dec | 33.54 131.29 223.88 245.27 265.50 252.41 263.78 CFB enc | 38.70 46.10 47.58 48.29 49.01 48.94 49.19 CFB dec | 32.79 128.40 223.23 244.87 265.77 253.31 262.79 CTR enc | 32.58 122.23 220.29 241.16 259.57 248.32 256.69 CTR dec | 32.81 122.47 218.99 241.54 258.42 248.58 256.61Signed-off-by: Tianjia Zhang Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

7d3d10e0 - x86/crypto: Enable objtool in crypto code

Wed, 24 Feb 2021 16:29:26 +0000

x86/crypto: Enable objtool in crypto codeNow that all the stack alignment prologues have been cleaned up in thecrypto code, enable objtool. Among other benefits, this will allow ORCunwinding to work.Signed-off-by: Josh Poimboeuf Tested-by: Ard Biesheuvel Acked-by: Ard Biesheuvel Tested-by: Sami Tolvanen Acked-by: Peter Zijlstra (Intel) Acked-by: Herbert Xu Link: https://lore.kernel.org/r/fc2a1918c50e33e46ef0e9a5de02743f2f6e3639.1614182415.git.jpoimboe@redhat.com List of files: /linux-6.15/arch/x86/crypto/Makefile

64ca771c - crypto: x86 - remove glue helper module

Tue, 05 Jan 2021 16:48:08 +0000

crypto: x86 - remove glue helper moduleAll dependencies on the x86 glue helper module have been replaced bylocal instantiations of the new ECB/CBC preprocessor helper macros, sothe glue helper module can be retired.Acked-by: Eric Biggers Signed-off-by: Ard Biesheuvel Signed-off-by: Herbert Xu List of files: /linux-6.15/arch/x86/crypto/Makefile

e6abef61 - x86: update AS_* macros to binutils >=2.23, supporting ADX and AVX2

Thu, 26 Mar 2020 20:26:00 +0000

x86: update AS_* macros to binutils >=2.23, supporting ADX and AVX2Now that the kernel specifies binutils 2.23 as the minimum version, wecan remove ifdefs for AVX2 and ADX throughout.Signed-off-by: Jason A. Donenfeld Acked-by: Ingo Molnar Reviewed-by: Nick Desaulniers Signed-off-by: Masahiro Yamada List of files: /linux-6.15/arch/x86/crypto/Makefile

d7e40ea8 - crypto: x86 - clean up poly1305-x86_64-cryptogams.S by 'make clean'

Thu, 26 Mar 2020 08:01:04 +0000

crypto: x86 - clean up poly1305-x86_64-cryptogams.S by 'make clean'poly1305-x86_64-cryptogams.S is a generated file, so it should becleaned up by 'make clean'.Assigning it to the variable 'targets' teaches Kbuild that it is agenerated file. However, this line is not evaluated when cleaningbecause scripts/Makefile.clean does not include include/config/auto.conf.Remove the ifneq-conditional, so this file is correctly cleaned up.Signed-off-by: Masahiro Yamada Acked-by: Herbert Xu Acked-by: Ingo Molnar List of files: /linux-6.15/arch/x86/crypto/Makefile