Changes in Makefile

65d1f550 - zstd: Import upstream v1.5.7

Sat, 08 Mar 2025 20:09:33 +0000

zstd: Import upstream v1.5.7In addition to keeping the kernel's copy of zstd up to date, this updatewas requested by Intel to expose upstream's APIs that allow QAT to acceleratethe LZ match finding stage of Zstd.This patch is imported from the upstream tag v1.5.7-kernel [0], which is signedwith upstream's signing key EF8FE99528B52FFD [1]. It was imported from upstreamusing this command: export ZSTD=/path/to/repo/zstd/ export LINUX=/path/to/repo/linux/ cd "$ZSTD/contrib/linux-kernel" git checkout v1.5.7-kernel make import LINUX="$LINUX"This patch has been tested on x86-64, and has been boot tested witha zstd compressed kernel & initramfs on i386 and aarch64. I benchmarkedthe patch on x86-64 with gcc-14.2.1 on an Intel i9-9900K by measruing theperformance of compressed filesystem reads and writes.Component, Level, Size delta, C. time delta, D. time deltaBtrfs , 1, +0.00%, -6.1%, +1.4%Btrfs , 3, +0.00%, -9.8%, +3.0%Btrfs , 5, +0.00%, +1.7%, +1.4%Btrfs , 7, +0.00%, -1.9%, +2.7%Btrfs , 9, +0.00%, -3.4%, +3.7%Btrfs , 15, +0.00%, -0.3%, +3.6%SquashFS , 1, +0.00%, N/A, +1.9%The major changes that impact the kernel use cases for each version are:v1.5.7: https://github.com/facebook/zstd/releases/tag/v1.5.7* Add zstd_compress_sequences_and_literals() for use by Intel's QAT driver to implement Zstd compression acceleration in the kernel.* Fix an underflow bug in 32-bit builds that can cause data corruption when processing more than 4GB of data with a single `ZSTD_CCtx` object, when an input crosses the 4GB boundry. I don't believe this impacts any current kernel use cases, because the `ZSTD_CCtx` is typically reconstructed between compressions.* Levels 1-4 see 5-10% compression speed improvements for inputs smaller than 128KB.v1.5.6: https://github.com/facebook/zstd/releases/tag/v1.5.6* Improved compression ratio for the highest compression levels. I don't expect these see much use however, due to their slow speeds.v1.5.5: https://github.com/facebook/zstd/releases/tag/v1.5.5* Fix a rare corruption bug that can trigger on levels 13 and above.* Improve compression speed of levels 5-11 on incompressible data.v1.5.4: https://github.com/facebook/zstd/releases/tag/v1.5.4* Improve copmression speed of levels 5-11 on ARM.* Improve dictionary compression speed.Signed-off-by: Nick Terrell List of files: /linux-6.15/lib/zstd/Makefile

4782c725 - zstd: Move zstd-common module exports to zstd_common_module.c

Fri, 14 Oct 2022 21:47:04 +0000

zstd: Move zstd-common module exports to zstd_common_module.cThe zstd codebase is imported from the upstream zstd repo, and is over-written onevery update. Upstream keeps the kernel specific code separate from the mainlibrary. So the module definition is moved into the zstd_common_module.c file.This matches the pattern followed by the zstd-compress and zstd-decompress files.I've done build and boot testing on x86-64, i386, and aarch64. I'veverified that zstd built both as modules and built-in build and boot.Signed-off-by: Nick Terrell List of files: /linux-6.15/lib/zstd/Makefile

637a642f - zstd: Fixing mixed module-builtin objects

Thu, 29 Sep 2022 02:08:23 +0000

zstd: Fixing mixed module-builtin objectsWith CONFIG_ZSTD_COMPRESS=m and CONFIG_ZSTD_DECOMPRESS=y we end up ina situation when files from lib/zstd/common/ are compiled once to belinked later for ZSTD_DECOMPRESS (build-in) and ZSTD_COMPRESS (module)even though CFLAGS are different for builtins and modules.So far somehow this was not a problem but enabling LLVM LTO exposesthe problem as:ld.lld: error: linking module flags 'Code Model': IDs have conflicting values in 'lib/built-in.a(zstd_common.o at 5868)' and 'ld-temp.o'This particular conflict is caused by KBUILD_CFLAGS=-mcmodel=medium vs.KBUILD_CFLAGS_MODULE=-mcmodel=large , modules use the large model onPOWERPC as explained athttps://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/Makefile?h=v5.18-rc4#n127but the current use of common files is wrong anyway.This works around the issue by introducing a zstd_common module withshared code.Signed-off-by: Alexey Kardashevskiy Signed-off-by: Masahiro Yamada List of files: /linux-6.15/lib/zstd/Makefile

7416cdc9 - lib: zstd: Don't add -O3 to cflags

Tue, 16 Nov 2021 23:11:39 +0000

lib: zstd: Don't add -O3 to cflagsAfter the update to zstd-1.4.10 passing -O3 is no longer necessary toget good performance from zstd. Using the default optimization level -O2is sufficient to get good performance.I've measured no significant change to compression speed, and a ~1%decompression speed loss, which is acceptable.This fixes the reported parisc -Wframe-larger-than=1536 errors [0]. Thegcc-8-hppa-linux-gnu compiler performed very poorly with -O3, generatingstacks that are ~3KB. With -O2 these same functions generate stacks inthe < 100B, completely fixing the problem. Function size deltas arelisted below:ZSTD_compressBlock_fast_extDict_generic: 3800 -> 68ZSTD_compressBlock_fast: 2216 -> 40ZSTD_compressBlock_fast_dictMatchState: 1848 -> 64ZSTD_compressBlock_doubleFast_extDict_generic: 3744 -> 76ZSTD_fillDoubleHashTable: 3252 -> 0ZSTD_compressBlock_doubleFast: 5856 -> 36ZSTD_compressBlock_doubleFast_dictMatchState: 5380 -> 84ZSTD_copmressBlock_lazy2: 2420 -> 72Additionally, this improves the reported code bloat [1]. With gcc-11bloat-o-meter shows an 80KB code size improvement:```> ../scripts/bloat-o-meter vmlinux.old vmlinuxadd/remove: 31/8 grow/shrink: 24/155 up/down: 25734/-107924 (-82190)Total: Before=6418562, After=6336372, chg -1.28%```Compared to before the zstd-1.4.10 update we see a total code sizeregression of 105KB, down from 374KB at v5.16-rc1:```> ../scripts/bloat-o-meter vmlinux.old vmlinuxadd/remove: 292/62 grow/shrink: 56/88 up/down: 235009/-127487 (107522)Total: Before=6228850, After=6336372, chg +1.73%```[0] https://lkml.org/lkml/2021/11/15/710[1] https://lkml.org/lkml/2021/11/14/189Link: https://lore.kernel.org/r/20211117014949.1169186-4-nickrterrell@gmail.com/Link: https://lore.kernel.org/r/20211117201459.1194876-4-nickrterrell@gmail.com/Reported-by: Geert Uytterhoeven Tested-by: Geert Uytterhoeven Reviewed-by: Geert Uytterhoeven Signed-off-by: Nick Terrell List of files: /linux-6.15/lib/zstd/Makefile

e0c1b49f - lib: zstd: Upgrade to latest upstream zstd version 1.4.10

Fri, 11 Sep 2020 23:37:08 +0000

lib: zstd: Upgrade to latest upstream zstd version 1.4.10Upgrade to the latest upstream zstd version 1.4.10.This patch is 100% generated from upstream zstd commit 20821a46f412 [0].This patch is very large because it is transitioning from the customkernel zstd to using upstream directly. The new zstd follows upstreamsfile structure which is different. Future update patches will be muchsmaller because they will only contain the changes from one upstreamzstd release.As an aid for review I've created a commit [1] that shows the diffbetween upstream zstd as-is (which doesn't compile), and the zstdcode imported in this patch. The verion of zstd in this patch isgenerated from upstream with changes applied by automation to replaceupstreams libc dependencies, remove unnecessary portability macros,replace `/**` comments with `/*` comments, and use the kernel's xxhashinstead of bundling it.The benefits of this patch are as follows:1. Using upstream directly with automated script to generate kernel code. This allows us to update the kernel every upstream release, so the kernel gets the latest bug fixes and performance improvements, and doesn't get 3 years out of date again. The automation and the translated code are tested every upstream commit to ensure it continues to work.2. Upgrades from a custom zstd based on 1.3.1 to 1.4.10, getting 3 years of performance improvements and bug fixes. On x86_64 I've measured 15% faster BtrFS and SquashFS decompression+read speeds, 35% faster kernel decompression, and 30% faster ZRAM decompression+read speeds.3. Zstd-1.4.10 supports negative compression levels, which allow zstd to match or subsume lzo's performance.4. Maintains the same kernel-specific wrapper API, so no callers have to be modified with zstd version updates.One concern that was brought up was stack usage. Upstream zstd hadalready removed most of its heavy stack usage functions, but I justremoved the last functions that allocate arrays on the stack. I'vemeasured the high water mark for both compression and decompressionbefore and after this patch. Decompression is approximately neutral,using about 1.2KB of stack space. Compression levels up to 3 regressedfrom 1.4KB -> 1.6KB, and higher compression levels regressed from 1.5KB-> 2KB. We've added unit tests upstream to prevent further regression.I believe that this is a reasonable increase, and if it does end upcausing problems, this commit can be cleanly reverted, because it onlytouches zstd.I chose the bulk update instead of replaying upstream commits becausethere have been ~3500 upstream commits since the 1.3.1 release, zstdwasn't ready to be used in the kernel as-is before a month ago, and notall upstream zstd commits build. The bulk update preserves bisectablitybecause bugs can be bisected to the zstd version update. At that pointthe update can be reverted, and we can work with upstream to find andfix the bug.Note that upstream zstd release 1.4.10 doesn't exist yet. I have cut astaging branch at 20821a46f412 [0] and will apply any changes requestedto the staging branch. Once we're ready to merge this update I will cuta zstd release at the commit we merge, so we have a known zstd releasein the kernel.The implementation of the kernel API is contained inzstd_compress_module.c and zstd_decompress_module.c.[0] https://github.com/facebook/zstd/commit/20821a46f4122f9abd7c7b245d28162dde8129c9[1] https://github.com/terrelln/linux/commit/e0fa481d0e3df26918da0a13749740a1f6777574Signed-off-by: Nick Terrell Tested By: Paul Jones Tested-by: Oleksandr Natalenko Tested-by: Sedat Dilek # LLVM/Clang v13.0.0 on x86-64Tested-by: Jean-Denis Girard List of files: /linux-6.15/lib/zstd/Makefile

b16838c6 - kbuild: trace functions in subdirectories of lib/

Tue, 07 Jul 2020 09:21:17 +0000

kbuild: trace functions in subdirectories of lib/ccflags-remove-$(CONFIG_FUNCTION_TRACER) += $(CC_FLAGS_FTRACE)exists here in sub-directories of lib/ to keep the behavior ofcommit 2464a609ded0 ("ftrace: do not trace library functions").Since that commit, not only the objects in lib/ but also the ones inthe sub-directories are excluded from ftrace (although the commitdescription did not explicitly mention this).However, most of library functions in sub-directories are not so hot.Re-add them to ftrace.Going forward, only the objects right under lib/ will be excluded.Cc: Ingo Molnar Signed-off-by: Masahiro Yamada Acked-by: Steven Rostedt (VMware) List of files: /linux-6.15/lib/zstd/Makefile

15d5761a - kbuild: introduce ccflags-remove-y and asflags-remove-y

Tue, 07 Jul 2020 09:21:16 +0000

kbuild: introduce ccflags-remove-y and asflags-remove-yCFLAGS_REMOVE_.o filters out flags when compiling a particularobject, but there is no convenient way to do that for every object ina directory.Add ccflags-remove-y and asflags-remove-y to make it easily.Use ccflags-remove-y to clean up some Makefiles.The add/remove order works as follows: [1] KBUILD_CFLAGS specifies compiler flags used globally [2] ccflags-y adds compiler flags for all objects in the current Makefile [3] ccflags-remove-y removes compiler flags for all objects in the current Makefile (New feature) [4] CFLAGS_ adds compiler flags per file. [5] CFLAGS_REMOVE_ removes compiler flags per file.Having [3] before [4] allows us to remove flags from most (but not all)objects in the current Makefile.For example, kernel/trace/Makefile removes $(CC_FLAGS_FTRACE)from all objects in the directory, then adds it back totrace_selftest_dynamic.o and CFLAGS_trace_kprobe_selftest.oThe same applies to lib/livepatch/Makefile.Please note ccflags-remove-y has no effect to the sub-directories.In contrast, the previous notation got rid of compiler flags also fromall the sub-directories.The following are not affected because they have no sub-directories: arch/arm/boot/compressed/ arch/powerpc/xmon/ arch/sh/ kernel/trace/However, lib/ has several sub-directories.To keep the behavior, I added ccflags-remove-y to all Makefilesin subdirectories of lib/, except the following: lib/vdso/Makefile - Kbuild does not descend into this Makefile lib/raid/test/Makefile - This is not used for the kernel buildI think commit 2464a609ded0 ("ftrace: do not trace library functions")excluded too much. In the next commit, I will remove ccflags-remove-yfrom the sub-directories of lib/.Suggested-by: Sami Tolvanen Signed-off-by: Masahiro Yamada Acked-by: Steven Rostedt (VMware) Acked-by: Michael Ellerman (powerpc)Acked-by: Brendan Higgins (KUnit)Tested-by: Anders Roxell List of files: /linux-6.15/lib/zstd/Makefile

ec8f24b7 - treewide: Add SPDX license identifier - Makefile/Kconfig

Sun, 19 May 2019 12:07:45 +0000

treewide: Add SPDX license identifier - Makefile/KconfigAdd SPDX license identifiers to all Make/Kconfig files which: - Have no license information of any formThese files fall under the project license, GPL v2 only. The resulting SPDXlicense identifier is: GPL-2.0-onlySigned-off-by: Thomas Gleixner Signed-off-by: Greg Kroah-Hartman List of files: /linux-6.15/lib/zstd/Makefile

dc35da16 - lib: zstd: clean up Makefile for simpler composite object handling

Mon, 19 Mar 2018 11:26:09 +0000

lib: zstd: clean up Makefile for simpler composite object handlingNow, Kbuild nicely handles composite objects to avoid multipledefinition.Makefiles can simply add the same objects multiple times acrosscomposite objects.Signed-off-by: Masahiro Yamada List of files: /linux-6.15/lib/zstd/Makefile

73f3d1b4 - lib: Add zstd modules

Thu, 10 Aug 2017 02:35:53 +0000

lib: Add zstd modulesAdd zstd compression and decompression kernel modules.zstd offers a wide varity of compression speed and quality trade-offs.It can compress at speeds approaching lz4, and quality approaching lzma.zstd decompressions at speeds more than twice as fast as zlib, anddecompression speed remains roughly the same across all compression levels.The code was ported from the upstream zstd source repository. The`linux/zstd.h` header was modified to match linux kernel style.The cross-platform and allocation code was stripped out. Instead zstdrequires the caller to pass a preallocated workspace. The source fileswere clang-formatted [1] to match the Linux Kernel style as much aspossible. Otherwise, the code was unmodified. We would like to avoidas much further manual modification to the source code as possible, so itwill be easier to keep the kernel zstd up to date.I benchmarked zstd compression as a special character device. I ran zstdand zlib compression at several levels, as well as performing nocompression, which measure the time spent copying the data to kernel space.Data is passed to the compresser 4096 B at a time. The benchmark file islocated in the upstream zstd source repository under`contrib/linux-kernel/zstd_compress_test.c` [2].I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,16 GB of RAM, and a SSD. I benchmarked using `silesia.tar` [3], which is211,988,480 B large. Run the following commands for the benchmark: sudo modprobe zstd_compress_test sudo mknod zstd_compress_test c 245 0 sudo cp silesia.tar zstd_compress_testThe time is reported by the time of the userland `cp`.The MB/s is computed with 1,536,217,008 B / time(buffer size, hash)which includes the time to copy from userland.The Adjusted MB/s is computed with 1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).The memory reported is the amount of memory the compressor requests.| Method | Size (B) | Time (s) | Ratio | MB/s | Adj MB/s | Mem (MB) ||----------|----------|----------|-------|---------|----------|----------|| none | 11988480 | 0.100 | 1 | 2119.88 | - | - || zstd -1 | 73645762 | 1.044 | 2.878 | 203.05 | 224.56 | 1.23 || zstd -3 | 66988878 | 1.761 | 3.165 | 120.38 | 127.63 | 2.47 || zstd -5 | 65001259 | 2.563 | 3.261 | 82.71 | 86.07 | 2.86 || zstd -10 | 60165346 | 13.242 | 3.523 | 16.01 | 16.13 | 13.22 || zstd -15 | 58009756 | 47.601 | 3.654 | 4.45 | 4.46 | 21.61 || zstd -19 | 54014593 | 102.835 | 3.925 | 2.06 | 2.06 | 60.15 || zlib -1 | 77260026 | 2.895 | 2.744 | 73.23 | 75.85 | 0.27 || zlib -3 | 72972206 | 4.116 | 2.905 | 51.50 | 52.79 | 0.27 || zlib -6 | 68190360 | 9.633 | 3.109 | 22.01 | 22.24 | 0.27 || zlib -9 | 67613382 | 22.554 | 3.135 | 9.40 | 9.44 | 0.27 |I benchmarked zstd decompression using the same method on the same machine.The benchmark file is located in the upstream zstd repo under`contrib/linux-kernel/zstd_decompress_test.c` [4]. The memory reported isthe amount of memory required to decompress data compressed with the givencompression level. If you know the maximum size of your input, you canreduce the memory usage of decompression irrespective of the compressionlevel.| Method | Time (s) | MB/s | Adjusted MB/s | Memory (MB) ||----------|----------|---------|---------------|-------------|| none | 0.025 | 8479.54 | - | - || zstd -1 | 0.358 | 592.15 | 636.60 | 0.84 || zstd -3 | 0.396 | 535.32 | 571.40 | 1.46 || zstd -5 | 0.396 | 535.32 | 571.40 | 1.46 || zstd -10 | 0.374 | 566.81 | 607.42 | 2.51 || zstd -15 | 0.379 | 559.34 | 598.84 | 4.61 || zstd -19 | 0.412 | 514.54 | 547.77 | 8.80 || zlib -1 | 0.940 | 225.52 | 231.68 | 0.04 || zlib -3 | 0.883 | 240.08 | 247.07 | 0.04 || zlib -6 | 0.844 | 251.17 | 258.84 | 0.04 || zlib -9 | 0.837 | 253.27 | 287.64 | 0.04 |Tested in userland using the test-suite in the zstd repo under`contrib/linux-kernel/test/UserlandTest.cpp` [5] by mocking the kernelfunctions. Fuzz tested using libfuzzer [6] with the fuzz harnesses under`contrib/linux-kernel/test/{RoundTripCrash.c,DecompressCrash.c}` [7] [8]with ASAN, UBSAN, and MSAN. Additionaly, it was tested while testing theBtrFS and SquashFS patches coming next.[1] https://clang.llvm.org/docs/ClangFormat.html[2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/zstd_compress_test.c[3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia[4] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/zstd_decompress_test.c[5] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/test/UserlandTest.cpp[6] http://llvm.org/docs/LibFuzzer.html[7] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/test/RoundTripCrash.c[8] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/test/DecompressCrash.czstd source repository: https://github.com/facebook/zstdSigned-off-by: Nick Terrell Signed-off-by: Chris Mason List of files: /linux-6.15/lib/zstd/Makefile