xz: adjust arch-specific options for better kernel compression

Use LZMA2 options that match the arch-specific alignment of instructions.
This change reduces compressed kernel size 0-2 % depending o

xz: adjust arch-specific options for better kernel compression

Use LZMA2 options that match the arch-specific alignment of instructions.
This change reduces compressed kernel size 0-2 % depending on the arch.
On 1-byte-aligned x86 it makes no difference and on 4-byte-aligned archs
it helps the most.

Use the ARM-Thumb filter for ARM-Thumb2 kernels. This reduces compressed
kernel size about 5 %.[1] Previously such kernels were compressed using
the ARM filter which didn't do anything useful with ARM-Thumb2 code.

Add BCJ filter support for ARM64 and RISC-V. Compared to unfiltered XZ or
plain LZMA, the compressed kernel size is reduced about 5 % on ARM64 and 7
% on RISC-V. A new enough version of the xz tool is required: 5.4.0 for
ARM64 and 5.6.0 for RISC-V. With an old xz version, a message is printed
to standard error and the kernel is compressed without the filter.

Update lib/decompress_unxz.c to match the changes to xz_wrap.sh.

Update the CONFIG_KERNEL_XZ help text in init/Kconfig:
- Add the RISC-V and ARM64 filters.
- Clarify that the PowerPC filter is for big endian only.
- Omit IA-64.

Link: https://lore.kernel.org/lkml/[email protected]/ [1]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Lasse Collin <[email protected]>
Reviewed-by: Sam James <[email protected]>
Cc: Simon Glass <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Paul Walmsley <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Jubin Zhong <[email protected]>
Cc: Jules Maselbas <[email protected]>
Cc: Emil Renner Berthing <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Herbert Xu <[email protected]>
Cc: Joel Stanley <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Krzysztof Kozlowski <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Rui Li <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

xz: use 128 MiB dictionary and force single-threaded mode

This only affects kernel image compression, not any other xz usage.

Desktop kernels on x86-64 are already around 60 MiB. Using a dictionar

xz: use 128 MiB dictionary and force single-threaded mode

This only affects kernel image compression, not any other xz usage.

Desktop kernels on x86-64 are already around 60 MiB. Using a dictionary
larger than 32 MiB should have no downsides nowadays as anyone building
the kernel should have plenty of RAM. 128 MiB dictionary needs 1346 MiB
of RAM with xz versions 5.0.x - 5.6.x in single-threaded mode. On archs
that use xz_wrap.sh, kernel decompression is done in single-call mode so a
larger dictionary doesn't affect boot-time memory requirements.

xz >= 5.6.0 uses multithreaded mode by default which compresses slightly
worse than single-threaded mode. Kernel compression rarely used more than
one thread anyway because with 32 MiB dictionary size the default block
size was 96 MiB in multithreaded mode. So only a single thread was used
anyway unless the kernel was over 96 MiB.

Comparison to CONFIG_KERNEL_LZMA: It uses "lzma -9" which mapped to 32 MiB
dictionary in LZMA Utils 4.32.7 (the final release in 2008). Nowadays the
lzma tool on most systems is from XZ Utils where -9 maps to 64 MiB
dictionary. So using a 32 MiB dictionary with CONFIG_KERNEL_XZ may have
compressed big kernels slightly worse than the old LZMA option.

Comparison to CONFIG_KERNEL_ZSTD: zstd uses 128 MiB dictionary.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Lasse Collin <[email protected]>
Reviewed-by: Sam James <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Emil Renner Berthing <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Herbert Xu <[email protected]>
Cc: Joel Stanley <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Jubin Zhong <[email protected]>
Cc: Jules Maselbas <[email protected]>
Cc: Krzysztof Kozlowski <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Paul Walmsley <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Rui Li <[email protected]>
Cc: Simon Glass <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

xz: switch from public domain to BSD Zero Clause License (0BSD)

Remove the public domain notices and add SPDX license identifiers.

Change MODULE_LICENSE from "GPL" to "Dual BSD/GPL" because 0BSD sh

xz: switch from public domain to BSD Zero Clause License (0BSD)

Remove the public domain notices and add SPDX license identifiers.

Change MODULE_LICENSE from "GPL" to "Dual BSD/GPL" because 0BSD should
count as a BSD license variant here.

The switch to 0BSD was done in the upstream XZ Embedded project because
public domain has (real or perceived) legal issues in some jurisdictions.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Lasse Collin <[email protected]>
Reviewed-by: Sam James <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Emil Renner Berthing <[email protected]>
Cc: Herbert Xu <[email protected]>
Cc: Joel Stanley <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Cc: Jubin Zhong <[email protected]>
Cc: Jules Maselbas <[email protected]>
Cc: Krzysztof Kozlowski <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Paul Walmsley <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Rui Li <[email protected]>
Cc: Simon Glass <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

scripts: clean up IA-64 code

A little more janitorial work after commit cf8e8658100d ("arch: Remove
Itanium (IA-64) architecture").

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by

scripts: clean up IA-64 code

A little more janitorial work after commit cf8e8658100d ("arch: Remove
Itanium (IA-64) architecture").

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Nicolas Schier <[email protected]>

show more ...

kbuild: add variables for compression tools

Allow user to use alternative implementations of compression tools,
such as pigz, pbzip2, pxz. For example, multi-threaded tools to
speed up the build:
$

kbuild: add variables for compression tools

Allow user to use alternative implementations of compression tools,
such as pigz, pbzip2, pxz. For example, multi-threaded tools to
speed up the build:
$ make GZIP=pigz BZIP2=pbzip2

Variables _GZIP, _BZIP2, _LZOP are used internally because original env
vars are reserved by the tools. The use of GZIP in gzip tool is obsolete
since 2015. However, alternative implementations (e.g., pigz) still rely
on it. BZIP2, BZIP, LZOP vars are not obsolescent.

The credit goes to @grsecurity.

As a sidenote, for multi-threaded lzma, xz compression one can use:
$ export XZ_OPT="--threads=0"

Signed-off-by: Denis Efremov <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

kbuild: Make scripts executable

The Makefiles call the respective interpreter explicitly, but this makes
it easier to use the scripts manually.

Signed-off-by: Michal Marek <[email protected]>

xz: Enable BCJ filters on SPARC and 32-bit x86

The BCJ filters were meant to be enabled already on these
archs, but the xz_wrap.sh script was buggy. Enabling the
filters should give smaller kernel i

xz: Enable BCJ filters on SPARC and 32-bit x86

The BCJ filters were meant to be enabled already on these
archs, but the xz_wrap.sh script was buggy. Enabling the
filters should give smaller kernel images.

xz_wrap.sh will now use $SRCARCH instead of $ARCH to detect
the architecture. That way it doesn't need to care about the
subarchs (like i386 vs. x86_64) since the BCJ filters don't
care either.

Signed-off-by: Lasse Collin <[email protected]>
Acked-by: Jan Beulich <[email protected]>
Acked-by: H. Peter Anvin <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

show more ...

decompressors: add XZ decompressor module

In userspace, the .lzma format has become mostly a legacy file format that
got superseded by the .xz format. Similarly, LZMA Utils was superseded by
XZ Uti

decompressors: add XZ decompressor module

In userspace, the .lzma format has become mostly a legacy file format that
got superseded by the .xz format. Similarly, LZMA Utils was superseded by
XZ Utils.

These patches add support for XZ decompression into the kernel. Most of
the code is as is from XZ Embedded <http://tukaani.org/xz/embedded.html>.
It was written for the Linux kernel but is usable in other projects too.

Advantages of XZ over the current LZMA code in the kernel:
- Nice API that can be used by other kernel modules; it's
not limited to kernel, initramfs, and initrd decompression.
- Integrity check support (CRC32)
- BCJ filters improve compression of executable code on
certain architectures. These together with LZMA2 can
produce a few percent smaller kernel or Squashfs images
than plain LZMA without making the decompression slower.

This patch: Add the main decompression code (xz_dec), testing module
(xz_dec_test), wrapper script (xz_wrap.sh) for the xz command line tool,
and documentation. The xz_dec module is enough to have a usable XZ
decompressor e.g. for Squashfs.

Signed-off-by: Lasse Collin <[email protected]>
Cc: "H. Peter Anvin" <[email protected]>
Cc: Alain Knaff <[email protected]>
Cc: Albin Tonnerre <[email protected]>
Cc: Phillip Lougher <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

show more ...