zfs: merge openzfs/zfs@52bad4f23 (zfs-2.1-release) into stable/13

OpenZFS release 2.1.4

Notable upstream pull request merges:
#13219 FreeBSD: add missing replay check to an assert in zfs_xvattr_s

zfs: merge openzfs/zfs@52bad4f23 (zfs-2.1-release) into stable/13

OpenZFS release 2.1.4

Notable upstream pull request merges:
#13219 FreeBSD: add missing replay check to an assert in zfs_xvattr_set
#13220 module: freebsd: avoid a taking a destroyed lock in zfs_zevent bits
#13221 Fix ACL checks for NFS kernel server

Obtained from: OpenZFS
OpenZFS tag: zfs-2.1.4
OpenZFS commit: 52bad4f23daaa5f827f802c8d05785a27b80275d
Relnotes: yes
Approved by: re (gjb)

(cherry picked from commit c088e4d539e4cc947896a3b156646b831d932539)

show more ...

zfs: merge openzfs/zfs@ef83e07db (zfs-2.1-release) into releng/13.1

OpenZFS release 2.1.3

Notable upstream pull request merges:
#12569 FreeBSD: Really zero the zero page
#12828 FreeBSD: Add vop

zfs: merge openzfs/zfs@ef83e07db (zfs-2.1-release) into releng/13.1

OpenZFS release 2.1.3

Notable upstream pull request merges:
#12569 FreeBSD: Really zero the zero page
#12828 FreeBSD: Add vop_standard_writecount_nomsyn
#12828 zfs: Fix a deadlock between page busy and the teardown lock
#12828 FreeBSD: Catch up with more VFS changes
#12851 FreeBSD: Provide correct file generation number
#12857 Verify dRAID empty sectors
#12874 FreeBSD: Update argument types for VOP_READDIR
#12896 Reduce number of arc_prune threads
#12934 FreeBSD: Fix zvol_*_open() locking
#12961 FreeBSD: Fix leaked strings in libspl mnttab
#12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD
#12981 Introduce a flag to skip comparing the local mac when
raw sending
#12985 Avoid memory allocations in the ARC eviction thread
#13014 Report dnodes with faulty bonuslen
#13016 FreeBSD: Fix zvol_cdev_open locking
#13027 Fix clearing set-uid and set-gid bits on a file when
replying a write
#13031 Add enumerated vdev names to 'zpool iostat -v' and
'zpool list -v'
#13074 Enable encrypted raw sending to pools with greater ashift
#13076 Receive checks should allow unencrypted child datasets
#13098 Avoid dirtying the final TXGs when exporting a pool
#13172 Fix ENOSPC when unlinking multiple files from full pool

Obtained from: OpenZFS
OpenZFS commit: ef83e07db53e5d1017d3afbf376f4dbb2f6feada
OpenZFS tag: zfs-2.1.3
Relnotes: yes

(cherry picked from commit bd2e56ef47d5a2c69f6f8e092abfd27a4d469d1e)

Approved by: re (delphij)

show more ...

Disable clang 14 warning about bitwise operators in zstd

Parts of zstd, used in openzfs and other places, trigger a new clang 14
-Werror warning:

```
sys/contrib/zstd/lib/decompress/huf_decompress.

Disable clang 14 warning about bitwise operators in zstd

Parts of zstd, used in openzfs and other places, trigger a new clang 14
-Werror warning:

```
sys/contrib/zstd/lib/decompress/huf_decompress.c:889:25: error: use of bitwise '&' with boolean operands [-Werror,-Wbitwise-instead-of-logical]
(BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

While the warning is benign, it should ideally be fixed upstream and
then vendor-imported, but for now silence it selectively.

MFC after: 3 days

(cherry picked from commit 5f2aca83940097d7d23b4137073fb601f8e74232)

show more ...

Include ccompile.h after opt_global.h.

This restores INVARIANTS enabling ZFS_DEBUG when built as module.

Discussed with: freqlabs
MFC after: 1 week

(cherry picked from commit 8ea35c45d5206ad57945b

Include ccompile.h after opt_global.h.

This restores INVARIANTS enabling ZFS_DEBUG when built as module.

Discussed with: freqlabs
MFC after: 1 week

(cherry picked from commit 8ea35c45d5206ad57945ba160484f04450c88b75)

show more ...

zfs: restore FreeBSD resource usage accounting

With the switch to OpenZFS 2.0 the platform-dependent resource
usage accounting code got missing.

Patch author: Ryan Moeller <[email protected]>

O

zfs: restore FreeBSD resource usage accounting

With the switch to OpenZFS 2.0 the platform-dependent resource
usage accounting code got missing.

Patch author: Ryan Moeller <[email protected]>

Obtained from: openzfs/zfs@64e0fe14fff465604952d576051ca80c2f1be898
MFS after: 3 days

(direct commit)

show more ...

ZFS: fix assertions with INVARIANTS

AFAICT, this was an oversight from
9e5787d2284e187abb5b654d924394a65772e004 (svn r364746). That revision
inadvertently disabled assertions unconditionally.

Revi

ZFS: fix assertions with INVARIANTS

AFAICT, this was an oversight from
9e5787d2284e187abb5b654d924394a65772e004 (svn r364746). That revision
inadvertently disabled assertions unconditionally.

Reviewed by: freqlabs
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D28256

(cherry picked from commit 174a7e578a33c01401e33f9bfcc077fc3155251c)

show more ...

OpenZFS: Fix issues caused by reversed commits + rebase

openzfs: fix gcc kernel module builds

- Suppress -Wredundant-decls. Ultimately this warning is harmless in
any case, and it does not look like there is a simple way to avoid
redundant declar

openzfs: fix gcc kernel module builds

- Suppress -Wredundant-decls. Ultimately this warning is harmless in
any case, and it does not look like there is a simple way to avoid
redundant declarations in this case without a lot of header pollution
(e.g. having openzfs's shim param.h pulling in sys/kernel.h for hz).
- Suppress -Wnested-externs, which is useless anyway.

Unfortunately it was not sufficient just to modify OPENZFS_CFLAGS,
because the warning suppressions need to appear on the command line
after they are explicitly enabled by CWARNFLAGS from sys/conf/kern.mk,
but OPENZFS_CFLAGS get added before due to use of -I for the shims.

Reviewed by: markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D27685

show more ...

zfs: quiet gcc -Wmissing-include-dirs

Don't tell it to look for headers in a non-existent directory.

Reviewed by: imp, mmacy
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.fre

zfs: quiet gcc -Wmissing-include-dirs

Don't tell it to look for headers in a non-existent directory.

Reviewed by: imp, mmacy
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D27565

show more ...

ZFS: remove some extra defines

When merging a number of defines that are needed in the standalone
build made it in to the module makefile.

Reported by: markj@

Stop setting DEBUG_FLAGS in the zfs.ko Makefile.

Rely on system settings to decide whether or not to build with debug
symbols and extract them to a separate file.

Reported and tested by: rm

[PowerPC] More preemptive powerpcspe ZFS build fixes

I went through the merge and found the rest of the instances where
${MACHINE_ARCH} == "powerpc" was being used to detect 32-bit and adjusted
the

[PowerPC] More preemptive powerpcspe ZFS build fixes

I went through the merge and found the rest of the instances where
${MACHINE_ARCH} == "powerpc" was being used to detect 32-bit and adjusted
the rest of the instances to also check for powerpcspe.

mips32* will probably want to do the same.

Sponsored by: Tag1 Consulting, Inc.

show more ...

Fix zstd in OpenZFS module with CPUTYPE?=<something with BMI>

The build breaks when something adds -march=<something with BMI> to the
compiler flags, for example CPUTYPE?=native. When the arch supp

Fix zstd in OpenZFS module with CPUTYPE?=<something with BMI>

The build breaks when something adds -march=<something with BMI> to the
compiler flags, for example CPUTYPE?=native. When the arch supports BMI,
__BMI__ is defined and zstd.c tries to include immintrin.h, which is not
present when building the kernel.

Disable experimental BMI intrinsics in zstd in the OpenZFS kernel module
by explicitly undefining __BMI__ for zstd.c.

A similar fix was needed for the original zstd import, done in r327738.

Reported by: Jakob Alvermark
Discussed with: mmacy
Sponsored by: iXsystems, Inc.

show more ...

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would a

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

show more ...

Remove duplicate lz4 implementations

Port illumos change: https://www.illumos.org/issues/11667

Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userlan

Remove duplicate lz4 implementations

Port illumos change: https://www.illumos.org/issues/11667

Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userland builds, so we can use just one single
source. Add lz4.h to declare lz4_compress() and lz4_decompress().

MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D22037

show more ...

Expose zlib's utility functions in Z_SOLO library when building kernel.
This allows kernel code to reuse zlib's implementation.

PR: 229763
Reviewed by: Yoshihiro Ota <ota j email ne jp>
Relnotes: y

Expose zlib's utility functions in Z_SOLO library when building kernel.
This allows kernel code to reuse zlib's implementation.

PR: 229763
Reviewed by: Yoshihiro Ota <ota j email ne jp>
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D21156

show more ...

Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,
with an eventual goal to convert all legacl zlib callers to the new zlib
version:

* Move generic zlib shims that are not s

Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,
with an eventual goal to convert all legacl zlib callers to the new zlib
version:

* Move generic zlib shims that are not specific to zlib 1.0.4 to
sys/dev/zlib.
* Connect new zlib (1.2.11) to the zlib kernel module, currently built
with Z_SOLO.
* Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace.
* Convert sys/opencrypto/cryptodeflate.c to use new zlib.
* Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make
it depend on the zlib module.
* Fix Z_SOLO build of new zlib.

PR: 229763
Submitted by: Yoshihiro Ota <ota j email ne jp>
Reviewed by: markm (sys/dev/zlib/zlib_kmod.c)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D19706

show more ...

Make dnode definition uniform on !x86

gcc4 requires -fms-extensions to accept anonymous union members

sys/modules: normalize .CURDIR-relative paths to SRCTOP

This simplifies make output/logic

Tested with: `cd sys/modules; make ALL_MODULES=` on amd64
MFC after: 1 month
Sponsored by: Dell EMC Isilon

zfs: clean up unused files and definitions

MFC after: 1 month
X-MFC after: r314048

Update CFLAGS for clang compatibility

* Clang/llvm does not (yet) support -m(no-)spe, so make it gcc-only
* Clang now supports -msoft-float, and does not appear to recognize
"-disable-ppc-float-in

Update CFLAGS for clang compatibility

* Clang/llvm does not (yet) support -m(no-)spe, so make it gcc-only
* Clang now supports -msoft-float, and does not appear to recognize
"-disable-ppc-float-in-variadic", which appears to have been a crutch until
soft-float was implemented. It's now implemented for both 32- and 64-bit.
* Clang/llvm use a 'medium' code model by default for powerpc64, supporting up
to 4GB TOC, and does not support the '-mminimal-toc' option. Given both of
these, make -mminimal-toc gcc-only.

MFC after: 2 weeks

show more ...

Connect the SHA-512t256 and Skein hashing algorithms to ZFS

Support for the new hashing algorithms in ZFS was introduced in r289422
However it was disconnected because FreeBSD lacked implementations

Connect the SHA-512t256 and Skein hashing algorithms to ZFS

Support for the new hashing algorithms in ZFS was introduced in r289422
However it was disconnected because FreeBSD lacked implementations of
SHA-512 (truncated to 256 bits), and Skein.

These implementations were introduced in r300921 and r300966 respectively

This commit connects them to ZFS and enabled these new checksum algorithms

This new algorithms are not supported by the boot blocks, so do not use them
on your root dataset if you boot from ZFS.

Relnotes: yes
Sponsored by: ScaleEngine Inc.

show more ...

Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA

Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation

Extend sbin/md5 to create sha384(1)

Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}

Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929

show more ...

Revert r286984 (adding opt_random.h to sys/modules/zfs/Makefile)

opt_random.h is no longer needed/referenced in the kernel build

X-MFC with: r287558

fix standalone build of zfs module

Not sure if this is a proper fix, but it does the job.