crypto: ahash - Disable request chaining

Disable hash request chaining in case a driver that copies an
ahash_request object by hand accidentally triggers chaining.

Reported-by: Manorit Chawdhry <m-

crypto: ahash - Disable request chaining

Disable hash request chaining in case a driver that copies an
ahash_request object by hand accidentally triggers chaining.

Reported-by: Manorit Chawdhry <[email protected]>
Fixes: f2ffe5a9183d ("crypto: hash - Add request chaining API")
Signed-off-by: Herbert Xu <[email protected]>
Tested-by: Manorit Chawdhry <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - Set default reqsize from ahash_alg

Add a reqsize field to struct ahash_alg and use it to set the
default reqsize so that algorithms with a static reqsize are
not forced to create an

crypto: ahash - Set default reqsize from ahash_alg

Add a reqsize field to struct ahash_alg and use it to set the
default reqsize so that algorithms with a static reqsize are
not forced to create an init_tfm function.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - Add virtual address support

This patch adds virtual address support to ahash. Virtual addresses
were previously only supported through shash. The user may choose
to use virtual add

crypto: ahash - Add virtual address support

This patch adds virtual address support to ahash. Virtual addresses
were previously only supported through shash. The user may choose
to use virtual addresses with ahash by calling ahash_request_set_virt
instead of ahash_request_set_crypt.

The API will take care of translating this to an SG list if necessary,
unless the algorithm declares that it supports chaining. Therefore
in order for an ahash algorithm to support chaining, it must also
support virtual addresses directly.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Add request chaining API

This adds request chaining to the ahash interface. Request chaining
allows multiple requests to be submitted in one shot. An algorithm
can elect to receive

crypto: hash - Add request chaining API

This adds request chaining to the ahash interface. Request chaining
allows multiple requests to be submitted in one shot. An algorithm
can elect to receive chained requests by setting the flag
CRYPTO_ALG_REQ_CHAIN. If this bit is not set, the API will break
up chained requests and submit them one-by-one.

A new err field is added to struct crypto_async_request to record
the return value for each individual request.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - Only save callback and data in ahash_save_req

As unaligned operations are supported by the underlying algorithm,
ahash_save_req and ahash_restore_req can be greatly simplified to
onl

crypto: ahash - Only save callback and data in ahash_save_req

As unaligned operations are supported by the underlying algorithm,
ahash_save_req and ahash_restore_req can be greatly simplified to
only preserve the callback and data.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

mm: change inlined allocation helpers to account at the call site

Main goal of memory allocation profiling patchset is to provide accounting
that is cheap enough to run in production. To achieve th

mm: change inlined allocation helpers to account at the call site

Main goal of memory allocation profiling patchset is to provide accounting
that is cheap enough to run in production. To achieve that we inject
counters using codetags at the allocation call sites to account every time
allocation is made. This injection allows us to perform accounting
efficiently because injected counters are immediately available as opposed
to the alternative methods, such as using _RET_IP_, which would require
counter lookup and appropriate locking that makes accounting much more
expensive. This method requires all allocation functions to inject
separate counters at their call sites so that their callers can be
individually accounted. Counter injection is implemented by allocation
hooks which should wrap all allocation functions.

Inlined functions which perform allocations but do not use allocation
hooks are directly charged for the allocations they perform. In most
cases these functions are just specialized allocation wrappers used from
multiple places to allocate objects of a specific type. It would be more
useful to do the accounting at their call sites instead. Instrument these
helpers to do accounting at the call site. Simple inlined allocation
wrappers are converted directly into macros. More complex allocators or
allocators with documentation are converted into _noprof versions and
allocation hooks are added. This allows memory allocation profiling
mechanism to charge allocations to the callers of these functions.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Suren Baghdasaryan <[email protected]>
Acked-by: Jan Kara <[email protected]> [jbd2]
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Benjamin Tissoires <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: David S. Miller <[email protected]>
Cc: Dennis Zhou <[email protected]>
Cc: Eric Dumazet <[email protected]>
Cc: Herbert Xu <[email protected]>
Cc: Jakub Kicinski <[email protected]>
Cc: Jakub Sitnicki <[email protected]>
Cc: Jiri Kosina <[email protected]>
Cc: Joerg Roedel <[email protected]>
Cc: Joonsoo Kim <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Cc: Paolo Abeni <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Tejun Heo <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

crypto: remove CONFIG_CRYPTO_STATS

Remove support for the "Crypto usage statistics" feature
(CONFIG_CRYPTO_STATS). This feature does not appear to have ever been
used, and it is harmful because it

crypto: remove CONFIG_CRYPTO_STATS

Remove support for the "Crypto usage statistics" feature
(CONFIG_CRYPTO_STATS). This feature does not appear to have ever been
used, and it is harmful because it significantly reduces performance and
is a large maintenance burden.

Covering each of these points in detail:

1. Feature is not being used

Since these generic crypto statistics are only readable using netlink,
it's fairly straightforward to look for programs that use them. I'm
unable to find any evidence that any such programs exist. For example,
Debian Code Search returns no hits except the kernel header and kernel
code itself and translations of the kernel header:
https://codesearch.debian.net/search?q=CRYPTOCFGA_STAT&literal=1&perpkg=1

The patch series that added this feature in 2018
(https://lore.kernel.org/linux-crypto/[email protected]/)
said "The goal is to have an ifconfig for crypto device." This doesn't
appear to have happened.

It's not clear that there is real demand for crypto statistics. Just
because the kernel provides other types of statistics such as I/O and
networking statistics and some people find those useful does not mean
that crypto statistics are useful too.

Further evidence that programs are not using CONFIG_CRYPTO_STATS is that
it was able to be disabled in RHEL and Fedora as a bug fix
(https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/2947).

Even further evidence comes from the fact that there are and have been
bugs in how the stats work, but they were never reported. For example,
before Linux v6.7 hash stats were double-counted in most cases.

There has also never been any documentation for this feature, so it
might be hard to use even if someone wanted to.

2. CONFIG_CRYPTO_STATS significantly reduces performance

Enabling CONFIG_CRYPTO_STATS significantly reduces the performance of
the crypto API, even if no program ever retrieves the statistics. This
primarily affects systems with a large number of CPUs. For example,
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2039576 reported
that Lustre client encryption performance improved from 21.7GB/s to
48.2GB/s by disabling CONFIG_CRYPTO_STATS.

It can be argued that this means that CONFIG_CRYPTO_STATS should be
optimized with per-cpu counters similar to many of the networking
counters. But no one has done this in 5+ years. This is consistent
with the fact that the feature appears to be unused, so there seems to
be little interest in improving it as opposed to just disabling it.

It can be argued that because CONFIG_CRYPTO_STATS is off by default,
performance doesn't matter. But Linux distros tend to error on the side
of enabling options. The option is enabled in Ubuntu and Arch Linux,
and until recently was enabled in RHEL and Fedora (see above). So, even
just having the option available is harmful to users.

3. CONFIG_CRYPTO_STATS is a large maintenance burden

There are over 1000 lines of code associated with CONFIG_CRYPTO_STATS,
spread among 32 files. It significantly complicates much of the
implementation of the crypto API. After the initial submission, many
fixes and refactorings have consumed effort of multiple people to keep
this feature "working". We should be spending this effort elsewhere.

Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Corentin Labbe <[email protected]>
Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

Revert "crypto: remove CONFIG_CRYPTO_STATS"

This reverts commit 2beb81fbf0c01a62515a1bcef326168494ee2bd0.

While removing CONFIG_CRYPTO_STATS is a worthy goal, this also
removed unrelated infrastruc

Revert "crypto: remove CONFIG_CRYPTO_STATS"

This reverts commit 2beb81fbf0c01a62515a1bcef326168494ee2bd0.

While removing CONFIG_CRYPTO_STATS is a worthy goal, this also
removed unrelated infrastructure such as crypto_comp_alg_common.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: remove CONFIG_CRYPTO_STATS

Remove support for the "Crypto usage statistics" feature
(CONFIG_CRYPTO_STATS). This feature does not appear to have ever been
used, and it is harmful because it

crypto: remove CONFIG_CRYPTO_STATS

Remove support for the "Crypto usage statistics" feature
(CONFIG_CRYPTO_STATS). This feature does not appear to have ever been
used, and it is harmful because it significantly reduces performance and
is a large maintenance burden.

Covering each of these points in detail:

1. Feature is not being used

Since these generic crypto statistics are only readable using netlink,
it's fairly straightforward to look for programs that use them. I'm
unable to find any evidence that any such programs exist. For example,
Debian Code Search returns no hits except the kernel header and kernel
code itself and translations of the kernel header:
https://codesearch.debian.net/search?q=CRYPTOCFGA_STAT&literal=1&perpkg=1

The patch series that added this feature in 2018
(https://lore.kernel.org/linux-crypto/[email protected]/)
said "The goal is to have an ifconfig for crypto device." This doesn't
appear to have happened.

It's not clear that there is real demand for crypto statistics. Just
because the kernel provides other types of statistics such as I/O and
networking statistics and some people find those useful does not mean
that crypto statistics are useful too.

Further evidence that programs are not using CONFIG_CRYPTO_STATS is that
it was able to be disabled in RHEL and Fedora as a bug fix
(https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/2947).

Even further evidence comes from the fact that there are and have been
bugs in how the stats work, but they were never reported. For example,
before Linux v6.7 hash stats were double-counted in most cases.

There has also never been any documentation for this feature, so it
might be hard to use even if someone wanted to.

2. CONFIG_CRYPTO_STATS significantly reduces performance

Enabling CONFIG_CRYPTO_STATS significantly reduces the performance of
the crypto API, even if no program ever retrieves the statistics. This
primarily affects systems with large number of CPUs. For example,
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2039576 reported
that Lustre client encryption performance improved from 21.7GB/s to
48.2GB/s by disabling CONFIG_CRYPTO_STATS.

It can be argued that this means that CONFIG_CRYPTO_STATS should be
optimized with per-cpu counters similar to many of the networking
counters. But no one has done this in 5+ years. This is consistent
with the fact that the feature appears to be unused, so there seems to
be little interest in improving it as opposed to just disabling it.

It can be argued that because CONFIG_CRYPTO_STATS is off by default,
performance doesn't matter. But Linux distros tend to error on the side
of enabling options. The option is enabled in Ubuntu and Arch Linux,
and until recently was enabled in RHEL and Fedora (see above). So, even
just having the option available is harmful to users.

3. CONFIG_CRYPTO_STATS is a large maintenance burden

There are over 1000 lines of code associated with CONFIG_CRYPTO_STATS,
spread among 32 files. It significantly complicates much of the
implementation of the crypto API. After the initial submission, many
fixes and refactorings have consumed effort of multiple people to keep
this feature "working". We should be spending this effort elsewhere.

Cc: Corentin Labbe <[email protected]>
Signed-off-by: Eric Biggers <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Acked-by: Corentin Labbe <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: shash - remove excess kerneldoc members

Commit 42808e5dc602 ("crypto: hash - Count error stats differently")
moved some fields from 'struct shash_alg' into HASH_ALG_COMMON but
didn't remove

crypto: shash - remove excess kerneldoc members

Commit 42808e5dc602 ("crypto: hash - Count error stats differently")
moved some fields from 'struct shash_alg' into HASH_ALG_COMMON but
didn't remove the corresponding kerneldoc members, which results in
these warnings when running 'make htmldocs':

./include/crypto/hash.h:248: warning: Excess struct member 'digestsize' description in 'shash_alg'
./include/crypto/hash.h:248: warning: Excess struct member 'statesize' description in 'shash_alg'
./include/crypto/hash.h:248: warning: Excess struct member 'stat' description in 'shash_alg'
./include/crypto/hash.h:248: warning: Excess struct member 'base' description in 'shash_alg'

HASH_ALG_COMMON already has the documentation for all these fields.

Fixes: 42808e5dc602 ("crypto: hash - Count error stats differently")
Cc: Randy Dunlap <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Vegard Nossum <[email protected]>
Reviewed-by: Randy Dunlap <[email protected]>
Tested-by: Randy Dunlap <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - optimize performance when wrapping shash

The "ahash" API provides access to both CPU-based and hardware offload-
based implementations of hash algorithms. Typically the former are
i

crypto: ahash - optimize performance when wrapping shash

The "ahash" API provides access to both CPU-based and hardware offload-
based implementations of hash algorithms. Typically the former are
implemented as "shash" algorithms under the hood, while the latter are
implemented as "ahash" algorithms. The "ahash" API provides access to
both. Various kernel subsystems use the ahash API because they want to
support hashing hardware offload without using a separate API for it.

Yet, the common case is that a crypto accelerator is not actually being
used, and ahash is just wrapping a CPU-based shash algorithm.

This patch optimizes the ahash API for that common case by eliminating
the extra indirect call for each ahash operation on top of shash.

It also fixes the double-counting of crypto stats in this scenario
(though CONFIG_CRYPTO_STATS should *not* be enabled by anyone interested
in performance anyway...), and it eliminates redundant checking of
CRYPTO_TFM_NEED_KEY. As a bonus, it also shrinks struct crypto_ahash.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - remove crypto_ahash_alignmask

crypto_ahash_alignmask() no longer has any callers, and it always
returns 0 now that neither ahash nor shash algorithms support nonzero
alignmasks anymo

crypto: ahash - remove crypto_ahash_alignmask

crypto_ahash_alignmask() no longer has any callers, and it always
returns 0 now that neither ahash nor shash algorithms support nonzero
alignmasks anymore. Therefore, remove it.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: shash - remove crypto_shash_alignmask

crypto_shash_alignmask() no longer has any callers, and it always
returns 0 now that the shash algorithm type no longer supports nonzero
alignmasks. Th

crypto: shash - remove crypto_shash_alignmask

crypto_shash_alignmask() no longer has any callers, and it always
returns 0 now that the shash algorithm type no longer supports nonzero
alignmasks. Therefore, remove it.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: shash - eliminate indirect call for default import and export

Most shash algorithms don't have custom ->import and ->export functions,
resulting in the memcpy() based default being used. Ye

crypto: shash - eliminate indirect call for default import and export

Most shash algorithms don't have custom ->import and ->export functions,
resulting in the memcpy() based default being used. Yet,
crypto_shash_import() and crypto_shash_export() still make an indirect
call, which is expensive. Therefore, change how the default import and
export are called to make it so that crypto_shash_import() and
crypto_shash_export() don't do an indirect call in this case.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Add statesize to crypto_ahash

As ahash drivers may need to use fallbacks, their state size
is thus variable. Deal with this by making it an attribute
of crypto_ahash.

Signed-off-by:

crypto: hash - Add statesize to crypto_ahash

As ahash drivers may need to use fallbacks, their state size
is thus variable. Deal with this by making it an attribute
of crypto_ahash.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Add crypto_clone_ahash/shash

This patch adds the helpers crypto_clone_ahash and crypto_clone_shash.
They are the hash-specific counterparts of crypto_clone_tfm.

This allows code path

crypto: hash - Add crypto_clone_ahash/shash

This patch adds the helpers crypto_clone_ahash and crypto_clone_shash.
They are the hash-specific counterparts of crypto_clone_tfm.

This allows code paths that cannot otherwise allocate a hash tfm
object to do so. Once a new tfm has been obtained its key could
then be changed without impacting other users.

Note that only algorithms that implement clone_tfm can be cloned.
However, all keyless hashes can be cloned by simply reusing the
tfm object.

Signed-off-by: Herbert Xu <[email protected]>
Reviewed-by: Simon Horman <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Remove maximum statesize limit

Remove the HASH_MAX_STATESIZE limit now that it is unused.

Signed-off-by: Herbert Xu <[email protected]>

crypto: hash - Fix kdoc errors about HASH_ALG_COMMON

The HASH_ALG_COMMON macro cannot be parsed by kdoc so mark it as
a normal comment instead of kdoc. Also add HASH_ALG_COMMON as a
structure membe

crypto: hash - Fix kdoc errors about HASH_ALG_COMMON

The HASH_ALG_COMMON macro cannot be parsed by kdoc so mark it as
a normal comment instead of kdoc. Also add HASH_ALG_COMMON as a
structure member of shash_alg.

Fixes: 0e4e6d7094df ("crypto: hash - Count error stats differently")
Reported-by: Stephen Rothwell <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Count error stats differently

Move all stat code specific to hash into the hash code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case

crypto: hash - Count error stats differently

Move all stat code specific to hash into the hash code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: add crypto_has_shash()

Add helper function to determine if a given synchronous hash is supported.

Signed-off-by: Hannes Reinecke <[email protected]>
Reviewed-by: Sagi Grimberg <[email protected]>

crypto: add crypto_has_shash()

Add helper function to determine if a given synchronous hash is supported.

Signed-off-by: Hannes Reinecke <[email protected]>
Reviewed-by: Sagi Grimberg <[email protected]>
Reviewed-by: Chaitanya Kulkarni <[email protected]>
Reviewed-by: Himanshu Madhani <[email protected]>
Acked-by: Herbert Xu <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Jens Axboe <[email protected]>

show more ...

crypto: header - Fix spelling errors

Fix some spelling mistakes in comments:
cipherntext ==> ciphertext
syncronise ==> synchronise
feeded ==> fed

Signed-off-by: Zhen Lei <[email protected]

crypto: header - Fix spelling errors

Fix some spelling mistakes in comments:
cipherntext ==> ciphertext
syncronise ==> synchronise
feeded ==> fed

Signed-off-by: Zhen Lei <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - check for ERR pointers in crypto_destroy_tfm()

Given that crypto_alloc_tfm() may return ERR pointers, and to avoid
crashes on obscure error paths where such pointers are presented to
c

crypto: api - check for ERR pointers in crypto_destroy_tfm()

Given that crypto_alloc_tfm() may return ERR pointers, and to avoid
crashes on obscure error paths where such pointers are presented to
crypto_destroy_tfm() (such as [0]), add an ERR_PTR check there
before dereferencing the second argument as a struct crypto_tfm
pointer.

[0] https://lore.kernel.org/linux-crypto/[email protected]/

Reported-by: [email protected]
Reviewed-by: Eric Biggers <[email protected]>
Signed-off-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto - shash: reduce minimum alignment of shash_desc structure

Unlike many other structure types defined in the crypto API, the
'shash_desc' structure is permitted to live on the stack, which
impl

crypto - shash: reduce minimum alignment of shash_desc structure

Unlike many other structure types defined in the crypto API, the
'shash_desc' structure is permitted to live on the stack, which
implies its contents may not be accessed by DMA masters. (This is
due to the fact that the stack may be located in the vmalloc area,
which requires a different virtual-to-physical translation than the
one implemented by the DMA subsystem)

Our definition of CRYPTO_MINALIGN_ATTR is based on ARCH_KMALLOC_MINALIGN,
which may take DMA constraints into account on architectures that support
non-cache coherent DMA such as ARM and arm64. In this case, the value is
chosen to reflect the largest cacheline size in the system, in order to
ensure that explicit cache maintenance as required by non-coherent DMA
masters does not affect adjacent, unrelated slab allocations. On arm64,
this value is currently set at 128 bytes.

This means that applying CRYPTO_MINALIGN_ATTR to struct shash_desc is both
unnecessary (as it is never used for DMA), and undesirable, given that it
wastes stack space (on arm64, performing the alignment costs 112 bytes in
the worst case, and the hole between the 'tfm' and '__ctx' members takes
up another 120 bytes, resulting in an increased stack footprint of up to
232 bytes.) So instead, let's switch to the minimum SLAB alignment, which
does not take DMA constraints into account.

Note that this is a no-op for x86.

Signed-off-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - Remove AHASH_REQUEST_ON_STACK

This patch removes AHASH_REQUEST_ON_STACK which is unused.

Signed-off-by: Herbert Xu <[email protected]>
Acked-by: Ard Biesheuvel <ardb@kerne

crypto: ahash - Remove AHASH_REQUEST_ON_STACK

This patch removes AHASH_REQUEST_ON_STACK which is unused.

Signed-off-by: Herbert Xu <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: ahash - Add init_tfm/exit_tfm

This patch adds the type-safe init_tfm/exit_tfm functions to the
ahash interface. This is meant to replace the unsafe cra_init and
cra_exit interface.

Signed-

crypto: ahash - Add init_tfm/exit_tfm

This patch adds the type-safe init_tfm/exit_tfm functions to the
ahash interface. This is meant to replace the unsafe cra_init and
cra_exit interface.

Signed-off-by: Herbert Xu <[email protected]>

show more ...