crypto: api - Call crypto_alg_put in crypto_unregister_alg

Instead of calling cra_destroy by hand, call it through
crypto_alg_put so that the correct unwinding functions are called
through crypto_de

crypto: api - Call crypto_alg_put in crypto_unregister_alg

Instead of calling cra_destroy by hand, call it through
crypto_alg_put so that the correct unwinding functions are called
through crypto_destroy_alg.

Fixes: 3d6979bf3bd5 ("crypto: api - Add cra_type->destroy hook")
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Add request chaining API

This adds request chaining to the ahash interface. Request chaining
allows multiple requests to be submitted in one shot. An algorithm
can elect to receive

crypto: hash - Add request chaining API

This adds request chaining to the ahash interface. Request chaining
allows multiple requests to be submitted in one shot. An algorithm
can elect to receive chained requests by setting the flag
CRYPTO_ALG_REQ_CHAIN. If this bit is not set, the API will break
up chained requests and submit them one-by-one.

A new err field is added to struct crypto_async_request to record
the return value for each individual request.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Call crypto_schedule_test outside of mutex

There is no need to hold the crypto mutex when scheduling a self-
test. In fact prior to the patch introducing asynchronous testing,
this wa

crypto: api - Call crypto_schedule_test outside of mutex

There is no need to hold the crypto mutex when scheduling a self-
test. In fact prior to the patch introducing asynchronous testing,
this was done outside of the locked area.

Move the crypto_schedule_test call back out of the locked area.

Also move crypto_remove_final to the else branch under the schedule-
test call as the list of algorithms to be removed is non-empty only
when the test larval is NULL (i.e., testing is disabled).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Fix boot-up self-test race

During the boot process self-tests are postponed so that all
algorithms are registered when the test starts. In the event
that algorithms are still being re

crypto: api - Fix boot-up self-test race

During the boot process self-tests are postponed so that all
algorithms are registered when the test starts. In the event
that algorithms are still being registered during these tests,
which can occur either because the algorithm is registered at
late_initcall, or because a self-test itself triggers the creation
of an instance, some self-tests may never start at all.

Fix this by setting the flag at the start of crypto_start_tests.

Note that this race is theoretical and has never been observed
in practice.

Fixes: adad556efcdd ("crypto: api - Fix built-in testing dependency failures")
Signed-off-by: Herbert Xu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - move crypto_simd_disabled_for_test to lib

Move crypto_simd_disabled_for_test to lib/ so that crypto_simd_usable()
can be used by library code.

This was discussed previously
(https://l

crypto: api - move crypto_simd_disabled_for_test to lib

Move crypto_simd_disabled_for_test to lib/ so that crypto_simd_usable()
can be used by library code.

This was discussed previously
(https://lore.kernel.org/linux-crypto/[email protected]/)
but was not done because there was no use case yet. However, this is
now needed for the arm64 CRC32 library code.

Tested with:
export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
echo CONFIG_CRC32=y > .config
echo CONFIG_MODULES=y >> .config
echo CONFIG_CRYPTO=m >> .config
echo CONFIG_DEBUG_KERNEL=y >> .config
echo CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=n >> .config
echo CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y >> .config
make olddefconfig
make -j$(nproc)

Signed-off-by: Eric Biggers <[email protected]>
Acked-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Fix liveliness check in crypto_alg_tested

As algorithm testing is carried out without holding the main crypto
lock, it is always possible for the algorithm to go away during the
test.

crypto: api - Fix liveliness check in crypto_alg_tested

As algorithm testing is carried out without holding the main crypto
lock, it is always possible for the algorithm to go away during the
test.

So before crypto_alg_tested updates the status of the tested alg,
it checks whether it's still on the list of all algorithms. This
is inaccurate because it may be off the main list but still on the
list of algorithms to be removed.

Updating the algorithm status is safe per se as the larval still
holds a reference to it. However, killing spawns of other algorithms
that are of lower priority is clearly a deficiency as it adds
unnecessary churn.

Fix the test by checking whether the algorithm is dead.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Do not wait for tests during registration

As registration is usually carried out during module init, this
is a context where as little work as possible should be carried
out. Testing

crypto: api - Do not wait for tests during registration

As registration is usually carried out during module init, this
is a context where as little work as possible should be carried
out. Testing may trigger module loads of underlying components,
which could even lead back to the module that is registering at
the moment. This may lead to dead-locks outside of the Crypto API.

Avoid this by not waiting for the tests to complete. They will
be scheduled but completion will be asynchronous. Any users will
still wait for completion.

Reported-by: Russell King <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Remove instance larval fulfilment

In order to allow testing to complete asynchronously after the
registration process, instance larvals need to complete prior
to having a test result.

crypto: api - Remove instance larval fulfilment

In order to allow testing to complete asynchronously after the
registration process, instance larvals need to complete prior
to having a test result. Support this by redoing the lookup for
instance larvals after completion. This should locate the pending
test larval and then repeat the wait on that (if it is still pending).

As the lookup is now repeated there is no longer any need to compute
the fulfilment status and all that code can be removed.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Disable boot-test-finished if algapi is a module

The boot-test-finished toggle is only necessary if algapi
is built into the kernel. Do not include this code if it is a module.

Signe

crypto: api - Disable boot-test-finished if algapi is a module

The boot-test-finished toggle is only necessary if algapi
is built into the kernel. Do not include this code if it is a module.

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Disallow identical driver names

Disallow registration of two algorithms with identical driver names.

Cc: <[email protected]>
Reported-by: Ovidiu Panait <[email protected]

crypto: api - Disallow identical driver names

Disallow registration of two algorithms with identical driver names.

Cc: <[email protected]>
Reported-by: Ovidiu Panait <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Use work queue in crypto_destroy_instance

The function crypto_drop_spawn expects to be called in process
context. However, when an instance is unregistered while it still
has active u

crypto: api - Use work queue in crypto_destroy_instance

The function crypto_drop_spawn expects to be called in process
context. However, when an instance is unregistered while it still
has active users, the last user may cause the instance to be freed
in atomic context.

Fix this by delaying the freeing to a work queue.

Fixes: 6bfd48096ff8 ("[CRYPTO] api: Added spawns")
Reported-by: Florent Revest <[email protected]>
Reported-by: [email protected]
Reported-by: [email protected]
Signed-off-by: Herbert Xu <[email protected]>
Tested-by: Florent Revest <[email protected]>
Acked-by: Florent Revest <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: engine - fix crypto_queue backlog handling

CRYPTO_TFM_REQ_MAY_BACKLOG tells the crypto driver that it should
internally backlog requests until the crypto hw's queue becomes
full. At that poi

crypto: engine - fix crypto_queue backlog handling

CRYPTO_TFM_REQ_MAY_BACKLOG tells the crypto driver that it should
internally backlog requests until the crypto hw's queue becomes
full. At that point, crypto_engine backlogs the request and returns
-EBUSY. Calling driver such as dm-crypt then waits until the
complete() function is called with a status of -EINPROGRESS before
sending a new request.

The problem lies in the call to complete() with a value of -EINPROGRESS
that is made when a backlog item is present on the queue. The call is
done before the successful execution of the crypto request. In the case
that do_one_request() returns < 0 and the retry support is available,
the request is put back in the queue. This leads upper drivers to send
a new request even if the queue is still full.

The problem can be reproduced by doing a large dd into a crypto
dm-crypt device. This is pretty easy to see when using
Freescale CAAM crypto driver and SWIOTLB dma. Since the actual amount
of requests that can be hold in the queue is unlimited we get IOs error
and dma allocation.

The fix is to call complete with a value of -EINPROGRESS only if
the request is not enqueued back in crypto_queue. This is done
by calling complete() later in the code. In order to delay the decision,
crypto_queue is modified to correctly set the backlog pointer
when a request is enqueued back.

Fixes: 6a89f492f8e5 ("crypto: engine - support for parallel requests based on retry mechanism")
Co-developed-by: Sylvain Ouellet <[email protected]>
Signed-off-by: Sylvain Ouellet <[email protected]>
Signed-off-by: Olivier Bacon <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - Demote BUG_ON() in crypto_unregister_alg() to a WARN_ON()

The crypto_unregister_alg() function expects callers to ensure that any
algorithm that is unregistered has a refcnt of exactly

crypto: api - Demote BUG_ON() in crypto_unregister_alg() to a WARN_ON()

The crypto_unregister_alg() function expects callers to ensure that any
algorithm that is unregistered has a refcnt of exactly 1, and issues a
BUG_ON() if this is not the case. However, there are in fact drivers that
will call crypto_unregister_alg() without ensuring that the refcnt has been
lowered first, most notably on system shutdown. This causes the BUG_ON() to
trigger, which prevents a clean shutdown and hangs the system.

To avoid such hangs on shutdown, demote the BUG_ON() in
crypto_unregister_alg() to a WARN_ON() with early return. Cc stable because
this problem was observed on a 6.2 kernel, cf the link below.

Link: https://lore.kernel.org/r/[email protected]
Cc: [email protected]
Signed-off-by: Toke Høiland-Jørgensen <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: rng - Count error stats differently

Move all stat code specific to rng into the rng code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of

crypto: rng - Count error stats differently

Move all stat code specific to rng into the rng code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: skcipher - Count error stats differently

Move all stat code specific to skcipher into the skcipher code.

While we're at it, change the stats so that bytes and counts
are always incremented

crypto: skcipher - Count error stats differently

Move all stat code specific to skcipher into the skcipher code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: kpp - Count error stats differently

Move all stat code specific to kpp into the kpp code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of

crypto: kpp - Count error stats differently

Move all stat code specific to kpp into the kpp code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: acomp - Count error stats differently

Move all stat code specific to acomp into the acomp code.

While we're at it, change the stats so that bytes and counts
are always incremented even in c

crypto: acomp - Count error stats differently

Move all stat code specific to acomp into the acomp code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: hash - Count error stats differently

Move all stat code specific to hash into the hash code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case

crypto: hash - Count error stats differently

Move all stat code specific to hash into the hash code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: akcipher - Count error stats differently

Move all stat code specific to akcipher into the akcipher code.

While we're at it, change the stats so that bytes and counts
are always incremented

crypto: akcipher - Count error stats differently

Move all stat code specific to akcipher into the akcipher code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: aead - Count error stats differently

Move all stat code specific to aead into the aead code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case

crypto: aead - Count error stats differently

Move all stat code specific to aead into the aead code.

While we're at it, change the stats so that bytes and counts
are always incremented even in case of error. This allows the
reference counting to be removed as we can now increment the
counters prior to the operation.

After the operation we simply increase the error count if necessary.
This is safe as errors can only occur synchronously (or rather,
the existing code already ignored asynchronous errors which are
only visible to the callback function).

Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - compile out crypto_boot_test_finished when tests disabled

The crypto_boot_test_finished static key is unnecessary when self-tests
are disabled in the kconfig, so optimize it out accord

crypto: api - compile out crypto_boot_test_finished when tests disabled

The crypto_boot_test_finished static key is unnecessary when self-tests
are disabled in the kconfig, so optimize it out accordingly, along with
the entirety of crypto_start_tests(). This mainly avoids the overhead
of an unnecessary static_branch_enable() on every boot.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: algboss - optimize registration of internal algorithms

Since algboss always skips testing of algorithms with the
CRYPTO_ALG_INTERNAL flag, there is no need to go through the dance of
creatin

crypto: algboss - optimize registration of internal algorithms

Since algboss always skips testing of algorithms with the
CRYPTO_ALG_INTERNAL flag, there is no need to go through the dance of
creating the test kthread, which creates a lot of overhead. Instead, we
can just directly finish the algorithm registration, like is now done
when self-tests are disabled entirely.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: api - optimize algorithm registration when self-tests disabled

Currently, registering an algorithm with the crypto API always causes a
notification to be posted to the "cryptomgr", which the

crypto: api - optimize algorithm registration when self-tests disabled

Currently, registering an algorithm with the crypto API always causes a
notification to be posted to the "cryptomgr", which then creates a
kthread to self-test the algorithm. However, if self-tests are disabled
in the kconfig (as is the default option), then this kthread just
notifies waiters that the algorithm has been tested, then exits.

This causes a significant amount of overhead, especially in the kthread
creation and destruction, which is not necessary at all. For example,
in a quick test I found that booting a "minimum" x86_64 kernel with all
the crypto options enabled (except for the self-tests) takes about 400ms
until PID 1 can start. Of that, a full 13ms is spent just doing this
pointless dance, involving a kthread being created, run, and destroyed
over 200 times. That's over 3% of the entire kernel start time.

Fix this by just skipping the creation of the test larval and the
posting of the registration notification entirely, when self-tests are
disabled.

Signed-off-by: Eric Biggers <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: lib - move __crypto_xor into utils

CRYPTO_LIB_CHACHA depends on CRYPTO for __crypto_xor, defined in
crypto/algapi.c. This is a layering violation because the dependencies
should only go in

crypto: lib - move __crypto_xor into utils

CRYPTO_LIB_CHACHA depends on CRYPTO for __crypto_xor, defined in
crypto/algapi.c. This is a layering violation because the dependencies
should only go in the other direction (crypto/ => lib/crypto/). Also
the correct dependency would be CRYPTO_ALGAPI, not CRYPTO. Fix this by
moving __crypto_xor into the utils module in lib/crypto/.

Note that CRYPTO_LIB_CHACHA_GENERIC selected XOR_BLOCKS, which is
unrelated and unnecessary. It was perhaps thought that XOR_BLOCKS was
needed for __crypto_xor, but that's not the case.

Signed-off-by: Eric Biggers <[email protected]>
Reviewed-by: Jason A. Donenfeld <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: crypto_xor - use helpers for unaligned accesses

Dereferencing a misaligned pointer is undefined behavior in C, and may
result in codegen on architectures such as ARM that trigger alignments

crypto: crypto_xor - use helpers for unaligned accesses

Dereferencing a misaligned pointer is undefined behavior in C, and may
result in codegen on architectures such as ARM that trigger alignments
traps and expensive fixups in software.

Instead, use the get_aligned()/put_aligned() accessors, which are cheap
or even completely free when CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y.

In the converse case, the prior alignment checks ensure that the casts
are safe, and so no unaligned accessors are necessary.

Signed-off-by: Ard Biesheuvel <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...