crypto: qat - reorder objects in qat_common Makefile

The objects in the qat_common Makefile are currently listed in a random
order.

Reorder the objects alphabetically to make it easier to find wher

crypto: qat - reorder objects in qat_common Makefile

The objects in the qat_common Makefile are currently listed in a random
order.

Reorder the objects alphabetically to make it easier to find where to
add a new object.

Signed-off-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Ahsan Atta <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - fix object goals in Makefiles

Align with kbuild documentation by using <module_name>-y instead of
<module_name>-objs, following the kernel convention for building modules
from multiple

crypto: qat - fix object goals in Makefiles

Align with kbuild documentation by using <module_name>-y instead of
<module_name>-objs, following the kernel convention for building modules
from multiple object files.

Link: https://docs.kernel.org/kbuild/makefiles.html#loadable-module-goals-obj-m
Signed-off-by: Giovanni Cabiddu <[email protected]>
Suggested-by: Andy Shevchenko <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

module: Convert default symbol namespace to string literal

Commit cdd30ebb1b9f ("module: Convert symbol namespace to string
literal") only converted MODULE_IMPORT_NS() and EXPORT_SYMBOL_NS(),
leavin

module: Convert default symbol namespace to string literal

Commit cdd30ebb1b9f ("module: Convert symbol namespace to string
literal") only converted MODULE_IMPORT_NS() and EXPORT_SYMBOL_NS(),
leaving DEFAULT_SYMBOL_NAMESPACE as a macro expansion.

This commit converts DEFAULT_SYMBOL_NAMESPACE in the same way to avoid
annoyance for the default namespace as well.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Uwe Kleine-König <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

show more ...

crypto: qat - fix linking errors when PCI_IOV is disabled

When CONFIG_PCI_IOV=n, the build of the QAT vfio pci variant driver
fails reporting the following linking errors:

ERROR: modpost: "qat_

crypto: qat - fix linking errors when PCI_IOV is disabled

When CONFIG_PCI_IOV=n, the build of the QAT vfio pci variant driver
fails reporting the following linking errors:

ERROR: modpost: "qat_vfmig_open" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_resume" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_save_state" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_suspend" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_load_state" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_reset" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_save_setup" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_destroy" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_close" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
ERROR: modpost: "qat_vfmig_cleanup" [drivers/vfio/pci/qat/qat_vfio_pci.ko] undefined!
WARNING: modpost: suppressed 1 unresolved symbol warnings because there were too many)

Make live migration helpers provided by QAT PF driver always available
even if CONFIG_PCI_IOV is not selected. This does not cause any side
effect.

Reported-by: Arnd Bergmann <[email protected]>
Closes: https://lore.kernel.org/lkml/[email protected]/T/
Fixes: bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VF devices")
Signed-off-by: Xin Zeng <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Kevin Tian <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - implement interface for live migration

Add logic to implement the interface for live migration defined in
qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions
(VFs).

Thi

crypto: qat - implement interface for live migration

Add logic to implement the interface for live migration defined in
qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions
(VFs).

This introduces a migration data manager which is used to handle the
device state during migration. The manager ensures that the device state
is stored in a format that can be restored in the destination node.

The VF state is organized into a hierarchical structure that includes a
preamble, a general state section, a MISC bar section and an ETR bar
section. The latter contains the state of the 4 ring pairs contained on
a VF. Here is a graphical representation of the state:

preamble | general state section | leaf state
| MISC bar state section| leaf state
| ETR bar state section | bank0 state section | leaf state
| bank1 state section | leaf state
| bank2 state section | leaf state
| bank3 state section | leaf state

In addition to the implementation of the qat_migdev_ops interface and
the state manager framework, add a mutex in pfvf to avoid pf2vf messages
during migration.

Signed-off-by: Xin Zeng <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add interface for live migration

Extend the driver with a new interface to be used for VF live migration.
This allows to create and destroy a qat_mig_dev object that contains
a set of

crypto: qat - add interface for live migration

Extend the driver with a new interface to be used for VF live migration.
This allows to create and destroy a qat_mig_dev object that contains
a set of methods to allow to save and restore the state of QAT VF.
This interface will be used by the qat-vfio-pci module.

Signed-off-by: Xin Zeng <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - relocate CSR access code

As the common hw_data files are growing and the adf_hw_csr_ops is going
to be extended with new operations, move all logic related to ring CSRs
to the newly cr

crypto: qat - relocate CSR access code

As the common hw_data files are growing and the adf_hw_csr_ops is going
to be extended with new operations, move all logic related to ring CSRs
to the newly created adf_gen[2|4]_hw_csr_data.[c|h] files.

This does not introduce any functional change.

Signed-off-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Xin Zeng <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add heartbeat error simulator

Add a mechanism that allows to inject a heartbeat error for testing
purposes.
A new attribute `inject_error` is added to debugfs for each QAT device.
Upon

crypto: qat - add heartbeat error simulator

Add a mechanism that allows to inject a heartbeat error for testing
purposes.
A new attribute `inject_error` is added to debugfs for each QAT device.
Upon a write on this attribute, the driver will inject an error on the
device which can then be detected by the heartbeat feature.
Errors are breaking the device functionality thus they require a
device reset in order to be recovered.

This functionality is not compiled by default, to enable it
CRYPTO_DEV_QAT_ERROR_INJECTION must be set.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Ahsan Atta <[email protected]>
Reviewed-by: Markas Rapoportas <[email protected]>
Signed-off-by: Mun Chun Yep <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add support for device telemetry

Expose through debugfs device telemetry data for QAT GEN4 devices.

This allows to gather metrics about the performance and the utilization
of a device

crypto: qat - add support for device telemetry

Expose through debugfs device telemetry data for QAT GEN4 devices.

This allows to gather metrics about the performance and the utilization
of a device. In particular, statistics on (1) the utilization of the
PCIe channel, (2) address translation, when SVA is enabled and (3) the
internal engines for crypto and data compression.

If telemetry is supported by the firmware, the driver allocates a DMA
region and a circular buffer. When telemetry is enabled, through the
`control` attribute in debugfs, the driver sends to the firmware, via
the admin interface, the `TL_START` command. This triggers the device to
periodically gather telemetry data from hardware registers and write it
into the DMA memory region. The device writes into the shared region
every second.

The driver, every 500ms, snapshots the DMA shared region into the
circular buffer. This is then used to compute basic metric
(min/max/average) on each counter, every time the `device_data` attribute
is queried.

Telemetry counters are exposed through debugfs in the folder
/sys/kernel/debug/qat_<device>_<BDF>/telemetry.

For details, refer to debugfs-driver-qat_telemetry in Documentation/ABI.

This patch is based on earlier work done by Wojciech Ziemba.

Signed-off-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Damian Muszynski <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - relocate portions of qat_4xxx code

Move logic that is common between QAT GEN4 accelerators to the
qat_common folder. This includes addresses of CSRs, setters and
configuration logic.
W

crypto: qat - relocate portions of qat_4xxx code

Move logic that is common between QAT GEN4 accelerators to the
qat_common folder. This includes addresses of CSRs, setters and
configuration logic.
When moved, functions and defines have been renamed from 4XXX to GEN4.

Code specific to the device is moved to the file adf_gen4_hw_data.c.
Code related to configuration is moved to the newly created
adf_gen4_config.c.

This does not introduce any functional change.

Signed-off-by: Jie Wang <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - move adf_cfg_services

The file adf_cfg_services.h cannot be included in header files since it
instantiates the structure adf_cfg_services. Move that structure to its
own file and expor

crypto: qat - move adf_cfg_services

The file adf_cfg_services.h cannot be included in header files since it
instantiates the structure adf_cfg_services. Move that structure to its
own file and export the symbol.

This does not introduce any functional change.

Signed-off-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Damian Muszynski <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add rate limiting sysfs interface

Add an interface for the rate limiting feature which allows to add,
remove and modify a QAT SLA (Service Level Agreement).

This adds a new sysfs attr

crypto: qat - add rate limiting sysfs interface

Add an interface for the rate limiting feature which allows to add,
remove and modify a QAT SLA (Service Level Agreement).

This adds a new sysfs attribute group, `qat_rl`, which can be accessed
from /sys/bus/pci/devices/<BUS:DEV:FUNCTION> with the following
hierarchy:
|-+ qat_rl
|---- id (RW) # SLA identifier
|---- cir (RW) # Committed Information Rate
|---- pir (RW) # Peak Information Rate
|---- srv (RW) # Service to be rate limited
|---- rp (RW) (HEX) # Ring pairs to be rate limited
|---- cap_rem (RW) # Remaining capability for a service
|---- sla_op (WO) # Allows to perform an operation on an SLA

The API works by setting the appropriate RW attributes and then
issuing a command through the `sla_op`. For example, to create an SLA, a
user needs to input the necessary data into the attributes cir, pir, srv
and rp and then write into `sla_op` the command `add` to execute the
operation.
The API also provides `cap_rem` attribute to get information about
the remaining device capability within a certain service which is
required when setting an SLA.

Signed-off-by: Ciunas Bennett <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Damian Muszynski <[email protected]>
Reviewed-by: Tero Kristo <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add rate limiting feature to qat_4xxx

The Rate Limiting (RL) feature allows to control the rate of requests
that can be submitted on a ring pair (RP). This allows sharing a QAT
device

crypto: qat - add rate limiting feature to qat_4xxx

The Rate Limiting (RL) feature allows to control the rate of requests
that can be submitted on a ring pair (RP). This allows sharing a QAT
device among multiple users while ensuring a guaranteed throughput.

The driver provides a mechanism that allows users to set policies, that
are programmed to the device. The device is then enforcing those policies.

Configuration of RL is accomplished through entities called SLAs
(Service Level Agreement). Each SLA object gets a unique identifier
and defines the limitations for a single service across up to four
ring pairs (RPs count allocated to a single VF).

The rate is determined using two fields:
* CIR (Committed Information Rate), i.e., the guaranteed rate.
* PIR (Peak Information Rate), i.e., the maximum rate achievable
when the device has available resources.
The rate values are expressed in permille scale i.e. 0-1000.
Ring pair selection is achieved by providing a 64-bit mask, where
each bit corresponds to one of the ring pairs.

This adds an interface and logic that allow to add, update, retrieve
and remove an SLA.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Tero Kristo <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add error counters

Introduce ras counters interface for counting QAT specific device
errors and expose them through the newly created qat_ras sysfs
group attribute.

This adds the foll

crypto: qat - add error counters

Introduce ras counters interface for counting QAT specific device
errors and expose them through the newly created qat_ras sysfs
group attribute.

This adds the following attributes:

- errors_correctable: number of correctable errors
- errors_nonfatal: number of uncorrectable non fatal errors
- errors_fatal: number of uncorrectable fatal errors
- reset_error_counters: resets all counters

These counters are initialized during device bring up and cleared
during device shutdown and are applicable only to QAT GEN4 devices.

Signed-off-by: Shashank Gupta <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Tero Kristo <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add infrastructure for error reporting

Add infrastructure for enabling, disabling and reporting errors in the QAT
driver. This adds a new structure, adf_ras_ops, to adf_hw_device_data

crypto: qat - add infrastructure for error reporting

Add infrastructure for enabling, disabling and reporting errors in the QAT
driver. This adds a new structure, adf_ras_ops, to adf_hw_device_data that
contains the following methods:
- enable_ras_errors(): allows to enable RAS errors at device
initialization.
- disable_ras_errors(): allows to disable RAS errors at device shutdown.
- handle_interrupt(): allows to detect if there is an error and report if
a reset is required. This is executed immediately after the error is
reported, in the context of an ISR.

An initial, empty, implementation of the methods above is provided
for QAT GEN4.

Signed-off-by: Shashank Gupta <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Tero Kristo <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add cnv_errors debugfs file

The Compress and Verify (CnV) feature check and ensures data integrity
in the compression operation. The implementation of CnV keeps a record
of the CnV err

crypto: qat - add cnv_errors debugfs file

The Compress and Verify (CnV) feature check and ensures data integrity
in the compression operation. The implementation of CnV keeps a record
of the CnV errors that have occurred since the driver was loaded.

Expose CnV error stats by providing the "cnv_errors" file under
debugfs. This includes the number of errors detected up to now and
the type of the last error. The error count is provided on a per
Acceleration Engine basis and it is reset every time the driver is loaded.

Signed-off-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add pm_status debugfs file

QAT devices implement a mechanism that allows them to go autonomously
to a low power state depending on the load.

Expose power management info by providing

crypto: qat - add pm_status debugfs file

QAT devices implement a mechanism that allows them to go autonomously
to a low power state depending on the load.

Expose power management info by providing the "pm_status" file under
debugfs. This includes PM state, PM event log, PM event counters, PM HW
CSRs, per-resource type constrain counters and per-domain power gating
status specific to the QAT device.

This information is retrieved from (1) the FW by means of
ICP_QAT_FW_PM_INFO command, (2) CSRs and (3) counters collected by the
device driver.

In addition, add logic to keep track and report power management event
interrupts and acks/nacks sent to FW to allow/prevent state transitions.

Signed-off-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add namespace to driver

Create CRYPTO_QAT namespace for symbols exported by the qat_common
module and import those in the QAT drivers. It will reduce the global
namespace crowdedness a

crypto: qat - add namespace to driver

Create CRYPTO_QAT namespace for symbols exported by the qat_common
module and import those in the QAT drivers. It will reduce the global
namespace crowdedness and potential misuse or the API.

This does not introduce any functional change.

Suggested-by: Andy Shevchenko <[email protected]>
Signed-off-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add heartbeat feature

Under some circumstances, firmware in the QAT devices could become
unresponsive. The Heartbeat feature provides a mechanism to detect
unresponsive devices.

The Q

crypto: qat - add heartbeat feature

Under some circumstances, firmware in the QAT devices could become
unresponsive. The Heartbeat feature provides a mechanism to detect
unresponsive devices.

The QAT FW periodically writes to memory a set of counters that allow
to detect the liveness of a device. This patch adds logic to enable
the reporting of those counters, analyze them and report if a device
is alive or not.

In particular this adds
(1) heartbeat enabling, reading and detection logic
(2) reporting of heartbeat status and configuration via debugfs
(3) documentation for the newly created sysfs entries
(4) configuration of FW settings related to heartbeat, e.g. tick period
(5) logic to convert time in ms (provided by the user) to clock ticks

This patch introduces a new folder in debugfs called heartbeat with the
following attributes:
- status
- queries_sent
- queries_failed
- config

All attributes except config are reading only. In particular:
- `status` file returns 0 when device is operational and -1 otherwise.
- `queries_sent` returns the total number of heartbeat queries sent.
- `queries_failed` returns the total number of heartbeat queries failed.
- `config` allows to adjust the frequency at which the firmware writes
counters to memory. This period is given in milliseconds and it is
fixed for GEN4 devices.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add measure clock frequency

The QAT hardware does not expose a mechanism to report its clock
frequency. This is required to implement the Heartbeat feature.

Add a clock measuring algo

crypto: qat - add measure clock frequency

The QAT hardware does not expose a mechanism to report its clock
frequency. This is required to implement the Heartbeat feature.

Add a clock measuring algorithm that estimates the frequency by
comparing the internal timestamp counter incremented by the firmware
with the time measured by the kernel.
The frequency value is only used internally and not exposed to
the user.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add internal timer for qat 4xxx

The power management feature in QAT 4xxx devices can disable clock
sources used to implement timers. Because of that, the firmware needs to
get an exter

crypto: qat - add internal timer for qat 4xxx

The power management feature in QAT 4xxx devices can disable clock
sources used to implement timers. Because of that, the firmware needs to
get an external reliable source of time.

Add a kernel delayed work that periodically sends an event to the
firmware. This is triggered every 200ms. At each execution, the driver
sends a sync request to the firmware reporting the current timestamp
counter value.

This is a pre-requisite for enabling the heartbeat, telemetry and
rate limiting features.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - add fw_counters debugfs file

Expose FW counters statistics by providing the "fw_counters" file
under debugfs. Currently the statistics include the number of
requests sent to the FW and

crypto: qat - add fw_counters debugfs file

Expose FW counters statistics by providing the "fw_counters" file
under debugfs. Currently the statistics include the number of
requests sent to the FW and the number of responses received
from the FW for each Acceleration Engine, for all the QAT product
line.

This patch is based on earlier work done by Marco Chiappero.

Co-developed-by: Adam Guerin <[email protected]>
Signed-off-by: Adam Guerin <[email protected]>
Signed-off-by: Lucas Segarra Fernandez <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Reviewed-by: Andy Shevchenko <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - move dbgfs init to separate file

Move initialization of debugfs entries to a separate file.
This simplifies the exclusion of the debugfs logic in the QAT driver
when the kernel is buil

crypto: qat - move dbgfs init to separate file

Move initialization of debugfs entries to a separate file.
This simplifies the exclusion of the debugfs logic in the QAT driver
when the kernel is built with CONFIG_DEBUG_FS=n.
In addition, it will allow to consolidate the addition of debugfs
entries to a single location in the code.

This implementation adds infrastructure to create (and remove) debugfs
entries at two different stages. The first, done when a device is probed,
allows to keep debugfs entries persistent between a transition in device
state (up to down or vice versa). The second, done after the initialization
phase, allows to have debugfs entries that are accessible only when
the device is up.

In addition, move the creation of debugfs entries for configuration
to the newly created function adf_dbgfs_init() and replace symbolic
permissions with octal permissions when creating the debugfs files.
This is to resolve the following warning reported by checkpatch:

WARNING: Symbolic permissions 'S_IRUSR' are not preferred. Consider using octal permissions '0400'.

Signed-off-by: Damian Muszynski <[email protected]>
Reviewed-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Giovanni Cabiddu <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...

crypto: qat - Move driver to drivers/crypto/intel/qat

With the growing number of Intel crypto drivers, it makes sense to
group them all into a single drivers/crypto/intel/ directory.

Signed-off-by:

crypto: qat - Move driver to drivers/crypto/intel/qat

With the growing number of Intel crypto drivers, it makes sense to
group them all into a single drivers/crypto/intel/ directory.

Signed-off-by: Tom Zanussi <[email protected]>
Signed-off-by: Herbert Xu <[email protected]>

show more ...