dma-mapping: clearly mark DMA ops as an architecture feature

DMA ops are a helper for architectures and not for drivers to override
the DMA implementation.

Unfortunately driver authors keep ignorin

dma-mapping: clearly mark DMA ops as an architecture feature

DMA ops are a helper for architectures and not for drivers to override
the DMA implementation.

Unfortunately driver authors keep ignoring this. Make the fact more
clear by renaming the symbol to ARCH_HAS_DMA_OPS and having the two drivers
overriding their dma_ops depend on that. These drivers should probably be
marked broken, but we can give them a bit of a grace period for that.

Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Thomas Gleixner <[email protected]>
Acked-by: Sakari Ailus <[email protected]> # for IPU6
Acked-by: Robin Murphy <[email protected]>

show more ...

dma-mapping: direct calls for dma-iommu

Directly call into dma-iommu just like we have been doing for dma-direct
for a while. This avoids the indirect call overhead for IOMMU ops and
removes the ne

dma-mapping: direct calls for dma-iommu

Directly call into dma-iommu just like we have been doing for dma-direct
for a while. This avoids the indirect call overhead for IOMMU ops and
removes the need to have DMA ops entirely for many common configurations.

Signed-off-by: Leon Romanovsky <[email protected]>
Signed-off-by: Leon Romanovsky <[email protected]>
Acked-by: Greg Kroah-Hartman <[email protected]>
Acked-by: Robin Murphy <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>

show more ...

dma-mapping: remove CONFIG_DMA_REMAP

CONFIG_DMA_REMAP is used to build a few helpers around the core
vmalloc code, and to use them in case there is a highmem page in
dma-direct, and to make dma cohe

dma-mapping: remove CONFIG_DMA_REMAP

CONFIG_DMA_REMAP is used to build a few helpers around the core
vmalloc code, and to use them in case there is a highmem page in
dma-direct, and to make dma coherent allocations be able to use
non-contiguous pages allocations for DMA allocations in the dma-iommu
layer.

Right now it needs to be explicitly selected by architectures, and
is only done so by architectures that require remapping to deal
with devices that are not DMA coherent. Make it unconditional for
builds with CONFIG_MMU as it is very little extra code, but makes
it much more likely that large DMA allocations succeed on x86.

This fixes hot plugging a NVMe thunderbolt SSD for me, which tries
to allocate a 1MB buffer that is otherwise hard to obtain due to
memory fragmentation on a heavily used laptop.

Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>

show more ...

dma-mapping: add benchmark support for streaming DMA APIs

Nowadays, there are increasing requirements to benchmark the performance
of dma_map and dma_unmap particually while the device is attached t

dma-mapping: add benchmark support for streaming DMA APIs

Nowadays, there are increasing requirements to benchmark the performance
of dma_map and dma_unmap particually while the device is attached to an
IOMMU.

This patch enables the support. Users can run specified number of threads
to do dma_map_page and dma_unmap_page on a specific NUMA node with the
specified duration. Then dma_map_benchmark will calculate the average
latency for map and unmap.

A difficulity for this benchmark is that dma_map/unmap APIs must run on
a particular device. Each device might have different backend of IOMMU or
non-IOMMU.

So we use the driver_override to bind dma_map_benchmark to a particual
device by:
For platform devices:
echo dma_map_benchmark > /sys/bus/platform/devices/xxx/driver_override
echo xxx > /sys/bus/platform/drivers/xxx/unbind
echo xxx > /sys/bus/platform/drivers/dma_map_benchmark/bind

For PCI devices:
echo dma_map_benchmark > /sys/bus/pci/devices/0000:00:01.0/driver_override
echo 0000:00:01.0 > /sys/bus/pci/drivers/xxx/unbind
echo 0000:00:01.0 > /sys/bus/pci/drivers/dma_map_benchmark/bind

Cc: Will Deacon <[email protected]>
Cc: Shuah Khan <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: Marek Szyprowski <[email protected]>
Cc: Robin Murphy <[email protected]>
Signed-off-by: Barry Song <[email protected]>
[hch: folded in two fixes from Colin Ian King <[email protected]>]
Signed-off-by: Christoph Hellwig <[email protected]>

show more ...

dma-mapping: remove dma_virt_ops

Now that the RDMA core deals with devices that only do DMA mapping in
lower layers properly, there is no user for dma_virt_ops and it can be
removed.

Link: https://

dma-mapping: remove dma_virt_ops

Now that the RDMA core deals with devices that only do DMA mapping in
lower layers properly, there is no user for dma_virt_ops and it can be
removed.

Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Jason Gunthorpe <[email protected]>

show more ...

dma-mapping: move dma_common_{mmap,get_sgtable} out of mapping.c

Add a new file that contains helpers for misc DMA ops, which is only
built when CONFIG_DMA_OPS is set.

Signed-off-by: Christoph Hell

dma-mapping: move dma_common_{mmap,get_sgtable} out of mapping.c

Add a new file that contains helpers for misc DMA ops, which is only
built when CONFIG_DMA_OPS is set.

Signed-off-by: Christoph Hellwig <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>

show more ...

dma-mapping: make support for dma ops optional

Avoid the overhead of the dma ops support for tiny builds that only
use the direct mapping.

Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: A

dma-mapping: make support for dma ops optional

Avoid the overhead of the dma ops support for tiny builds that only
use the direct mapping.

Signed-off-by: Christoph Hellwig <[email protected]>
Tested-by: Alexey Kardashevskiy <[email protected]>
Reviewed-by: Alexey Kardashevskiy <[email protected]>

show more ...

dma-remap: separate DMA atomic pools from direct remap code

DMA atomic pools will be needed beyond only CONFIG_DMA_DIRECT_REMAP so
separate them out into their own file.

This also adds a new Kconfi

dma-remap: separate DMA atomic pools from direct remap code

DMA atomic pools will be needed beyond only CONFIG_DMA_DIRECT_REMAP so
separate them out into their own file.

This also adds a new Kconfig option that can be subsequently used for
options, such as CONFIG_AMD_MEM_ENCRYPT, that will utilize the coherent
pools but do not have a dependency on direct remapping.

For this patch alone, there is no functional change introduced.

Reviewed-by: Christoph Hellwig <[email protected]>
Signed-off-by: David Rientjes <[email protected]>
[hch: fixup copyrights and remove unused includes]
Signed-off-by: Christoph Hellwig <[email protected]>

show more ...

dma-mapping: improve selection of dma_declare_coherent availability

This API is primarily used through DT entries, but two architectures
and two drivers call it directly. So instead of selecting th

dma-mapping: improve selection of dma_declare_coherent availability

This API is primarily used through DT entries, but two architectures
and two drivers call it directly. So instead of selecting the config
symbol for random architectures pull it in implicitly for the actual
users. Also rename the Kconfig option to describe the feature better.

Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Paul Burton <[email protected]> # MIPS
Acked-by: Lee Jones <[email protected]>
Reviewed-by: Greg Kroah-Hartman <[email protected]>

show more ...

dma-mapping: factor out dummy DMA ops

The dummy DMA ops are currently used by arm64 for any device which has
an invalid ACPI description and is thus barred from using DMA due to not
knowing whether

dma-mapping: factor out dummy DMA ops

The dummy DMA ops are currently used by arm64 for any device which has
an invalid ACPI description and is thus barred from using DMA due to not
knowing whether is is cache-coherent or not. Factor these out into
general dma-mapping code so that they can be referenced from other
common code paths. In the process, we can prune all the optional
callbacks which just do the same thing as the default behaviour, and
fill in .map_resource for completeness.

Signed-off-by: Robin Murphy <[email protected]>
[hch: moved to a separate source file]
Reviewed-by: Rafael J. Wysocki <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>

show more ...

dma-mapping: always build the direct mapping code

All architectures except for sparc64 use the dma-direct code in some
form, and even for sparc64 we had the discussion of a direct mapping
mode a whi

dma-mapping: always build the direct mapping code

All architectures except for sparc64 use the dma-direct code in some
form, and even for sparc64 we had the discussion of a direct mapping
mode a while ago. In preparation for directly calling the direct
mapping code don't bother having it optionally but always build the
code in. This is a minor hardship for some powerpc and arm configs
that don't pull it in yet (although they should in a relase ot two),
and sparc64 which currently doesn't need it at all, but it will
reduce the ifdef mess we'd otherwise need significantly.

Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Jesper Dangaard Brouer <[email protected]>
Tested-by: Tony Luck <[email protected]>

show more ...

dma-mapping: move the remap helpers to a separate file

The dma remap code only makes sense for not cache coherent architectures
(or possibly the corner case of highmem CMA allocations) and currently

dma-mapping: move the remap helpers to a separate file

The dma remap code only makes sense for not cache coherent architectures
(or possibly the corner case of highmem CMA allocations) and currently
is only used by arm, arm64, csky and xtensa. Split it out into a
separate file with a separate Kconfig symbol, which gets the right
copyright notice given that this code was written by Laura Abbott
working for Code Aurora at that point.

Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Laura Abbott <[email protected]>
Reviewed-by: Robin Murphy <[email protected]>

show more ...

dma-mapping: merge direct and noncoherent ops

All the cache maintainance is already stubbed out when not enabled,
but merging the two allows us to nicely handle the case where
cache maintainance is

dma-mapping: merge direct and noncoherent ops

All the cache maintainance is already stubbed out when not enabled,
but merging the two allows us to nicely handle the case where
cache maintainance is required for some devices, but not others.

Signed-off-by: Christoph Hellwig <[email protected]>
Acked-by: Paul Burton <[email protected]> # MIPS parts

show more ...

dma-mapping: move all DMA mapping code to kernel/dma

Currently the code is split over various files with dma- prefixes in the
lib/ and drives/base directories, and the number of files keeps growing.

dma-mapping: move all DMA mapping code to kernel/dma

Currently the code is split over various files with dma- prefixes in the
lib/ and drives/base directories, and the number of files keeps growing.
Move them into a single directory to keep the code together and remove
the file name prefixes. To match the irq infrastructure this directory
is placed under the kernel/ directory.

Signed-off-by: Christoph Hellwig <[email protected]>

show more ...