drm/amdgpu: nuke the VM PD/PT shadow handling

This was only used as workaround for recovering the page tables after
VRAM was lost and is no longer necessary after the function
amdgpu_vm_bo_reset_sta

drm/amdgpu: nuke the VM PD/PT shadow handling

This was only used as workaround for recovering the page tables after
VRAM was lost and is no longer necessary after the function
amdgpu_vm_bo_reset_state_machine() started to do the same.

Compute never used shadows either, so the only proplematic case left is
SVM and that is most likely not recoverable in any way when VRAM is
lost.

Signed-off-by: Christian König <[email protected]>
Acked-by: Lijo Lazar <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: re-work VM syncing

Rework how VM operations synchronize to submissions. Provide an
amdgpu_sync container to the backends instead of an reservation
object and fill in the amdgpu_sync obje

drm/amdgpu: re-work VM syncing

Rework how VM operations synchronize to submissions. Provide an
amdgpu_sync container to the backends instead of an reservation
object and fill in the amdgpu_sync object in the higher layers
of the code.

No intended functional change, just prepares for upcomming changes.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Friedrich Vock <[email protected]>
Acked-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: fix potential resource leak warning

Clear resource leak warning that when the prepare fails,
the allocated amdgpu job object will never be released.

Signed-off-by: Tim Huang <Tim.Huang@

drm/amdgpu: fix potential resource leak warning

Clear resource leak warning that when the prepare fails,
the allocated amdgpu job object will never be released.

Signed-off-by: Tim Huang <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: implement TLB flush fence

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before t

drm/amdgpu: implement TLB flush fence

The problem is that when (for example) 4k pages are replaced
with a single 2M page we need to wait for change to be flushed
out by invalidating the TLB before the PT can be freed.

Solve this by moving the TLB flush into a DMA-fence object which
can be used to delay the freeing of the PT BOs until it is signaled.

V2: (Shashank)
- rebase
- set dma_fence_error only in case of error
- add tlb_flush fence only when PT/PD BO is locked (Felix)
- use vm->pasid when f is NULL (Mukul)

V4: - add a wait for (f->dependency) in tlb_fence_work (Christian)
- move the misplaced fence_create call to the end (Philip)

V5: - free the f->dependency properly

V6: (Shashank)
- light code movement, moved all the clean-up in previous patch
- introduce params.needs_flush and its usage in this patch
- rebase without TLB HW sequence patch

V7:
- Keep the vm->last_update_fence and tlb_cb code until
we can fix the HW sequencing (Christian)
- Move all the tlb_fence related code in a separate function so that
its easier to read and review

V9: Addressed review comments from Christian
- start PT update only when we have callback memory allocated

V10:
- handle device unlock in OOM case (Christian, Mukul)
- added Christian's R-B

Cc: Christian Koenig <[email protected]>
Cc: Felix Kuehling <[email protected]>
Cc: Rajneesh Bhardwaj <[email protected]>
Cc: Alex Deucher <[email protected]>
Acked-by: Felix Kuehling <[email protected]>
Acked-by: Rajneesh Bhardwaj <[email protected]>
Tested-by: Rajneesh Bhardwaj <[email protected]>
Reviewed-by: Shashank Sharma <[email protected]>
Reviewed-by: Christian Koenig <[email protected]>
Signed-off-by: Christian Koenig <[email protected]>
Signed-off-by: Shashank Sharma <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: drop the extra sign extension

amdgpu_bo_gpu_offset_no_check() already calls
amdgpu_gmc_sign_extend() so no need to call it twice.

Reviewed-by: Yang Wang <[email protected]>
Reviewe

drm/amdgpu: drop the extra sign extension

amdgpu_bo_gpu_offset_no_check() already calls
amdgpu_gmc_sign_extend() so no need to call it twice.

Reviewed-by: Yang Wang <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: grab extra fence reference for drm_sched_job_add_dependency

That function consumes the reference.

Reviewed-by: Luben Tuikov <[email protected]>
Reported-by: Borislav Petkov (AMD) <bp

drm/amdgpu: grab extra fence reference for drm_sched_job_add_dependency

That function consumes the reference.

Reviewed-by: Luben Tuikov <[email protected]>
Reported-by: Borislav Petkov (AMD) <[email protected]>
Tested-by: Borislav Petkov (AMD) <[email protected]>
Signed-off-by: Christian König <[email protected]>
Fixes: aab9cf7b6954 ("drm/amdgpu: use scheduler dependencies for VM updates")
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: use scheduler dependencies for VM updates

Instead of putting that into the job sync object.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <luben.tu

drm/amdgpu: use scheduler dependencies for VM updates

Instead of putting that into the job sync object.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: cleanup scheduler job initialization v2

Init the DRM scheduler base class while allocating the job.

This makes the whole handling much more cleaner.

v2: fix coding style

Signed-off-by

drm/amdgpu: cleanup scheduler job initialization v2

Init the DRM scheduler base class while allocating the job.

This makes the whole handling much more cleaner.

v2: fix coding style

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: use DRM_SCHED_FENCE_DONT_PIPELINE for VM updates

Make sure that we always have a CPU round trip to let the submission
code correctly decide if a TLB flush is necessary or not.

Signed-of

drm/amdgpu: use DRM_SCHED_FENCE_DONT_PIPELINE for VM updates

Make sure that we always have a CPU round trip to let the submission
code correctly decide if a TLB flush is necessary or not.

Signed-off-by: Christian König <[email protected]>
Bug: https://gitlab.freedesktop.org/drm/amd/-/issues/2113#note_1579296
Reviewed-by: Alex Deucher <[email protected]>
Acked-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: SDMA update use unlocked iterator

SDMA update page table may be called from unlocked context, this
generate below warning. Use unlocked iterator to handle this case.

WARNING: CPU: 0 PID

drm/amdgpu: SDMA update use unlocked iterator

SDMA update page table may be called from unlocked context, this
generate below warning. Use unlocked iterator to handle this case.

WARNING: CPU: 0 PID: 1475 at
drivers/dma-buf/dma-resv.c:483 dma_resv_iter_next
Call Trace:
dma_resv_iter_first+0x43/0xa0
amdgpu_vm_sdma_update+0x69/0x2d0 [amdgpu]
amdgpu_vm_ptes_update+0x29c/0x870 [amdgpu]
amdgpu_vm_update_range+0x2f6/0x6c0 [amdgpu]
svm_range_unmap_from_gpus+0x115/0x300 [amdgpu]
svm_range_cpu_invalidate_pagetables+0x510/0x5e0 [amdgpu]
__mmu_notifier_invalidate_range_start+0x1d3/0x230
unmap_vmas+0x140/0x150
unmap_region+0xa8/0x110

Signed-off-by: Philip Yang <[email protected]>
Suggested-by: Felix Kuehling <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: use DMA_RESV_USAGE_BOOKKEEP v2

Use DMA_RESV_USAGE_BOOKKEEP for VM page table updates and KFD preemption fence.

v2: actually update all usages for KFD

Signed-off-by: Christian König <ch

drm/amdgpu: use DMA_RESV_USAGE_BOOKKEEP v2

Use DMA_RESV_USAGE_BOOKKEEP for VM page table updates and KFD preemption fence.

v2: actually update all usages for KFD

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/ttm: remove bo->moving

This is now handled by the DMA-buf framework in the dma_resv obj.

Also remove the workaround inside VMWGFX to update the moving fence.

Signed-off-by: Christian König <ch

drm/ttm: remove bo->moving

This is now handled by the DMA-buf framework in the dma_resv obj.

Also remove the workaround inside VMWGFX to update the moving fence.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Daniel Vetter <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: Correct unlocked update fence handling

To fix two issues with unlocked update fence:

1. vm->last_unlocked store the latest fence without taking refcount.
2. amdgpu_vm_bo_update_mapping

drm/amdgpu: Correct unlocked update fence handling

To fix two issues with unlocked update fence:

1. vm->last_unlocked store the latest fence without taking refcount.
2. amdgpu_vm_bo_update_mapping returns old fence, not the latest fence.

Signed-off-by: Philip Yang <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: remove amdgpu_vm_pt

Page table entries are now in embedded in VM BO, so
we do not need struct amdgpu_vm_pt. This patch replaces
struct amdgpu_vm_pt with struct amdgpu_vm_bo_base.

Signed

drm/amdgpu: remove amdgpu_vm_pt

Page table entries are now in embedded in VM BO, so
we do not need struct amdgpu_vm_pt. This patch replaces
struct amdgpu_vm_pt with struct amdgpu_vm_bo_base.

Signed-off-by: Nirmoy Das <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: switch to amdgpu_bo_vm for vm code

The subclass, amdgpu_bo_vm is intended for PT/PD BOs which are also
shadowed, so switch to amdgpu_bo_vm BO for PT/PD BOs.

v4: update amdgpu_vm_update_

drm/amdgpu: switch to amdgpu_bo_vm for vm code

The subclass, amdgpu_bo_vm is intended for PT/PD BOs which are also
shadowed, so switch to amdgpu_bo_vm BO for PT/PD BOs.

v4: update amdgpu_vm_update_funcs to accept amdgpu_bo_vm.
v3: simplify code.
check also if shadow bo exist instead of checking bo only type.
v2: squash three related patches.

Signed-off-by: Nirmoy Das <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amd/amdgpu/amdgpu_vm_sdma: Fix 'amdgpu_vm_sdma_prepare()'s doc-rot

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c:63: warning: Function parameter

drm/amd/amdgpu/amdgpu_vm_sdma: Fix 'amdgpu_vm_sdma_prepare()'s doc-rot

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c:63: warning: Function parameter or member 'resv' not described in 'amdgpu_vm_sdma_prepare'
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c:63: warning: Function parameter or member 'sync_mode' not described in 'amdgpu_vm_sdma_prepare'
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c:63: warning: Excess function parameter 'owner' description in 'amdgpu_vm_sdma_prepare'
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c:63: warning: Excess function parameter 'exclusive' description in 'amdgpu_vm_sdma_prepare'

Cc: Alex Deucher <[email protected]>
Cc: "Christian König" <[email protected]>
Cc: David Airlie <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Sumit Semwal <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Signed-off-by: Lee Jones <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: More accurate description of a function param

Add more accurate description of the pe parameter of function
amdgpu_vm_sdma_udpate and amdgpu_vm_cpu_update

Signed-off-by: Oak Zeng <Oak.Z

drm/amdgpu: More accurate description of a function param

Add more accurate description of the pe parameter of function
amdgpu_vm_sdma_udpate and amdgpu_vm_cpu_update

Signed-off-by: Oak Zeng <[email protected]>
Reviewed-by: Christian Konig <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: remove distinction between explicit and implicit sync (v2)

According to Marek a pipeline sync should be inserted for implicit syncs well.

v2: bump the driver version

Signed-off-by: Chr

drm/amdgpu: remove distinction between explicit and implicit sync (v2)

According to Marek a pipeline sync should be inserted for implicit syncs well.

v2: bump the driver version

Signed-off-by: Christian König <[email protected]>
Tested-by: Marek Olšák <[email protected]>
Signed-off-by: Marek Olšák <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: move ttm bo->offset to amdgpu_bo

GPU address should belong to driver not in memory management.
This patch moves ttm bo.offset and gpu_offset calculation to amdgpu driver.

Signed-off-by:

drm/amdgpu: move ttm bo->offset to amdgpu_bo

GPU address should belong to driver not in memory management.
This patch moves ttm bo.offset and gpu_offset calculation to amdgpu driver.

Signed-off-by: Nirmoy Das <[email protected]>
Acked-by: Huang Rui <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Christian König <[email protected]>
Link: https://patchwork.freedesktop.org/patch/372930/

show more ...

drm/amdgpu: add new unlocked flag for PTE updates

For HMM support we need the ability to invalidate PTEs from
a MM callback where we can't lock the root PD.

Add a new flag to better support this in

drm/amdgpu: add new unlocked flag for PTE updates

For HMM support we need the ability to invalidate PTEs from
a MM callback where we can't lock the root PD.

Add a new flag to better support this instead of assuming
that all invalidation updates are unlocked.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: rename direct to immediate for VM updates

To avoid confusion with direct ring submissions rename bottom
of pipe VM table changes to immediate updates.

Signed-off-by: Christian König <ch

drm/amdgpu: rename direct to immediate for VM updates

To avoid confusion with direct ring submissions rename bottom
of pipe VM table changes to immediate updates.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: cleanup IB pool handling a bit

Fix the coding style, move and rename the definitions to
better match what they are supposed to be doing.

Signed-off-by: Christian König <christian.koenig

drm/amdgpu: cleanup IB pool handling a bit

Fix the coding style, move and rename the definitions to
better match what they are supposed to be doing.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: implement more ib pools (v2)

We have three ib pools, they are normal, VM, direct pools.

Any jobs which schedule IBs without dependence on gpu scheduler should
use DIRECT pool.

Any jobs

drm/amdgpu: implement more ib pools (v2)

We have three ib pools, they are normal, VM, direct pools.

Any jobs which schedule IBs without dependence on gpu scheduler should
use DIRECT pool.

Any jobs schedule direct VM update IBs should use VM pool.

Any other jobs use NORMAL pool.

v2: squash in coding style fix

Signed-off-by: xinhui pan <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm_amdgpu: Add job fence to resv conditionally

Job fence on page table should be a shared one, so add it to the root
page talbe bo resv.
last_delayed field is not needed anymore. so remove it.

Cc:

drm_amdgpu: Add job fence to resv conditionally

Job fence on page table should be a shared one, so add it to the root
page talbe bo resv.
last_delayed field is not needed anymore. so remove it.

Cc: Christian König <[email protected]>
Cc: Alex Deucher <[email protected]>
Cc: Felix Kuehling <[email protected]>
Suggested-by: Christian König <[email protected]>
Signed-off-by: xinhui pan <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: rework synchronization of VM updates v4

If provided we only sync to the BOs reservation
object and no longer to the root PD.

v2: update comment, cleanup amdgpu_bo_sync_wait_resv
v3: use

drm/amdgpu: rework synchronization of VM updates v4

If provided we only sync to the BOs reservation
object and no longer to the root PD.

v2: update comment, cleanup amdgpu_bo_sync_wait_resv
v3: use correct reservation object while clearing
v4: fix typo in amdgpu_bo_sync_wait_resv

Signed-off-by: Christian König <[email protected]>
Tested-by: Tom St Denis <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...