drm/amdgpu: Optimize VM invalidation engine allocation and synchronize GPU TLB flush

- Modify the VM invalidation engine allocation logic to handle SDMA page rings.
SDMA page rings now share the V

drm/amdgpu: Optimize VM invalidation engine allocation and synchronize GPU TLB flush

- Modify the VM invalidation engine allocation logic to handle SDMA page rings.
SDMA page rings now share the VM invalidation engine with SDMA gfx rings instead of
allocating a separate engine. This change ensures efficient resource management and
avoids the issue of insufficient VM invalidation engines.

- Add synchronization for GPU TLB flush operations in gmc_v9_0.c.
Use spin_lock and spin_unlock to ensure thread safety and prevent race conditions
during TLB flush operations. This improves the stability and reliability of the driver,
especially in multi-threaded environments.

v2: replace the sdma ring check with a function `amdgpu_sdma_is_page_queue`
to check if a ring is an SDMA page queue.(Lijo)

v3: Add GC version check, only enabled on GC9.4.3/9.4.4/9.5.0
v4: Fix code style and add more detailed description (Christian)
v5: Remove dependency on vm_inv_eng loop order, explicitly lookup shared inv_eng(Christian/Lijo)
v6: Added search shared ring function amdgpu_sdma_get_shared_ring (Lijo)

Suggested-by: Lijo Lazar <[email protected]>
Signed-off-by: Jesse Zhang <[email protected]>
Reviewed-by: Lijo Lazar <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu/sdma: guilty tracking is per instance

The gfx and page queues are per instance, so track them
per instance.

v2: drop extra parameter (Lijo)

Fixes: fdbfaaaae06b ("drm/amdgpu: Improve SDM

drm/amdgpu/sdma: guilty tracking is per instance

The gfx and page queues are per instance, so track them
per instance.

v2: drop extra parameter (Lijo)

Fixes: fdbfaaaae06b ("drm/amdgpu: Improve SDMA reset logic with guilty queue tracking")
Reviewed-by: Lijo Lazar <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu/sdma: fix engine reset handling

Move the kfd suspend/resume code into the caller. That
is where the KFD is likely to detect a reset so on the KFD
side there is no need to call them. Als

drm/amdgpu/sdma: fix engine reset handling

Move the kfd suspend/resume code into the caller. That
is where the KFD is likely to detect a reset so on the KFD
side there is no need to call them. Also add a mutex to
lock the actual reset sequence.

v2: make the locking per instance

Fixes: bac38ca8c475 ("drm/amdkfd: implement per queue sdma reset for gfx 9.4+")
Reviewed-by: Jesse Zhang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Improve SDMA reset logic with guilty queue tracking

This patch includes the remaining improvements to the SDMA reset logic:
- Added `gfx_guilty` and `page_guilty` flags to track guilty q

drm/amdgpu: Improve SDMA reset logic with guilty queue tracking

This patch includes the remaining improvements to the SDMA reset logic:
- Added `gfx_guilty` and `page_guilty` flags to track guilty queues.
- Updated the reset and resume functions to handle the guilty state.
- Cached the `rptr` before reset.

v2:
1.replace the caller with a guilty bool.
If the queue is the guilty one, set the rptr and wptr to the saved wptr value,
else, set the rptr and wptr to the saved rptr value. (Alex)
2. cache the rptr before the reset. (Alex)

v3: Keeping intermediate variables like u64 rwptr simplifies resotre rptr/wptr.(Lijo)

Suggested-by: Alex Deucher <[email protected]>
Suggested-by: Jiadong Zhu <[email protected]>
Signed-off-by: Jesse Zhang <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Introduce conditional user queue suspension for SDMA resets

- Modify the `amdgpu_sdma_reset_engine` function to accept a `suspend_user_queues` parameter.
- This parameter allows the func

drm/amdgpu: Introduce conditional user queue suspension for SDMA resets

- Modify the `amdgpu_sdma_reset_engine` function to accept a `suspend_user_queues` parameter.
- This parameter allows the function to conditionally suspend and resume user queues during SDMA resets.
- Ensure that user queues are suspended only when necessary to avoid unnecessary overhead and potential deadlocks.
- Restart the scheduler's work queue for the GFX and page rings after the reset to allow new tasks to be submitted.

This change improves synchronization between the KGD and the KFD during SDMA resets,
ensuring proper handling of user queues and avoiding race conditions.

V2: replace the ring_lock with the existed the scheduler
locks for the queues (ring->sched) on the sdma engine.(Alex)

v3: call drm_sched_wqueue_stop() rather than job_list_lock.
If a GPU ring reset was already initiated for one ring at amdgpu_job_timedout,
skip resetting that ring and call drm_sched_wqueue_stop()
for the other rings (Alex)

replace the common lock (sdma_reset_lock) with DQM lock to
to resolve reset races between the two driver sections during KFD eviction.(Jon)

Rename the caller to Reset_src and
Change AMDGPU_RESET_SRC_SDMA_KGD/KFD to AMDGPU_RESET_SRC_SDMA_HWS/RING (Jon)

v4: restart the wqueue if the reset was successful,
or fall back to a full adapter reset. (Alex)

move definition of reset source to enumeration AMDGPU_RESET_SRCS, and
check reset src in amdgpu_sdma_reset_instance (Jon)

v5: Call amdgpu_amdkfd_suspend/resume at the start/end of reset function respectively under !SRC_HWS
conditions only (Jon)

v6: replace the paramter src with a bool suspend_user_queues,
remove the paramter src in pre/post func. (Jon)

Suggested-by: Alex Deucher <[email protected]>
Suggested-by: Jiadong Zhu <[email protected]>
Suggested-by: Jonathan Kim <[email protected]>
Signed-off-by: Jesse Zhang <[email protected]>
Acked-by: Jonathan Kim <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu/kfd: Add shared SDMA reset functionality with callback support

This patch introduces shared SDMA reset functionality between AMDGPU and KFD.
The implementation includes the following key

drm/amdgpu/kfd: Add shared SDMA reset functionality with callback support

This patch introduces shared SDMA reset functionality between AMDGPU and KFD.
The implementation includes the following key changes:

1. Added `amdgpu_sdma_reset_queue`:
- Resets a specific SDMA queue by instance ID.
- Invokes registered pre-reset and post-reset callbacks to allow KFD and AMDGPU
to save/restore their state during the reset process.

2. Added `amdgpu_set_on_reset_callbacks`:
- Allows KFD and AMDGPU to register callback functions for pre-reset and
post-reset operations.
- Callbacks are stored in a global linked list and invoked in the correct order
during SDMA reset.

This patch ensures that both AMDGPU and KFD can handle SDMA reset events
gracefully, with proper state saving and restoration. It also provides a flexible
callback mechanism for future extensions.

v2: fix CamelCase and put the SDMA helper into amdgpu_sdma.c (Alex)

v3: rename the `amdgpu_register_on_reset_callbacks` function to
`amdgpu_sdma_register_on_reset_callbacks`
move global reset_callback_list to struct amdgpu_sdma (Alex)

v4: Update the reset callback function description and
rename the reset function to amdgpu_sdma_reset_engine (Alex)

Suggested-by: Alex Deucher <[email protected]>
Suggested-by: Jiadong Zhu <[email protected]>
Signed-off-by: Jesse Zhang <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add handler for SDMA context empty

Context empty interrupt is enabled for SDMA 4.4.2. Add a handler for
context empty interrupt so that it is disposed of fast, and not
propagated to KFD

drm/amdgpu: Add handler for SDMA context empty

Context empty interrupt is enabled for SDMA 4.4.2. Add a handler for
context empty interrupt so that it is disposed of fast, and not
propagated to KFD layer.

Signed-off-by: Lijo Lazar <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Suggested-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add sysfs interface for sdma reset mask

Add the sysfs interface for sdma:
sdma_reset_mask

The interface is read-only and show the resets supported by the IP.
For example, full adapter r

drm/amdgpu: Add sysfs interface for sdma reset mask

Add the sysfs interface for sdma:
sdma_reset_mask

The interface is read-only and show the resets supported by the IP.
For example, full adapter reset (mode1/mode2/BACO/etc),
soft reset, queue reset, and pipe reset.

V2: the sysfs node returns a text string instead of some flags (Christian)
v3: add a generic helper which takes the ring as parameter
and print the strings in the order they are applied (Christian)

check amdgpu_gpu_recovery before creating sysfs file itself,
and initialize supported_reset_types in IP version files (Lijo)

Signed-off-by: Jesse Zhang <[email protected]>
Suggested-by: Alex Deucher <[email protected]>
Reviewed-by: Tim Huang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add amdgpu_sdma_sched_mask debugfs

Userspace wants to run jobs on a specific sdma ring for verification purposes.
This debugfs entry helps to disable or enable submitting jobs to a speci

drm/amdgpu: add amdgpu_sdma_sched_mask debugfs

Userspace wants to run jobs on a specific sdma ring for verification purposes.
This debugfs entry helps to disable or enable submitting jobs to a specific ring.
This entry is populated only if there are at least two or more cores in the sdma ip.

Signed-off-by: Jesse Zhang <[email protected]>
Suggested-by: Alex Deucher <[email protected]>
Reviewed-by: Tim Huang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add sdma_v5_2 ip dump for devcoredump

Add ip dump for sdma_v5_2 for devcoredump for all
instances of sdma.

Signed-off-by: Sunil Khatri <[email protected]>
Reviewed-by: Alex Deucher <

drm/amdgpu: Add sdma_v5_2 ip dump for devcoredump

Add ip dump for sdma_v5_2 for devcoredump for all
instances of sdma.

Signed-off-by: Sunil Khatri <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add new members for sdma v7_0 fw

Add new members in sdma instance structure
for sdma v7_0 firmware.

Signed-off-by: Likun Gao <[email protected]>
Reviewed-by: Hawking Zhang <Hawking.Zhan

drm/amdgpu: Add new members for sdma v7_0 fw

Add new members in sdma instance structure
for sdma v7_0 firmware.

Signed-off-by: Likun Gao <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: replace tmz flag into buffer flag

Replace tmz flag into buffer flag to make it easier to understand
and extend

Signed-off-by: Likun Gao <[email protected]>
Signed-off-by: Frank Min <Fra

drm/amdgpu: replace tmz flag into buffer flag

Replace tmz flag into buffer flag to make it easier to understand
and extend

Signed-off-by: Likun Gao <[email protected]>
Signed-off-by: Frank Min <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: move buffer funcs setting up a level

Rather than doing this in the IP code for the SDMA paging
engine, move it up to the core device level init level.
This should fix the scheduler init

drm/amdgpu: move buffer funcs setting up a level

Rather than doing this in the IP code for the SDMA paging
engine, move it up to the core device level init level.
This should fix the scheduler init ordering.

v2: drop extra parens
v3: drop SDMA helpers
v4: Added a Fixes tag because amdgpu dereferences an uninitialized
scheduler without this patch, and this patch fixes this. (Luben)

Tested-by: Luben Tuikov <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Acked-by: Christian König <[email protected]>
Fixes: 56e449603f0ac5 ("drm/sched: Convert the GPU scheduler to variable number of run-queues")
Signed-off-by: Luben Tuikov <[email protected]>

show more ...

drm/amdgpu: Add query_ras_error_count for sdma v4_4_2

Add query_ras_error_count callback for sdma
v4_4_2. It will be used to query and log sdma
uncorrectable error count and memory block.

Signed-of

drm/amdgpu: Add query_ras_error_count for sdma v4_4_2

Add query_ras_error_count callback for sdma
v4_4_2. It will be used to query and log sdma
uncorrectable error count and memory block.

Signed-off-by: Hawking Zhang <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add mask for SDMA instances

Add a mask of SDMA instances available for use. On certain ASIC configs,
not all SDMA instances are available for software use.

v2:
Change sdma mask type to

drm/amdgpu: Add mask for SDMA instances

Add a mask of SDMA instances available for use. On certain ASIC configs,
not all SDMA instances are available for software use.

v2:
Change sdma mask type to uint32_t (Le)

Signed-off-by: Lijo Lazar <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Reviewed-by: Le Ma <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add support for SDMA on multiple AIDs

Initialize SDMA instances on each AID.

v2: revise coding fault in hw_fini

Signed-off-by: Le Ma <[email protected]>
Acked-by: Felix Kuehling <Felix.Kue

drm/amdgpu: add support for SDMA on multiple AIDs

Initialize SDMA instances on each AID.

v2: revise coding fault in hw_fini

Signed-off-by: Le Ma <[email protected]>
Acked-by: Felix Kuehling <[email protected]>
Reviewed-by: Lijo Lazar <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: adjust some basic elements for multiple AID case

add some elements below:
- num_aid
- aid_id for each sdma instance
- num_inst_per_aid for sdma

and extend macro size below:
- SDMA_M

drm/amdgpu: adjust some basic elements for multiple AID case

add some elements below:
- num_aid
- aid_id for each sdma instance
- num_inst_per_aid for sdma

and extend macro size below:
- SDMA_MAX_INSTANCES to 16
- AMDGPU_MAX_RINGS to 96
- AMDGPU_MAX_HWIP_RINGS to 32

v2: move aid_id from amdgpu_ring to amdgpu_sdma_instance. (Lijo)

Signed-off-by: Le Ma <[email protected]>
Acked-by: Felix Kuehling <[email protected]>
Reviewed-by: Lijo Lazar <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Add sdma ras function on sdma v6_0_3

Add sdma ras function on sdma v6_0_3.

Signed-off-by: YiPeng Chai <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex

drm/amdgpu: Add sdma ras function on sdma v6_0_3

Add sdma ras function on sdma v6_0_3.

Signed-off-by: YiPeng Chai <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amd: Convert SDMA to use `amdgpu_ucode_ip_version_decode`

Simplifies the code so that all SDMA versions will get the firmware
name from `amdgpu_ucode_ip_version_decode`.

v2: squash in fix from

drm/amd: Convert SDMA to use `amdgpu_ucode_ip_version_decode`

Simplifies the code so that all SDMA versions will get the firmware
name from `amdgpu_ucode_ip_version_decode`.

v2: squash in fix from Srinivasan

Reviewed-by: Alex Deucher <[email protected]>
Reviewed-by: Lijo Lazar <[email protected]>
Signed-off-by: Mario Limonciello <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: switch sdma buffer function tear down to a helper

Switch all of the SDMA implementations to use the helper to
tear down the ttm buffer manager.

Tested-by: Bokun Zhang <[email protected]

drm/amdgpu: switch sdma buffer function tear down to a helper

Switch all of the SDMA implementations to use the helper to
tear down the ttm buffer manager.

Tested-by: Bokun Zhang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add function to init SDMA microcode

Add an common function to init SDMA related microcode.

Signed-off-by: Likun Gao <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]

drm/amdgpu: add function to init SDMA microcode

Add an common function to init SDMA related microcode.

Signed-off-by: Likun Gao <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Remove redundant calls of amdgpu_ras_block_late_fini in sdma ras block

Remove redundant calls of amdgpu_ras_block_late_fini in sdma ras block.

Signed-off-by: yipechai <[email protected]

drm/amdgpu: Remove redundant calls of amdgpu_ras_block_late_fini in sdma ras block

Remove redundant calls of amdgpu_ras_block_late_fini in sdma ras block.

Signed-off-by: yipechai <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Modify .ras_fini function pointer parameter

Modify .ras_fini function pointer parameter so that
we can remove redundant intermediate calls in some
ras blocks.

Signed-off-by: yipechai <Y

drm/amdgpu: Modify .ras_fini function pointer parameter

Modify .ras_fini function pointer parameter so that
we can remove redundant intermediate calls in some
ras blocks.

Signed-off-by: yipechai <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Modify .ras_late_init function pointer parameter

Modify .ras_late_init function pointer parameter so that
it can remove redundant intermediate calls in some ras blocks.

Signed-off-by: y

drm/amdgpu: Modify .ras_late_init function pointer parameter

Modify .ras_late_init function pointer parameter so that
it can remove redundant intermediate calls in some ras blocks.

Signed-off-by: yipechai <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Modify sdma block to fit for the unified ras block data and ops

1.Modify sdma block to fit for the unified ras block data and ops.
2.Change amdgpu_sdma_ras_funcs to amdgpu_sdma_ras, and

drm/amdgpu: Modify sdma block to fit for the unified ras block data and ops

1.Modify sdma block to fit for the unified ras block data and ops.
2.Change amdgpu_sdma_ras_funcs to amdgpu_sdma_ras, and the corresponding variable name remove _funcs suffix.
3.Remove the const flag of sdma ras variable so that sdma ras block can be able to be inserted into amdgpu device ras block link list.
4.Invoke amdgpu_ras_register_ras_block function to register sdma ras block into amdgpu device ras block link list.
5.Remove the redundant code about sdma in amdgpu_ras.c after using the unified ras block.
6.Fill unified ras block .name .block .ras_late_init and .ras_fini for all of sdma versions. If .ras_late_init and .ras_fini had been defined by the selected sdma version, the defined functions will take effect; if not defined, default fill them with amdgpu_sdma_ras_late_init and amdgpu_sdma_ras_fini.

v2: squash in warning fix (Alex)

Signed-off-by: yipechai <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Reviewed-by: John Clements <[email protected]>
Reviewed-by: Tao Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...