drm/amdgpu: Enforce isolation as part of the job

This patch adds a new parameter 'enforce_isolation' to the amdgpu_job
structure. This parameter is used to determine whether shader isolation
should

drm/amdgpu: Enforce isolation as part of the job

This patch adds a new parameter 'enforce_isolation' to the amdgpu_job
structure. This parameter is used to determine whether shader isolation
should be enforced for a job. The enforce_isolation parameter is then
stored in the amdgpu_job structure and used when flushing the VM.

The enforce_isolation field of the amdgpu_job structure is set directly
after the job is allocated

This change allows more fine-grained control over shader isolation,
making it possible to enforce isolation on a per-job basis rather than
globally. This can be useful in scenarios where only certain jobs
require isolation.

Cc: Christian König <[email protected]>
Cc: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>
Signed-off-by: Srinivasan Shanmugam <[email protected]>
Suggested-by: Christian König <[email protected]>

show more ...

drm/amdgpu: add VM generation token

Instead of using the VRAM lost counter add a 64bit token which indicates
if a context or job is still valid to use.

Should the VRAM be lost or the page tables ne

drm/amdgpu: add VM generation token

Instead of using the VRAM lost counter add a 64bit token which indicates
if a context or job is still valid to use.

Should the VRAM be lost or the page tables need re-creation the token will
change indicating that userspace needs to act and re-create the contexts
and re-submit the work.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add gfx shadow CS IOCTL support

Add support for submitting the shadow update packet
when submitting an IB. Needed for MCBP on GFX11.

v2: update API for CSA (Alex)
v3: fix ordering; SET

drm/amdgpu: add gfx shadow CS IOCTL support

Add support for submitting the shadow update packet
when submitting an IB. Needed for MCBP on GFX11.

v2: update API for CSA (Alex)
v3: fix ordering; SET_Q_PREEMPTION_MODE most come before COND_EXEC
Add missing check for AMDGPU_CHUNK_ID_CP_GFX_SHADOW in
amdgpu_cs_pass1()
Only initialize shadow on first use
(Alex)
v4: Pass parameters rather than job to new ring callback (Alex)
v5: squash in change to call SET_Q_PREEMPTION_MODE/COND_EXEC
before RELEASE_MEM to complete the UMDs use of the shadow (Alex)

Reviewed-by: Christian König <[email protected]>
Signed-off-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: cleanup SPM support a bit

This should probably not access job->vm and also emit the SPM switch
under the conditional execute.

Signed-off-by: Christian König <[email protected]>
R

drm/amdgpu: cleanup SPM support a bit

This should probably not access job->vm and also emit the SPM switch
under the conditional execute.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: fix GDS/GWS/OA switch handling

Bas pointed out that this isn't working as expected and could cause
crashes. Fix the handling by storing the marker that a switch is needed
inside the job

drm/amdgpu: fix GDS/GWS/OA switch handling

Bas pointed out that this isn't working as expected and could cause
crashes. Fix the handling by storing the marker that a switch is needed
inside the job instead.

Reported-by: Bas Nieuwenhuizen <[email protected]>
Signed-off-by: Christian König <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: use scheduler dependencies for CS

Entirely remove the sync obj in the job.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Lin

drm/amdgpu: use scheduler dependencies for CS

Entirely remove the sync obj in the job.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: move explicit sync check into the CS

This moves the memory allocation out of the critical code path.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov

drm/amdgpu: move explicit sync check into the CS

This moves the memory allocation out of the critical code path.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: cleanup scheduler job initialization v2

Init the DRM scheduler base class while allocating the job.

This makes the whole handling much more cleaner.

v2: fix coding style

Signed-off-by

drm/amdgpu: cleanup scheduler job initialization v2

Init the DRM scheduler base class while allocating the job.

This makes the whole handling much more cleaner.

v2: fix coding style

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Link: https://patchwork.freedesktop.org/patch/msgid/[email protected]

show more ...

drm/amdgpu: add gang submit backend v2

Allows submitting jobs as gang which needs to run on multiple
engines at the same time.

Basic idea is that we have a global gang submit fence representing whe

drm/amdgpu: add gang submit backend v2

Allows submitting jobs as gang which needs to run on multiple
engines at the same time.

Basic idea is that we have a global gang submit fence representing when the
gang leader is finally pushed to run on the hardware last.

Jobs submitted as gang are never re-submitted in case of a GPU reset since this
won't work and will just deadlock the hardware immediately again.

v2: fix logic inversion, improve documentation, fix rcu

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: move entity selection and job init earlier during CS

Initialize the entity for the CS and scheduler job much earlier.

v2: fix job initialisation order and use correct scheduler instance

drm/amdgpu: move entity selection and job init earlier during CS

Initialize the entity for the CS and scheduler job much earlier.

v2: fix job initialisation order and use correct scheduler instance

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: move setting the job resources

Move setting the job resources into amdgpu_job.c

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Andrey Grodzovsky <andrey.grodzovs

drm/amdgpu: move setting the job resources

Move setting the job resources into amdgpu_job.c

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Andrey Grodzovsky <[email protected]>
Reviewed-by: Luben Tuikov <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Get rid of amdgpu_job->external_hw_fence

This is a follow-up cleanup to [1]. See bellow refcount balancing
for calling amdgpu_job_submit_direct after this cleanup as far
as I calculated.

drm/amdgpu: Get rid of amdgpu_job->external_hw_fence

This is a follow-up cleanup to [1]. See bellow refcount balancing
for calling amdgpu_job_submit_direct after this cleanup as far
as I calculated.

amdgpu_fence_emit
dma_fence_init 1
dma_fence_get(fence) 2
rcu_assign_pointer(*ptr, dma_fence_get(fence) 3

---> amdgpu_job_submit_direct completes before fence signaled
amdgpu_sa_bo_free
(*sa_bo)->fence = dma_fence_get(fence) 4

amdgpu_job_free
dma_fence_put 3

amdgpu_vcn_enc_get_destroy_msg
*fence = dma_fence_get(f) 4
dma_fence_put(f); 3

amdgpu_vcn_enc_ring_test_ib
dma_fence_put(fence) 2

amdgpu_fence_process
dma_fence_put 1

amdgpu_sa_bo_remove_locked
dma_fence_put 0

---> amdgpu_job_submit_direct completes after fence signaled
amdgpu_fence_process
dma_fence_put 2

amdgpu_job_free
dma_fence_put 1

amdgpu_vcn_enc_get_destroy_msg
*fence = dma_fence_get(f) 2
dma_fence_put(f); 1

amdgpu_vcn_enc_ring_test_ib
dma_fence_put(fence) 0

[1] - https://patchwork.kernel.org/project/dri-devel/cover/[email protected]/

Signed-off-by: Andrey Grodzovsky <[email protected]>
Suggested-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: properly embed the IBs into the job

We now have standard macros for that.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Andrey Grodzovsky <andrey.grodzovsky@amd

drm/amdgpu: properly embed the IBs into the job

We now have standard macros for that.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Andrey Grodzovsky <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: header cleanup

No function change, just move a bunch of definitions from amdgpu.h into
separate header files.

Signed-off-by: Christian König <[email protected]>
Acked-by: Andrey

drm/amdgpu: header cleanup

No function change, just move a bunch of definitions from amdgpu.h into
separate header files.

Signed-off-by: Christian König <[email protected]>
Acked-by: Andrey Grodzovsky <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amd/amdgpu embed hw_fence into amdgpu_job

Why: Previously hw fence is alloced separately with job.
It caused historical lifetime issues and corner cases.
The ideal situation is to take fence to

drm/amd/amdgpu embed hw_fence into amdgpu_job

Why: Previously hw fence is alloced separately with job.
It caused historical lifetime issues and corner cases.
The ideal situation is to take fence to manage both job
and fence's lifetime, and simplify the design of gpu-scheduler.

How:
We propose to embed hw_fence into amdgpu_job.
1. We cover the normal job submission by this method.
2. For ib_test, and submit without a parent job keep the
legacy way to create a hw fence separately.
v2:
use AMDGPU_FENCE_FLAG_EMBED_IN_JOB_BIT to show that the fence is
embedded in a job.
v3:
remove redundant variable ring in amdgpu_job
v4:
add tdr sequence support for this feature. Add a job_run_counter to
indicate whether this job is a resubmit job.
v5
add missing handling in amdgpu_fence_enable_signaling

Signed-off-by: Jingwen Chen <[email protected]>
Signed-off-by: Jack Zhang <[email protected]>
Reviewed-by: Andrey Grodzovsky <[email protected]>
Reviewed by: Monk Liu <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Move to a per-IB secure flag (TMZ)

Move from a per-CS secure flag (TMZ) to a per-IB
secure flag.

Signed-off-by: Luben Tuikov <[email protected]>
Reviewed-by: Huang Rui <ray.huang@amd

drm/amdgpu: Move to a per-IB secure flag (TMZ)

Move from a per-CS secure flag (TMZ) to a per-IB
secure flag.

Signed-off-by: Luben Tuikov <[email protected]>
Reviewed-by: Huang Rui <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: job is secure iff CS is secure (v5)

Mark a job as secure, if and only if the command
submission flag has the secure flag set.

v2: fix the null job pointer while in vmid 0
submission.
v3

drm/amdgpu: job is secure iff CS is secure (v5)

Mark a job as secure, if and only if the command
submission flag has the secure flag set.

v2: fix the null job pointer while in vmid 0
submission.
v3: Context --> Command submission.
v4: filling cs parser with cs->in.flags
v5: move the job secure flag setting out of amdgpu_cs_submit()

Signed-off-by: Huang Rui <[email protected]>
Signed-off-by: Luben Tuikov <[email protected]>
Reviewed-by: Alex Deucher <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: implement more ib pools (v2)

We have three ib pools, they are normal, VM, direct pools.

Any jobs which schedule IBs without dependence on gpu scheduler should
use DIRECT pool.

Any jobs

drm/amdgpu: implement more ib pools (v2)

We have three ib pools, they are normal, VM, direct pools.

Any jobs which schedule IBs without dependence on gpu scheduler should
use DIRECT pool.

Any jobs schedule direct VM update IBs should use VM pool.

Any other jobs use NORMAL pool.

v2: squash in coding style fix

Signed-off-by: xinhui pan <[email protected]>
Reviewed-by: Christian König <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: drop amdgpu_job.owner

Entirely unused.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <alexan

drm/amdgpu: drop amdgpu_job.owner

Entirely unused.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Felix Kuehling <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Avoid HW GPU reset for RAS.

Problem:
Under certain conditions, when some IP bocks take a RAS error,
we can get into a situation where a GPU reset is not possible
due to issues in RAS in

drm/amdgpu: Avoid HW GPU reset for RAS.

Problem:
Under certain conditions, when some IP bocks take a RAS error,
we can get into a situation where a GPU reset is not possible
due to issues in RAS in SMU/PSP.

Temporary fix until proper solution in PSP/SMU is ready:
When uncorrectable error happens the DF will unconditionally
broadcast error event packets to all its clients/slave upon
receiving fatal error event and freeze all its outbound queues,
err_event_athub interrupt will be triggered.
In such case and we use this interrupt
to issue GPU reset. THe GPU reset code is modified for such case to avoid HW
reset, only stops schedulers, deatches all in progress and not yet scheduled
job's fences, set error code on them and signals.
Also reject any new incoming job submissions from user space.
All this is done to notify the applications of the problem.

v2:
Extract amdgpu_amdkfd_pre/post_reset from amdgpu_device_lock/unlock_adev
Move amdgpu_job_stop_all_jobs_on_sched to amdgpu_job.c
Remove print param from amdgpu_ras_query_error_count

v3:
Update based on prevoius bug fixing patch to properly call amdgpu_amdkfd_pre_reset
for other XGMI hive memebers.

Signed-off-by: Andrey Grodzovsky <[email protected]>
Acked-by: Felix Kuehling <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add ib preemption status in amdgpu_job (v2)

Add ib preemption status in amdgpu_job, so that ring level function
can detect preemption and program for resuming it.

v2: squash in fix to r

drm/amdgpu: add ib preemption status in amdgpu_job (v2)

Add ib preemption status in amdgpu_job, so that ring level function
can detect preemption and program for resuming it.

v2: squash in fix to restore job->preamble_status back to status value (Jack)

Acked-by: Alex Deucher <[email protected]>
Reviewed-by: Hawking Zhang <[email protected]>
Signed-off-by: Jack Xiao <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: Modify the argument of emit_ib interface

use the point of struct amdgpu_job as the function
argument instand of vmid, so the other members of
struct amdgpu_job can be visit in emit_ib fu

drm/amdgpu: Modify the argument of emit_ib interface

use the point of struct amdgpu_job as the function
argument instand of vmid, so the other members of
struct amdgpu_job can be visit in emit_ib function.

v2: add a wrapper for getting the VMID
add the job before the ib on the parameter list.
v3: refine the wrapper name

Reviewed-by: Alex Deucher <[email protected]>
Signed-off-by: Rex Zhu <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: remove job->adev (v2)

We can get that from the ring.

v2: squash in "drm/amdgpu: always initialize job->base.sched" (Alex)

Signed-off-by: Christian König <[email protected]>
Revi

drm/amdgpu: remove job->adev (v2)

We can get that from the ring.

v2: squash in "drm/amdgpu: always initialize job->base.sched" (Alex)

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Junwei Zhang <[email protected]>
Acked-by: Chunming Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: add amdgpu_job_submit_direct helper

Make sure that we properly initialize at least the sched member.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Junwei Zhang

drm/amdgpu: add amdgpu_job_submit_direct helper

Make sure that we properly initialize at least the sched member.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Junwei Zhang <[email protected]>
Acked-by: Chunming Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...

drm/amdgpu: remove job->ring

We can easily get that from the scheduler.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Junwei Zhang <[email protected]>
Acked-by: Chunming

drm/amdgpu: remove job->ring

We can easily get that from the scheduler.

Signed-off-by: Christian König <[email protected]>
Reviewed-by: Junwei Zhang <[email protected]>
Acked-by: Chunming Zhou <[email protected]>
Signed-off-by: Alex Deucher <[email protected]>

show more ...