riscv: vector: Support xtheadvector save/restore

Use alternatives to add support for xtheadvector vector save/restore
routines.

Signed-off-by: Charlie Jenkins <[email protected]>
Reviewed-by: Co

riscv: vector: Support xtheadvector save/restore

Use alternatives to add support for xtheadvector vector save/restore
routines.

Signed-off-by: Charlie Jenkins <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Tested-by: Yangyu Chen <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

RISC-V: Detect unaligned vector accesses supported

Run an unaligned vector access to test if the system supports
vector unaligned access. Add the result to a new key in hwprobe.
This is useful for u

RISC-V: Detect unaligned vector accesses supported

Run an unaligned vector access to test if the system supports
vector unaligned access. Add the result to a new key in hwprobe.
This is useful for usermode to know if vector misaligned accesses are
supported and if they are faster or slower than equivalent byte accesses.

Signed-off-by: Jesse Taube <[email protected]>
Signed-off-by: Charlie Jenkins <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: vector: adjust minimum Vector requirement to ZVE32X

Make has_vector() to check for ZVE32X. Every in-kernel usage of V that
requires a more complicate version of V must then call out explicitl

riscv: vector: adjust minimum Vector requirement to ZVE32X

Make has_vector() to check for ZVE32X. Every in-kernel usage of V that
requires a more complicate version of V must then call out explicitly.

Also, change riscv_v_first_use_handler(), and boot code that calls
riscv_v_setup_vsize() to accept ZVE32X.

Most kernel/user interfaces requires minimum of ZVE32X. Thus, programs
compiled and run with ZVE32X should be supported by the kernel on most
aspects. This includes context-switch, signal, ptrace, prctl, and
hwprobe.

One exception is that ELF_HWCAP returns 'V' only if full V is supported
on the platform. This means that the system without a full V must not
rely on ELF_HWCAP to tell whether it is allowable to execute Vector
without first invoking a prctl() check.

Signed-off-by: Andy Chiu <[email protected]>
Acked-by: Joel Granados <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

RISC-V: add helper function to read the vector VLEN

VLEN describes the length of each vector register and some instructions
need specific minimal VLENs to work correctly.

The vector code already in

RISC-V: add helper function to read the vector VLEN

VLEN describes the length of each vector register and some instructions
need specific minimal VLENs to work correctly.

The vector code already includes a variable riscv_v_vsize that contains
the value of "32 vector registers with vlenb length" that gets filled
during boot. vlenb is the value contained in the CSR_VLENB register and
the value represents "VLEN / 8".

So add riscv_vector_vlen() to return the actual VLEN value for in-kernel
users when they need to check the available VLEN.

Signed-off-by: Heiko Stuebner <[email protected]>
Reviewed-by: Eric Biggers <[email protected]>
Signed-off-by: Jerry Shih <[email protected]>
Signed-off-by: Eric Biggers <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: vector: allow kernel-mode Vector with preemption

Add kernel_vstate to keep track of kernel-mode Vector registers when
trap introduced context switch happens. Also, provide riscv_v_flags to
le

riscv: vector: allow kernel-mode Vector with preemption

Add kernel_vstate to keep track of kernel-mode Vector registers when
trap introduced context switch happens. Also, provide riscv_v_flags to
let context save/restore routine track context status. Context tracking
happens whenever the core starts its in-kernel Vector executions. An
active (dirty) kernel task's V contexts will be saved to memory whenever
a trap-introduced context switch happens. Or, when a softirq, which
happens to nest on top of it, uses Vector. Context retoring happens when
the execution transfer back to the original Kernel context where it
first enable preempt_v.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an
option to disable preemptible kernel-mode Vector at build time. Users
with constraint memory may want to disable this config as preemptible
kernel-mode Vector needs extra space for tracking of per thread's
kernel-mode V context. Or, users might as well want to disable it if all
kernel-mode Vector code is time sensitive and cannot tolerate context
switch overhead.

Signed-off-by: Andy Chiu <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Tested-by: Lad Prabhakar <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: vector: use kmem_cache to manage vector context

The allocation size of thread.vstate.datap is always riscv_v_vsize. So
it is possbile to use kmem_cache_* to manage the allocation. This gives

riscv: vector: use kmem_cache to manage vector context

The allocation size of thread.vstate.datap is always riscv_v_vsize. So
it is possbile to use kmem_cache_* to manage the allocation. This gives
users more information regarding allocation of vector context via
/proc/slabinfo. And it potentially reduces the latency of the first-use
trap because of the allocation caches.

Signed-off-by: Andy Chiu <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Tested-by: Lad Prabhakar <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()

riscv_v_vstate_{save,restore}() can operate only on the knowlege of
struct __riscv_v_ext_state, and struct pt_regs. Let th

riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()

riscv_v_vstate_{save,restore}() can operate only on the knowlege of
struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides
which should be passed into the function. Meanwhile, the kernel-mode
Vector is going to introduce another vstate, so this also makes functions
potentially able to be reused.

Signed-off-by: Andy Chiu <[email protected]>
Acked-by: Conor Dooley <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Tested-by: Lad Prabhakar <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: sched: defer restoring Vector context for user

User will use its Vector registers only after the kernel really returns
to the userspace. So we can delay restoring Vector registers as long as

riscv: sched: defer restoring Vector context for user

User will use its Vector registers only after the kernel really returns
to the userspace. So we can delay restoring Vector registers as long as
we are still running in kernel mode. So, add a thread flag to indicates
the need of restoring Vector and do the restore at the last
arch-specific exit-to-user hook. This save the context restoring cost
when we switch over multiple processes that run V in kernel mode. For
example, if the kernel performs a context swicth from A->B->C, and
returns to C's userspace, then there is no need to restore B's
V-register.

Besides, this also prevents us from repeatedly restoring V context when
executing kernel-mode Vector multiple times.

The cost of this is that we must disable preemption and mark vector as
busy during vstate_{save,restore}. Because then the V context will not
get restored back immediately when a trap-causing context switch happens
in the middle of vstate_{save,restore}.

Signed-off-by: Andy Chiu <[email protected]>
Acked-by: Conor Dooley <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Tested-by: Lad Prabhakar <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Add support for kernel mode vector

Add kernel_vector_begin() and kernel_vector_end() function declarations
and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses

riscv: Add support for kernel mode vector

Add kernel_vector_begin() and kernel_vector_end() function declarations
and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

Co-developed-by: Vincent Chen <[email protected]>
Signed-off-by: Vincent Chen <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Eric Biggers <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Tested-by: Lad Prabhakar <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Rearrange hwcap.h and cpufeature.h

Now hwcap.h and cpufeature.h are mutually including each other, and most of
the variable/API declarations in hwcap.h are implemented in cpufeature.c,
so, it

riscv: Rearrange hwcap.h and cpufeature.h

Now hwcap.h and cpufeature.h are mutually including each other, and most of
the variable/API declarations in hwcap.h are implemented in cpufeature.c,
so, it's better to move them into cpufeature.h and leave only macros for
ISA extension logical IDs in hwcap.h.

BTW, the riscv_isa_extension_mask macro is not used now, so this patch
removes it.

Suggested-by: Andrew Jones <[email protected]>
Signed-off-by: Xiao Wang <[email protected]>
Reviewed-by: Andrew Jones <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

RISC-V: vector: export VLENB csr in __sc_riscv_v_state

VLENB is critical for callers of ptrace to reconstruct Vector register
files from the register dump of NT_RISCV_VECTOR. Also, future systems
ma

RISC-V: vector: export VLENB csr in __sc_riscv_v_state

VLENB is critical for callers of ptrace to reconstruct Vector register
files from the register dump of NT_RISCV_VECTOR. Also, future systems
may will have a writable VLENB, so add it now to potentially save future
compatibility issue.

Fixes: 0c59922c769a ("riscv: Add ptrace vector support")
Signed-off-by: Andy Chiu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Discard vector state on syscalls

The RISC-V vector specification states:
Executing a system call causes all caller-saved vector registers
(v0-v31, vl, vtype) and vstart to become unspecif

riscv: Discard vector state on syscalls

The RISC-V vector specification states:
Executing a system call causes all caller-saved vector registers
(v0-v31, vl, vtype) and vstart to become unspecified.

The vector registers are set to all 1s, vill is set (invalid), and the
vector status is set to Dirty.

That way we can prevent userspace from accidentally relying on the
stated save.

Rémi pointed out [1] that writing to the registers might be
superfluous, and setting vill is sufficient.

Link: https://lore.kernel.org/linux-riscv/[email protected]/ # [1]
Suggested-by: Darius Rad <[email protected]>
Suggested-by: Palmer Dabbelt <[email protected]>
Suggested-by: Rémi Denis-Courmont <[email protected]>
Signed-off-by: Björn Töpel <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Add prctl controls for userspace vector management

This patch add two riscv-specific prctls, to allow usespace control the
use of vector unit:

* PR_RISCV_V_SET_CONTROL: control the permissi

riscv: Add prctl controls for userspace vector management

This patch add two riscv-specific prctls, to allow usespace control the
use of vector unit:

* PR_RISCV_V_SET_CONTROL: control the permission to use Vector at next,
or all following execve for a thread. Turning off a thread's Vector
live is not possible since libraries may have registered ifunc that
may execute Vector instructions.
* PR_RISCV_V_GET_CONTROL: get the same permission setting for the
current thread, and the setting for following execve(s).

Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Greentime Hu <[email protected]>
Reviewed-by: Vincent Chen <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Allocate user's vector context in the first-use trap

Vector unit is disabled by default for all user processes. Thus, a
process will take a trap (illegal instruction) into kernel at the first

riscv: Allocate user's vector context in the first-use trap

Vector unit is disabled by default for all user processes. Thus, a
process will take a trap (illegal instruction) into kernel at the first
time when it uses Vector. Only after then, the kernel allocates V
context and starts take care of the context for that user process.

Suggested-by: Richard Henderson <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Add task switch support for vector

This patch adds task switch support for vector. It also supports all
lengths of vlen.

Suggested-by: Andrew Waterman <[email protected]>
Co-developed-by: Ni

riscv: Add task switch support for vector

This patch adds task switch support for vector. It also supports all
lengths of vlen.

Suggested-by: Andrew Waterman <[email protected]>
Co-developed-by: Nick Knight <[email protected]>
Signed-off-by: Nick Knight <[email protected]>
Co-developed-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Co-developed-by: Vincent Chen <[email protected]>
Signed-off-by: Vincent Chen <[email protected]>
Co-developed-by: Ruinland Tsai <[email protected]>
Signed-off-by: Ruinland Tsai <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Signed-off-by: Vineet Gupta <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Tested-by: Heiko Stuebner <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Introduce struct/helpers to save/restore per-task Vector state

Add vector state context struct to be added later in thread_struct. And
prepare low-level helper functions to save/restore vecto

riscv: Introduce struct/helpers to save/restore per-task Vector state

Add vector state context struct to be added later in thread_struct. And
prepare low-level helper functions to save/restore vector contexts.

This include Vector Regfile and CSRs holding dynamic configuration state
(vstart, vl, vtype, vcsr). The Vec Register width could be implementation
defined, but same for all processes, so that is saved separately.

This is not yet wired into final thread_struct - will be done when
__switch_to actually starts doing this in later patches.

Given the variable (and potentially large) size of regfile, they are
saved in dynamically allocated memory, pointed to by datap pointer in
__riscv_v_ext_state.

Co-developed-by: Vincent Chen <[email protected]>
Signed-off-by: Vincent Chen <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Signed-off-by: Vineet Gupta <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Acked-by: Conor Dooley <[email protected]>
Reviewed-by: Guo Ren <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Tested-by: Heiko Stuebner <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Introduce riscv_v_vsize to record size of Vector context

This patch is used to detect the size of CPU vector registers and use
riscv_v_vsize to save the size of all the vector registers. It a

riscv: Introduce riscv_v_vsize to record size of Vector context

This patch is used to detect the size of CPU vector registers and use
riscv_v_vsize to save the size of all the vector registers. It assumes all
harts has the same capabilities in a SMP system. If a core detects VLENB
that is different from the boot core, then it warns and turns off V
support for user space.

Co-developed-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Co-developed-by: Vincent Chen <[email protected]>
Signed-off-by: Vincent Chen <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Tested-by: Heiko Stuebner <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Introduce Vector enable/disable helpers

These are small and likely to be frequently called so implement as
inline routines (vs. function call).

Co-developed-by: Guo Ren <[email protected]

riscv: Introduce Vector enable/disable helpers

These are small and likely to be frequently called so implement as
inline routines (vs. function call).

Co-developed-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Co-developed-by: Vincent Chen <[email protected]>
Signed-off-by: Vincent Chen <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Signed-off-by: Vineet Gupta <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Tested-by: Heiko Stuebner <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Extending cpufeature.c to detect V-extension

Add V-extension into riscv_isa_ext_keys array and detect it with isa
string parsing.

Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Gu

riscv: Extending cpufeature.c to detect V-extension

Add V-extension into riscv_isa_ext_keys array and detect it with isa
string parsing.

Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Greentime Hu <[email protected]>
Suggested-by: Vineet Gupta <[email protected]>
Co-developed-by: Andy Chiu <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Tested-by: Heiko Stuebner <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...