s390/processor: Use bitop functions for cpu flag helper functions

Use bitop functions to implement cpu flag helper functions. This way
it is guaranteed that bits cannot get lost if modified in diffe

s390/processor: Use bitop functions for cpu flag helper functions

Use bitop functions to implement cpu flag helper functions. This way
it is guaranteed that bits cannot get lost if modified in different
contexts on a cpu.

E.g. if process context is interrupted in the middle of a
read-modify-write sequence while modifying cpu flags, and within
interrupt context cpu flags are also modified, bits can get lost.

There is currently no code which is doing this, however upcoming code
could potentially run into this problem.

Acked-by: Vasily Gorbik <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390: Use inline qualifier for all EX_TABLE and ALTERNATIVE inline assemblies

Use asm_inline for all inline assemblies which make use of the EX_TABLE or
ALTERNATIVE macros.

These macros expand to m

s390: Use inline qualifier for all EX_TABLE and ALTERNATIVE inline assemblies

Use asm_inline for all inline assemblies which make use of the EX_TABLE or
ALTERNATIVE macros.

These macros expand to many lines and the compiler assumes the number of
lines within an inline assembly is the same as the number of instructions
within an inline assembly. This has an effect on inlining and loop
unrolling decisions.

In order to avoid incorrect assumptions use asm_inline, which tells the
compiler that an inline assembly has the smallest possible size.

In order to avoid confusion when asm_inline should be used or not, since a
couple of inline assemblies are quite large: the rule is to always use
asm_inline whenever the EX_TABLE or ALTERNATIVE macro is used. In specific
cases there may be reasons to not follow this guideline, but that should
be documented with the corresponding code.

Using the inline qualifier everywhere has only a small effect on the kernel
image size:

add/remove: 0/10 grow/shrink: 19/8 up/down: 1492/-1858 (-366)

The only location where this seems to matter is load_unaligned_zeropad()
from word-at-a-time.h where the compiler inlines more functions within the
dcache code, which is indeed code where performance matters.

Suggested-by: Juergen Christ <[email protected]>
Reviewed-by: Juergen Christ <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390/stackleak: Use exrl instead of ex in __stackleak_poison()

exrl is present in all machines currently supported, therefore prefer
it over ex. This saves one instruction and doesn't need an additi

s390/stackleak: Use exrl instead of ex in __stackleak_poison()

exrl is present in all machines currently supported, therefore prefer
it over ex. This saves one instruction and doesn't need an additional
register to hold the address of the target instruction.

Signed-off-by: Sven Schnelle <[email protected]>
Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/mm/fault: Handle guest-related program interrupts in KVM

Any program interrupt that happens in the host during the execution of
a KVM guest will now short circuit the fault handler and return t

s390/mm/fault: Handle guest-related program interrupts in KVM

Any program interrupt that happens in the host during the execution of
a KVM guest will now short circuit the fault handler and return to KVM
immediately. Guest fault handling (including pfault) will happen
entirely inside KVM.

When sie64a() returns zero, current->thread.gmap_int_code will contain
the program interrupt number that caused the exit, or zero if the exit
was not caused by a host program interrupt.

KVM will now take care of handling all guest faults in vcpu_post_run().

Since gmap faults will not be visible by the rest of the kernel, remove
GMAP_FAULT, the linux fault handlers for secure execution faults, the
exception table entries for the sie instruction, the nop padding after
the sie instruction, and all other references to guest faults from the
s390 code.

Signed-off-by: Claudio Imbrenda <[email protected]>
Co-developed-by: Heiko Carstens <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/smp: Add cpu capacities

Linux scheduler allows architectures to assign capacity values to
individual CPUs. This hints scheduler the performance difference between
CPUs and allows more efficient

s390/smp: Add cpu capacities

Linux scheduler allows architectures to assign capacity values to
individual CPUs. This hints scheduler the performance difference between
CPUs and allows more efficient task distribution them. Implement
helper methods to set and get CPU capacities for s390. This is
particularly helpful in vertical polarization configurations of LPARs.
On vertical polarization an LPARs CPUs can get different polarization
values depending on the CEC configuration. CPUs with different
polarization values can perform different from each other, using CPU
capacities this can be reflected to linux scheduler.

Acked-by: Vasily Gorbik <[email protected]>
Signed-off-by: Mete Durlu <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390/entry: Move SIE indicator flag to thread info

CIF_SIE indicates if a thread is running in SIE context. This is the
state of a thread and not the CPU. Therefore move this indicator to
thread inf

s390/entry: Move SIE indicator flag to thread info

CIF_SIE indicates if a thread is running in SIE context. This is the
state of a thread and not the CPU. Therefore move this indicator to
thread info.

Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390/nospec: Push down alternative handling

The nospec implementation is deeply integrated into the alternatives
code: only for nospec an alternative facility list is implemented and
used by the alt

s390/nospec: Push down alternative handling

The nospec implementation is deeply integrated into the alternatives
code: only for nospec an alternative facility list is implemented and
used by the alternative code, while it is modified by nospec specific
needs.

Push down the nospec alternative handling into the nospec by
introducing a new alternative type and a specific nospec callback to
decide if alternatives should be applied.

Also introduce a new global nobp variable which together with facility
82 can be used to decide if nobp is enabled or not.

Acked-by: Alexander Gordeev <[email protected]>
Tested-by: Sven Schnelle <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390/alternatives: Rework to allow for callbacks

Rework alternatives to allow for callbacks. With this every
alternative entry has additional data encoded:

- When (aka context) an alternative is su

s390/alternatives: Rework to allow for callbacks

Rework alternatives to allow for callbacks. With this every
alternative entry has additional data encoded:

- When (aka context) an alternative is supposed to be applied

- The type of an alternative, which allows for type specific handling
and callbacks

- Extra type specific payload (patch information), which can be passed
to callbacks in order to decide if an alternative should be applied
or not

With this only the "late" context is implemented, which means there is
no change to the previous behaviour. All code is just converted to the
more generic new infrastructure.

Reviewed-by: Alexander Gordeev <[email protected]>
Tested-by: Sven Schnelle <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390: Move CIF flags to struct pcpu

To allow testing flags for offline CPUs, move the CIF flags
to struct pcpu. To avoid having to calculate the array index
for each access, add a pointer to the pcp

s390: Move CIF flags to struct pcpu

To allow testing flags for offline CPUs, move the CIF flags
to struct pcpu. To avoid having to calculate the array index
for each access, add a pointer to the pcpu member for the current
cpu to lowcore.

Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390: Replace S390_lowcore by get_lowcore()

Replace all S390_lowcore usages in arch/s390/ by get_lowcore().

Acked-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]

s390: Replace S390_lowcore by get_lowcore()

Replace all S390_lowcore usages in arch/s390/ by get_lowcore().

Acked-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390: Mark psw in __load_psw_mask() as __unitialized

Without __unitialized, the following code is generated when
INIT_STACK_ALL_ZERO is enabled:

86: d7 0f f0 a0 f0 a0 xc 160(16,%r15), 160(

s390: Mark psw in __load_psw_mask() as __unitialized

Without __unitialized, the following code is generated when
INIT_STACK_ALL_ZERO is enabled:

86: d7 0f f0 a0 f0 a0 xc 160(16,%r15), 160(%r15)
8c: e3 40 f0 a0 00 24 stg %r4, 160(%r15)
92: c0 10 00 00 00 08 larl %r1, 0xa2
98: e3 10 f0 a8 00 24 stg %r1, 168(%r15)
9e: b2 b2 f0 a0 lpswe 160(%r15)

The xc is not adding any security because psw is fully initialized
with the following instructions. Add __unitialized to the psw
definitiation to avoid the superfluous clearing of psw.

Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/idle: Rewrite psw_idle() in C

To ease maintenance and further enhancements, convert
the psw_idle() function to C.

Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <

s390/idle: Rewrite psw_idle() in C

To ease maintenance and further enhancements, convert
the psw_idle() function to C.

Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Sven Schnelle <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/stackstrace: Detect vdso stack frames

Clear the backchain of the extra stack frame added by the vdso user wrapper
code. This allows the user stack walker to detect and skip the non-standard
sta

s390/stackstrace: Detect vdso stack frames

Clear the backchain of the extra stack frame added by the vdso user wrapper
code. This allows the user stack walker to detect and skip the non-standard
stack frame. Without this an incorrect instruction pointer would be added
to stack traces, and stack frame walking would be continued with a more or
less random back chain.

Fixes: aa44433ac4ee ("s390: add USER_STACKTRACE support")
Reviewed-by: Jens Remus <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/entry: add CIF_SIE flag and remove sie64a() address check

When a program check, interrupt or machine check is triggered, the
PSW address is compared to a certain range of the sie64a() function

s390/entry: add CIF_SIE flag and remove sie64a() address check

When a program check, interrupt or machine check is triggered, the
PSW address is compared to a certain range of the sie64a() function
to figure out whether SIE was interrupted and a cleanup of SIE is
needed.

This doesn't work with kprobes: If kprobes probes an instruction, it
copies the instruction to the kprobes instruction page and overwrites the
original instruction with an undefind instruction (Opcode 00). When this
instruction is hit later, kprobes single-steps the instruction on the
kprobes_instruction page.

However, if this instruction is a relative branch instruction it will now
point to a different location in memory due to being moved to the kprobes
instruction page. If the new branch target points into sie64a() the kernel
assumes it interrupted SIE when processing the breakpoint and will crash
trying to access the SIE control block.

Instead of comparing the address, introduce a new CIF_SIE flag which
indicates whether SIE was interrupted.

Signed-off-by: Sven Schnelle <[email protected]>
Suggested-by: Heiko Carstens <[email protected]>
Reviewed-by: Heiko Carstens <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: limit save and restore to used registers

The first invocation of kernel_fpu_begin() after switching from user to
kernel context will save all vector registers, even if only parts of the
ve

s390/fpu: limit save and restore to used registers

The first invocation of kernel_fpu_begin() after switching from user to
kernel context will save all vector registers, even if only parts of the
vector registers are used within the kernel fpu context. Given that save
and restore of all vector registers is quite expensive change the current
approach in several ways:

- Instead of saving and restoring all user registers limit this to those
registers which are actually used within an kernel fpu context.

- On context switch save all remaining user fpu registers, so they can be
restored when the task is rescheduled.

- Saving user registers within kernel_fpu_begin() is done without disabling
and enabling interrupts - which also slightly reduces runtime. In worst
case (e.g. interrupt context uses the same registers) this may lead to
the situation that registers are saved several times, however the
assumption is that this will not happen frequently, so that the new
method is faster in nearly all cases.

- save_user_fpu_regs() can still be called from all contexts and saves all
(or all remaining) user registers to a tasks ufpu user fpu save area.

Overall this reduces the time required to save and restore the user fpu
context for nearly all cases.

Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: remove regs member from struct fpu

KVM was the only user which modified the regs pointer in struct fpu. Remove
the pointer and convert the rest of the core fpu code to directly access
the

s390/fpu: remove regs member from struct fpu

KVM was the only user which modified the regs pointer in struct fpu. Remove
the pointer and convert the rest of the core fpu code to directly access
the save area embedded within struct fpu.

Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: make kernel fpu context preemptible

Make the kernel fpu context preemptible. Add another fpu structure to the
thread_struct, and use it to save and restore the kernel fpu context if its
ta

s390/fpu: make kernel fpu context preemptible

Make the kernel fpu context preemptible. Add another fpu structure to the
thread_struct, and use it to save and restore the kernel fpu context if its
task uses fpu registers when it is preempted.

Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: rename save_fpu_regs() to save_user_fpu_regs(), etc

Rename save_fpu_regs(), load_fpu_regs(), and struct thread_struct's fpu
member to save_user_fpu_regs(), load_user_fpu_regs(), and ufpu.

s390/fpu: rename save_fpu_regs() to save_user_fpu_regs(), etc

Rename save_fpu_regs(), load_fpu_regs(), and struct thread_struct's fpu
member to save_user_fpu_regs(), load_user_fpu_regs(), and ufpu. This way
the function and variable names reflect for which context they are supposed
to be used.

This large and trivial conversion is a prerequisite for making the kernel
fpu usage preemptible.

Reviewed-by: Claudio Imbrenda <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: convert FPU CIF flag to regular TIF flag

The FPU state, as represented by the CIF_FPU flag reflects the FPU state of
a task, not the CPU it is running on. Therefore convert the flag to a
r

s390/fpu: convert FPU CIF flag to regular TIF flag

The FPU state, as represented by the CIF_FPU flag reflects the FPU state of
a task, not the CPU it is running on. Therefore convert the flag to a
regular TIF flag.

This removes the magic in switch_to() where a save_fpu_regs() call for the
currently (previous) running task sets the per-cpu CIF_FPU flag, which is
required to restore FPU register contents of the next task, when it returns
to user space.

Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: move, rename, and merge header files

Move, rename, and merge the fpu and vx header files. This way fpu header
files have a consistent naming scheme (fpu*.h).

Also get rid of the fpu subdi

s390/fpu: move, rename, and merge header files

Move, rename, and merge the fpu and vx header files. This way fpu header
files have a consistent naming scheme (fpu*.h).

Also get rid of the fpu subdirectory and move header files to asm
directory, so that all fpu and vx header files can be found at the same
location.

Merge internal.h header file into other header files, since the internal
helpers are used at many locations. so those helper functions are really
not internal.

Signed-off-by: Heiko Carstens <[email protected]>

show more ...

s390/fpu: remove ARCH_WANTS_DYNAMIC_TASK_STRUCT support

s390 selects ARCH_WANTS_DYNAMIC_TASK_STRUCT in order to make the size of
the task structure dependent on the availability of the vector
facili

s390/fpu: remove ARCH_WANTS_DYNAMIC_TASK_STRUCT support

s390 selects ARCH_WANTS_DYNAMIC_TASK_STRUCT in order to make the size of
the task structure dependent on the availability of the vector
facility. This doesn't make sense anymore because since many years all
machines provide the vector facility.

Therefore simplify the code a bit and remove s390 support for
ARCH_WANTS_DYNAMIC_TASK_STRUCT.

Reviewed-by: Alexander Gordeev <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/nmi: implement and use local_mcck_save() / local_mcck_restore()

Instead of using local_mcck_disable() / local_mcck_enable() implement and
use local_mcck_save() / local_mcck_restore() to disable

s390/nmi: implement and use local_mcck_save() / local_mcck_restore()

Instead of using local_mcck_disable() / local_mcck_enable() implement and
use local_mcck_save() / local_mcck_restore() to disable machine checks, and
restoring the previous state.

The problem with using local_mcck_disable() / local_mcck_enable() is that
there is an assumption that machine checks are always enabled. While this
is currently the case the code still looks quite odd, readers need to
double check if the code is correct.

In order to increase readability save and then restore the old machine
check mask bit, instead of assuming that it must have been enabled.

Reviewed-by: Alexander Gordeev <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390: remove odd comment

In the meantime hopefully most people got used to forward
declarations, therefore remove the explanation.

Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: A

s390: remove odd comment

In the meantime hopefully most people got used to forward
declarations, therefore remove the explanation.

Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Alexander Gordeev <[email protected]>

show more ...

s390/stackleak: provide fast __stackleak_poison() implementation

Provide an s390 specific __stackleak_poison() implementation which is
faster than the generic variant.

For the original implementati

s390/stackleak: provide fast __stackleak_poison() implementation

Provide an s390 specific __stackleak_poison() implementation which is
faster than the generic variant.

For the original implementation with an enforced 4kb stackframe for the
getpid() system call the system call overhead increases by a factor of 3 if
the stackleak feature is enabled. Using the s390 mvc based variant this is
reduced to an increase of 25% instead.

This is within the expected area, since the mvc based implementation is
more or less a memset64() variant which comes with similar results. See
commit 0b77d6701cf8 ("s390: implement memset16, memset32 & memset64").

Reviewed-by: Vasily Gorbik <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...

s390: move on_thread_stack() to processor.h

As preparation for the stackleak feature move on_thread_stack() to
processor.h like x86.

Also make it __always_inline, and slightly optimize it by readin

s390: move on_thread_stack() to processor.h

As preparation for the stackleak feature move on_thread_stack() to
processor.h like x86.

Also make it __always_inline, and slightly optimize it by reading
current task's kernel stack pointer from lowcore.

Reviewed-by: Vasily Gorbik <[email protected]>
Signed-off-by: Heiko Carstens <[email protected]>
Signed-off-by: Vasily Gorbik <[email protected]>

show more ...