asm-generic: introduce text-patching.h

Several architectures support text patching, but they name the header
files that declare patching functions differently.

Make all such headers consistently na

asm-generic: introduce text-patching.h

Several architectures support text patching, but they name the header
files that declare patching functions differently.

Make all such headers consistently named text-patching.h and add an empty
header in asm-generic for architectures that do not support text patching.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Mike Rapoport (Microsoft) <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
Acked-by: Geert Uytterhoeven <[email protected]> # m68k
Acked-by: Arnd Bergmann <[email protected]>
Reviewed-by: Luis Chamberlain <[email protected]>
Tested-by: kdevops <[email protected]>
Cc: Andreas Larsson <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Ard Biesheuvel <[email protected]>
Cc: Borislav Petkov (AMD) <[email protected]>
Cc: Brian Cain <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Christophe Leroy <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Dinh Nguyen <[email protected]>
Cc: Guo Ren <[email protected]>
Cc: Helge Deller <[email protected]>
Cc: Huacai Chen <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Johannes Berg <[email protected]>
Cc: John Paul Adrian Glaubitz <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Liam R. Howlett <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Masami Hiramatsu (Google) <[email protected]>
Cc: Matt Turner <[email protected]>
Cc: Max Filippov <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Michal Simek <[email protected]>
Cc: Oleg Nesterov <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Russell King <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stafford Horne <[email protected]>
Cc: Steven Rostedt (Google) <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Thomas Bogendoerfer <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Uladzislau Rezki (Sony) <[email protected]>
Cc: Vineet Gupta <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

riscv: Re-introduce global icache flush in patch_text_XXX()

commit edf2d546bfd6 ("riscv: patch: Flush the icache right after
patching to avoid illegal insns") mistakenly removed the global icache
fl

riscv: Re-introduce global icache flush in patch_text_XXX()

commit edf2d546bfd6 ("riscv: patch: Flush the icache right after
patching to avoid illegal insns") mistakenly removed the global icache
flush in patch_text_nosync() and patch_text_set_nosync() functions, so
reintroduce them.

Fixes: edf2d546bfd6 ("riscv: patch: Flush the icache right after patching to avoid illegal insns")
Reported-by: Samuel Holland <[email protected]>
Closes: https://lore.kernel.org/linux-riscv/[email protected]/
Signed-off-by: Alexandre Ghiti <[email protected]>
Reviewed-by: Samuel Holland <[email protected]>
Reviewed-by: Charlie Jenkins <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: patch: Flush the icache right after patching to avoid illegal insns

We cannot delay the icache flush after patching some functions as we may
have patched a function that will get called befor

riscv: patch: Flush the icache right after patching to avoid illegal insns

We cannot delay the icache flush after patching some functions as we may
have patched a function that will get called before the icache flush.

The only way to completely avoid such scenario is by flushing the icache
as soon as we patch a function. This will probably be costly as we don't
batch the icache maintenance anymore.

Fixes: 6ca445d8af0e ("riscv: Fix early ftrace nop patching")
Reported-by: Conor Dooley <[email protected]>
Closes: https://lore.kernel.org/linux-riscv/20240613-lubricant-breath-061192a9489a@wendy/
Signed-off-by: Alexandre Ghiti <[email protected]>
Reviewed-by: Andy Chiu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Remove extra variable in patch_text_nosync()

This cast is superfluous, and is incorrect anyway if compressed
instructions may be present.

Reviewed-by: Björn Töpel <[email protected]>
Signed

riscv: Remove extra variable in patch_text_nosync()

This cast is superfluous, and is incorrect anyway if compressed
instructions may be present.

Reviewed-by: Björn Töpel <[email protected]>
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Use offset_in_page() in text patching functions

This is a bit easier to parse than the equivalent bit manipulation.

Reviewed-by: Björn Töpel <[email protected]>
Signed-off-by: Samuel Hollan

riscv: Use offset_in_page() in text patching functions

This is a bit easier to parse than the equivalent bit manipulation.

Reviewed-by: Björn Töpel <[email protected]>
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Pass patch_text() the length in bytes

patch_text_nosync() already handles an arbitrary length of code, so this
removes a superfluous loop and reduces the number of icache flushes.

Reviewed-b

riscv: Pass patch_text() the length in bytes

patch_text_nosync() already handles an arbitrary length of code, so this
removes a superfluous loop and reduces the number of icache flushes.

Reviewed-by: Björn Töpel <[email protected]>
Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Simplify text patching loops

This reduces the number of variables and makes the code easier to parse.

Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Björn Töpel <bjor

riscv: Simplify text patching loops

This reduces the number of variables and makes the code easier to parse.

Signed-off-by: Samuel Holland <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Fix text patching when IPI are used

For now, we use stop_machine() to patch the text and when we use IPIs for
remote icache flushes (which is emitted in patch_text_nosync()), the system
hangs

riscv: Fix text patching when IPI are used

For now, we use stop_machine() to patch the text and when we use IPIs for
remote icache flushes (which is emitted in patch_text_nosync()), the system
hangs.

So instead, make sure every CPU executes the stop_machine() patching
function and emit a local icache flush there.

Co-developed-by: Björn Töpel <[email protected]>
Signed-off-by: Björn Töpel <[email protected]>
Signed-off-by: Alexandre Ghiti <[email protected]>
Reviewed-by: Andrea Parri <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Remove superfluous smp_mb()

This memory barrier is not needed and not documented so simply remove
it.

Suggested-by: Andrea Parri <[email protected]>
Signed-off-by: Alexandre Ghiti <alexghi

riscv: Remove superfluous smp_mb()

This memory barrier is not needed and not documented so simply remove
it.

Suggested-by: Andrea Parri <[email protected]>
Signed-off-by: Alexandre Ghiti <[email protected]>
Reviewed-by: Andrea Parri <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Disable preemption when using patch_map()

patch_map() uses fixmap mappings to circumvent the non-writability of
the kernel text mapping.

The __set_fixmap() function only flushes the current

riscv: Disable preemption when using patch_map()

patch_map() uses fixmap mappings to circumvent the non-writability of
the kernel text mapping.

The __set_fixmap() function only flushes the current cpu tlb, it does
not emit an IPI so we must make sure that while we use a fixmap mapping,
the current task is not migrated on another cpu which could miss the
newly introduced fixmap mapping.

So in order to avoid any task migration, disable the preemption.

Reported-by: Andrea Parri <[email protected]>
Closes: https://lore.kernel.org/all/ZcS+GAaM25LXsBOl@andrea/
Reported-by: Andy Chiu <[email protected]>
Closes: https://lore.kernel.org/linux-riscv/CABgGipUMz3Sffu-CkmeUB1dKVwVQ73+7=sgC45-m0AE9RCjOZg@mail.gmail.com/
Fixes: cad539baa48f ("riscv: implement a memset like function for text")
Fixes: 0ff7c3b33127 ("riscv: Use text_mutex instead of patch_lock")
Co-developed-by: Andy Chiu <[email protected]>
Signed-off-by: Andy Chiu <[email protected]>
Signed-off-by: Alexandre Ghiti <[email protected]>
Acked-by: Puranjay Mohan <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Check if the code to patch lies in the exit section

Otherwise we fall through to vmalloc_to_page() which panics since the
address does not lie in the vmalloc region.

Fixes: 043cb41a85de ("ri

riscv: Check if the code to patch lies in the exit section

Otherwise we fall through to vmalloc_to_page() which panics since the
address does not lie in the vmalloc region.

Fixes: 043cb41a85de ("riscv: introduce interfaces to patch kernel code")
Signed-off-by: Alexandre Ghiti <[email protected]>
Reviewed-by: Charlie Jenkins <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: implement a memset like function for text

The BPF JIT needs to write invalid instructions to RX regions of memory to
invalidate removed BPF programs. This needs a function like memset() that

riscv: implement a memset like function for text

The BPF JIT needs to write invalid instructions to RX regions of memory to
invalidate removed BPF programs. This needs a function like memset() that
can work with RX memory.

Implement patch_text_set_nosync() which is similar to text_poke_set() of
x86.

Signed-off-by: Puranjay Mohan <[email protected]>
Reviewed-by: Pu Lehui <[email protected]>
Acked-by: Björn Töpel <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Acked-by: Daniel Borkmann <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: extend patch_text_nosync() for multiple pages

The patch_insn_write() function currently doesn't work for multiple pages
of instructions, therefore patch_text_nosync() will fail with a page fa

riscv: extend patch_text_nosync() for multiple pages

The patch_insn_write() function currently doesn't work for multiple pages
of instructions, therefore patch_text_nosync() will fail with a page fault
if called with lengths spanning multiple pages.

This commit extends the patch_insn_write() function to support multiple
pages by copying at max 2 pages at a time in a loop. This implementation
is similar to text_poke_copy() function of x86.

Signed-off-by: Puranjay Mohan <[email protected]>
Reviewed-by: Pu Lehui <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Acked-by: Daniel Borkmann <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

RISC-V: Don't check text_mutex during stop_machine

We're currently using stop_machine() to update ftrace & kprobes, which
means that the thread that takes text_mutex during may not be the same
as th

RISC-V: Don't check text_mutex during stop_machine

We're currently using stop_machine() to update ftrace & kprobes, which
means that the thread that takes text_mutex during may not be the same
as the thread that eventually patches the code. This isn't actually a
race because the lock is still held (preventing any other concurrent
accesses) and there is only one thread running during stop_machine(),
but it does trigger a lockdep failure.

This patch just elides the lockdep check during stop_machine.

Fixes: c15ac4fd60d5 ("riscv/ftrace: Add dynamic function tracer support")
Suggested-by: Steven Rostedt <[email protected]>
Reported-by: Changbin Du <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>
Signed-off-by: Conor Dooley <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Extend patch_text for multiple instructions

Extend patch_text for multiple instructions. This is the preparaiton for
multiple instructions text patching in riscv BPF trampoline, and may be
us

riscv: Extend patch_text for multiple instructions

Extend patch_text for multiple instructions. This is the preparaiton for
multiple instructions text patching in riscv BPF trampoline, and may be
useful for other scenario.

Signed-off-by: Pu Lehui <[email protected]>
Signed-off-by: Daniel Borkmann <[email protected]>
Tested-by: Björn Töpel <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
Acked-by: Björn Töpel <[email protected]>
Link: https://lore.kernel.org/bpf/[email protected]

show more ...

riscv: patch_text: Fixup last cpu should be master

These patch_text implementations are using stop_machine_cpuslocked
infrastructure with atomic cpu_count. The original idea: When the
master CPU pat

riscv: patch_text: Fixup last cpu should be master

These patch_text implementations are using stop_machine_cpuslocked
infrastructure with atomic cpu_count. The original idea: When the
master CPU patch_text, the others should wait for it. But current
implementation is using the first CPU as master, which couldn't
guarantee the remaining CPUs are waiting. This patch changes the
last CPU as the master to solve the potential risk.

Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Acked-by: Palmer Dabbelt <[email protected]>
Reviewed-by: Masami Hiramatsu <[email protected]>
Fixes: 043cb41a85de ("riscv: introduce interfaces to patch kernel code")
Cc: [email protected]
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Fixup compile error BUILD_BUG_ON failed

Unfortunately, the current code couldn't be compiled:

CC arch/riscv/kernel/patch.o
In file included from ./include/linux/kernel.h:11,

riscv: Fixup compile error BUILD_BUG_ON failed

Unfortunately, the current code couldn't be compiled:

CC arch/riscv/kernel/patch.o
In file included from ./include/linux/kernel.h:11,
from ./include/linux/list.h:9,
from ./include/linux/preempt.h:11,
from ./include/linux/spinlock.h:51,
from arch/riscv/kernel/patch.c:6:
In function ‘fix_to_virt’,
inlined from ‘patch_map’ at arch/riscv/kernel/patch.c:37:17:
./include/linux/compiler.h:392:38: error: call to ‘__compiletime_assert_205’ declared with attribute error: BUILD_BUG_ON failed: idx >= __end_of_fixed_addresses
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^
./include/linux/compiler.h:373:4: note: in definition of macro ‘__compiletime_assert’
prefix ## suffix(); \
^~~~~~
./include/linux/compiler.h:392:2: note: in expansion of macro ‘_compiletime_assert’
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro ‘compiletime_assert’
#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
^~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:50:2: note: in expansion of macro ‘BUILD_BUG_ON_MSG’
BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
^~~~~~~~~~~~~~~~
./include/asm-generic/fixmap.h:32:2: note: in expansion of macro ‘BUILD_BUG_ON’
BUILD_BUG_ON(idx >= __end_of_fixed_addresses);
^~~~~~~~~~~~

Because fix_to_virt(, idx) needs a const value, not a dynamic variable of
reg-a0 or BUILD_BUG_ON failed with "idx >= __end_of_fixed_addresses".

Signed-off-by: Guo Ren <[email protected]>
Reviewed-by: Masami Hiramatsu <[email protected]>
Reviewed-by: Pekka Enberg <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

maccess: rename probe_kernel_{read,write} to copy_{from,to}_kernel_nofault

Better describe what these functions do.

Suggested-by: Linus Torvalds <[email protected]>
Signed-off-by: Chris

maccess: rename probe_kernel_{read,write} to copy_{from,to}_kernel_nofault

Better describe what these functions do.

Suggested-by: Linus Torvalds <[email protected]>
Signed-off-by: Christoph Hellwig <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

show more ...

riscv: fix build warning of missing prototypes

Add the missing header in file, it was lost in original implementation.

The warning message as follows:
- no previous prototype for 'patch_text_nosyn

riscv: fix build warning of missing prototypes

Add the missing header in file, it was lost in original implementation.

The warning message as follows:
- no previous prototype for 'patch_text_nosync' [-Wmissing-prototypes]
- no previous prototype for 'patch_text' [-Wmissing-prototypes]

Changed in v2:
- Correct the typo of commit message.

Signed-off-by: Zong Li <[email protected]>
Reported-by: kbuild test robot <[email protected]>
Reviewed-by: Atish Patra <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Use text_mutex instead of patch_lock

We don't need the additional lock protection when patching the text.

There are two patching interfaces here:
- patch_text: patch code and always synchro

riscv: Use text_mutex instead of patch_lock

We don't need the additional lock protection when patching the text.

There are two patching interfaces here:
- patch_text: patch code and always synchronize with stop_machine()
- patch_text_nosync: patch code without synchronization, it's caller's
responsibility to synchronize all CPUs if needed.

For the first one, stop_machine() is protected by its own mutex, and
also the irq is already disabled here.

For the second one, in risc-v real case now, it would be used to ftrace
patching the mcount function, since it already running under
kstop_machine(), no other thread will run, so we could use text_mutex
on ftrace side.

Signed-off-by: Zong Li <[email protected]>
Reviewed-by: Masami Hiramatsu <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Use NOKPROBE_SYMBOL() instead of __krpobes annotation

The __kprobes annotation is old style, so change it to NOKPROBE_SYMBOL().

Signed-off-by: Zong Li <[email protected]>
Reviewed-by: Masam

riscv: Use NOKPROBE_SYMBOL() instead of __krpobes annotation

The __kprobes annotation is old style, so change it to NOKPROBE_SYMBOL().

Signed-off-by: Zong Li <[email protected]>
Reviewed-by: Masami Hiramatsu <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: Remove the 'riscv_' prefix of function name

Refactor the function name by removing the 'riscv_' prefix, it would be
better unless it could mix up with arch-independent functions.

Signed-off-

riscv: Remove the 'riscv_' prefix of function name

Refactor the function name by removing the 'riscv_' prefix, it would be
better unless it could mix up with arch-independent functions.

Signed-off-by: Zong Li <[email protected]>
Reviewed-by: Masami Hiramatsu <[email protected]>
Reviewed-by: Palmer Dabbelt <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...

riscv: introduce interfaces to patch kernel code

On strict kernel memory permission, we couldn't patch code without
writable permission. Preserve two holes in fixmap area, so we can map
the kernel c

riscv: introduce interfaces to patch kernel code

On strict kernel memory permission, we couldn't patch code without
writable permission. Preserve two holes in fixmap area, so we can map
the kernel code temporarily to fixmap area, then patch the instructions.

We need two pages here because we support the compressed instruction, so
the instruction might be align to 2 bytes. When patching the 32-bit
length instruction which is 2 bytes alignment, it will across two pages.

Introduce two interfaces to patch kernel code:
riscv_patch_text_nosync:
- patch code without synchronization, it's caller's responsibility to
synchronize all CPUs if needed.
riscv_patch_text:
- patch code and always synchronize with stop_machine()

Signed-off-by: Zong Li <[email protected]>
Signed-off-by: Palmer Dabbelt <[email protected]>

show more ...