Changes in traps.c

4769437d - arm64: Rely on generic printing of preemption model

Fri, 14 Mar 2025 16:08:05 +0000

arm64: Rely on generic printing of preemption model__die() invokes later show_regs() -> show_regs_print_info() which printsthe current preemption model.Remove it from the initial line.Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Acked-by: Will Deacon Acked-by: Catalin Marinas Link: https://lore.kernel.org/r/20250314160810.2373416-5-bigeasy@linutronix.de List of files: /linux-6.15/arch/arm64/kernel/traps.c

0c3beacf - asm-generic: introduce text-patching.h

Wed, 23 Oct 2024 16:27:06 +0000

asm-generic: introduce text-patching.hSeveral architectures support text patching, but they name the headerfiles that declare patching functions differently.Make all such headers consistently named text-patching.h and add an emptyheader in asm-generic for architectures that do not support text patching.Link: https://lkml.kernel.org/r/20241023162711.2579610-4-rppt@kernel.orgSigned-off-by: Mike Rapoport (Microsoft) Reviewed-by: Christoph Hellwig Acked-by: Geert Uytterhoeven # m68kAcked-by: Arnd Bergmann Reviewed-by: Luis Chamberlain Tested-by: kdevops Cc: Andreas Larsson Cc: Andy Lutomirski Cc: Ard Biesheuvel Cc: Borislav Petkov (AMD) Cc: Brian Cain Cc: Catalin Marinas Cc: Christophe Leroy Cc: Dave Hansen Cc: Dinh Nguyen Cc: Guo Ren Cc: Helge Deller Cc: Huacai Chen Cc: Ingo Molnar Cc: Johannes Berg Cc: John Paul Adrian Glaubitz Cc: Kent Overstreet Cc: Liam R. Howlett Cc: Mark Rutland Cc: Masami Hiramatsu (Google) Cc: Matt Turner Cc: Max Filippov Cc: Michael Ellerman Cc: Michal Simek Cc: Oleg Nesterov Cc: Palmer Dabbelt Cc: Peter Zijlstra Cc: Richard Weinberger Cc: Russell King Cc: Song Liu Cc: Stafford Horne Cc: Steven Rostedt (Google) Cc: Suren Baghdasaryan Cc: Thomas Bogendoerfer Cc: Thomas Gleixner Cc: Uladzislau Rezki (Sony) Cc: Vineet Gupta Cc: Will Deacon Signed-off-by: Andrew Morton List of files: /linux-6.15/arch/arm64/kernel/traps.c

13840229 - arm64: mops: Handle MOPS exceptions from EL1

Mon, 30 Sep 2024 16:10:48 +0000

arm64: mops: Handle MOPS exceptions from EL1We will soon be using MOPS instructions in the kernel, so wire up theexception handler to handle exceptions from EL1 caused by the copy/setoperation being stopped and resumed on a different type of CPU.Add a helper for advancing the single step state machine, similarly towhat the EL0 exception handler does.Signed-off-by: Kristina Martsenko Link: https://lore.kernel.org/r/20240930161051.3777828-3-kristina.martsenko@arm.comSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

8ce71d27 - arm64/traps: Handle GCS exceptions

Tue, 01 Oct 2024 22:58:58 +0000

arm64/traps: Handle GCS exceptionsA new exception code is defined for GCS specific faults other thanstandard load/store faults, for example GCS token validation failures,add handling for this. These faults are reported to userspace assegfaults with code SEGV_CPERR (protection error), mirroring thereporting for x86 shadow stack errors.GCS faults due to memory load/store operations generate data aborts witha flag set, these will be handled separately as part of the data aborthandling.Since we do not currently enable GCS for EL1 we should not get any faultsthere but while we're at it we wire things up there, treating any GCSfault as fatal.Reviewed-by: Thiago Jung Bauermann Reviewed-by: Catalin Marinas Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20241001-arm64-gcs-v13-19-222b78d87eee@kernel.orgSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

7f0ab607 - arm64: handle PKEY/POE faults

Thu, 22 Aug 2024 15:10:59 +0000

arm64: handle PKEY/POE faultsIf a memory fault occurs that is due to an overlay/pkey fault, report that touserspace with a SEGV_PKUERR.Signed-off-by: Joey Gouly Cc: Catalin Marinas Cc: Will Deacon Reviewed-by: Catalin Marinas Link: https://lore.kernel.org/r/20240822151113.1479789-17-joey.gouly@arm.com[will: Add ESR.FSC check to data abort handler]Signed-off-by: Will Deacon List of files: /linux-6.15/arch/arm64/kernel/traps.c

3e9e67e1 - arm64: Implement prctl(PR_{G,S}ET_TSC)

Sat, 24 Aug 2024 01:54:13 +0000

arm64: Implement prctl(PR_{G,S}ET_TSC)On arm64, this prctl controls access to CNTVCT_EL0, CNTVCTSS_EL0 andCNTFRQ_EL0 via CNTKCTL_EL1.EL0VCTEN. Since this bit is also used toimplement various erratum workarounds, check whether the CPU needsa workaround whenever we potentially need to change it.This is needed for a correct implementation of non-instrumentingrecord-replay debugging on arm64 (i.e. rr; https://rr-project.org/).rr must trap and record any sources of non-determinism from theuserspace program's perspective so it can be replayed later. Thisincludes the results of syscalls as well as the results of accessto architected timers exposed directly to the program. This prctlwas originally added for x86 by commit 8fb402bccf20 ("generic, x86:add prctl commands PR_GET_TSC and PR_SET_TSC"), and rr uses it totrap RDTSC on x86 for the same reason.We also considered exposing this as a PTRACE_EVENT. However, prctlseems like a better choice for these reasons:1) In general an in-process control seems more useful than an out-of-process control, since anything that you would be able to do with ptrace could also be done with prctl (tracer can inject a call to the prctl and handle signal-delivery-stops), and it avoids needing an additional process (which will complicate debugging of the ptraced process since it cannot have more than one tracer, and will be incompatible with ptrace_scope=3) in cases where that is not otherwise necessary.2) Consistency with x86_64. Note that on x86_64, RDTSC has been there since the start, so it's the same situation as on arm64.Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I233a1867d1ccebe2933a347552e7eae862344421Link: https://lore.kernel.org/r/20240824015415.488474-1-pcc@google.comSigned-off-by: Will Deacon List of files: /linux-6.15/arch/arm64/kernel/traps.c

7a928b32 - arm64: Introduce esr_brk_comment, esr_is_cfi_brk

Mon, 10 Jun 2024 06:32:35 +0000

arm64: Introduce esr_brk_comment, esr_is_cfi_brkAs it is already used in two places, move esr_comment() to a header forre-use, with a clearer name.Introduce esr_is_cfi_brk() to detect kCFI BRK syndromes, currently usedby early_brk64() but soon to also be used by hypervisor code.Signed-off-by: Pierre-Clément Tosi Acked-by: Will Deacon Link: https://lore.kernel.org/r/20240610063244.2828978-7-ptosi@google.comSigned-off-by: Oliver Upton List of files: /linux-6.15/arch/arm64/kernel/traps.c

d1e40f82 - arm64: Avoid cpus_have_const_cap() for ARM64_WORKAROUND_1542419

Mon, 16 Oct 2023 10:24:55 +0000

arm64: Avoid cpus_have_const_cap() for ARM64_WORKAROUND_1542419We use cpus_have_const_cap() to check for ARM64_WORKAROUND_1542419 butthis is not necessary and cpus_have_final_cap() would be preferable.For historical reasons, cpus_have_const_cap() is more complicated thanit needs to be. Before cpucaps are finalized, it will perform a bitmaptest of the system_cpucaps bitmap, and once cpucaps are finalized itwill use an alternative branch. This used to be necessary to handle somerace conditions in the window between cpucap detection and thesubsequent patching of alternatives and static branches, where differentbranches could be out-of-sync with one another (or w.r.t. alternativesequences). Now that we use alternative branches instead of staticbranches, these are all patched atomically w.r.t. one another, and thereare only a handful of cases that need special care in the window betweencpucap detection and alternative patching.Due to the above, it would be nice to remove cpus_have_const_cap(), andmigrate callers over to alternative_has_cap_*(), cpus_have_final_cap(),or cpus_have_cap() depending on when their requirements. This willremove redundant instructions and improve code generation, and will makeit easier to determine how each callsite will behave before, during, andafter alternative patching.The ARM64_WORKAROUND_1542419 cpucap is detected and patched before anyuserspace code can run, and the both __do_compat_cache_op() andctr_read_handler() are only reachable from exceptions taken fromuserspace. Thus it is not necessary for either to usecpus_have_const_cap(), and cpus_have_final_cap() is equivalent.This patch replaces the use of cpus_have_const_cap() withcpus_have_final_cap(), which will avoid generating code to test thesystem_cpucaps bitmap and should be better for all subsequent calls atruntime. Using cpus_have_final_cap() clearly documents that we do notexpect this code to run before cpucaps are finalized, and will make iteasier to spot issues if code is changed in future to allow thesefunctions to be reached earlier.Signed-off-by: Mark Rutland Cc: Suzuki K Poulose Cc: Will Deacon Signed-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

2de451a3 - KVM: arm64: Add handler for MOPS exceptions

Fri, 22 Sep 2023 11:25:07 +0000

KVM: arm64: Add handler for MOPS exceptionsAn Armv8.8 FEAT_MOPS main or epilogue instruction will take an exceptionif executed on a CPU with a different MOPS implementation option (A orB) than the CPU where the preceding prologue instruction ran. In thiscase the OS exception handler is expected to reset the registers andrestart execution from the prologue instruction.A KVM guest may use the instructions at EL1 at times when the guest isnot able to handle the exception, expecting that the instructions willonly run on one CPU (e.g. when running UEFI boot services in the guest).As KVM may reschedule the guest between different types of CPUs at anytime (on an asymmetric system), it needs to also handle the resultingexception itself in case the guest is not able to. A similar situationwill also occur in the future when live migrating a guest from one typeof CPU to another.Add handling for the MOPS exception to KVM. The handling can be sharedwith the EL0 exception handler, as the logic and register layouts arethe same. The exception can be handled right after exiting a guest,which avoids the cost of returning to the host exit handler.Similarly to the EL0 exception handler, in case the main or epilogueinstruction is being single stepped, it makes sense to finish the stepbefore executing the prologue instruction, so advance the single stepstate machine.Signed-off-by: Kristina Martsenko Reviewed-by: Marc Zyngier Link: https://lore.kernel.org/r/20230922112508.1774352-2-kristina.martsenko@arm.comSigned-off-by: Oliver Upton List of files: /linux-6.15/arch/arm64/kernel/traps.c

bb6e04a1 - kasan: use internal prototypes matching gcc-13 builtins

Tue, 09 May 2023 14:57:21 +0000

kasan: use internal prototypes matching gcc-13 builtinsgcc-13 warns about function definitions for builtin interfaces that have adifferent prototype, e.g.:In file included from kasan_test.c:31:kasan.h:574:6: error: conflicting types for built-in function '__asan_register_globals'; expected 'void(void *, long int)' [-Werror=builtin-declaration-mismatch] 574 | void __asan_register_globals(struct kasan_global *globals, size_t size);kasan.h:577:6: error: conflicting types for built-in function '__asan_alloca_poison'; expected 'void(void *, long int)' [-Werror=builtin-declaration-mismatch] 577 | void __asan_alloca_poison(unsigned long addr, size_t size);kasan.h:580:6: error: conflicting types for built-in function '__asan_load1'; expected 'void(void *)' [-Werror=builtin-declaration-mismatch] 580 | void __asan_load1(unsigned long addr);kasan.h:581:6: error: conflicting types for built-in function '__asan_store1'; expected 'void(void *)' [-Werror=builtin-declaration-mismatch] 581 | void __asan_store1(unsigned long addr);kasan.h:643:6: error: conflicting types for built-in function '__hwasan_tag_memory'; expected 'void(void *, unsigned char, long int)' [-Werror=builtin-declaration-mismatch] 643 | void __hwasan_tag_memory(unsigned long addr, u8 tag, unsigned long size);The two problems are: - Addresses are passes as 'unsigned long' in the kernel, but gcc-13 expects a 'void *'. - sizes meant to use a signed ssize_t rather than size_t.Change all the prototypes to match these. Using 'void *' consistently foraddresses gets rid of a couple of type casts, so push that down to theleaf functions where possible.This now passes all randconfig builds on arm, arm64 and x86, but I havenot tested it on the other architectures that support kasan, since theytend to fail randconfig builds in other ways. This might fail if any ofthe 32-bit architectures expect a 'long' instead of 'int' for the sizeargument.The __asan_allocas_unpoison() function prototype is somewhat weird, sinceit uses a pointer for 'stack_top' and an size_t for 'stack_bottom'. Thislooks like it is meant to be 'addr' and 'size' like the others, but theimplementation clearly treats them as 'top' and 'bottom'.Link: https://lkml.kernel.org/r/20230509145735.9263-2-arnd@kernel.orgSigned-off-by: Arnd Bergmann Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Andrey Ryabinin Cc: Dmitry Vyukov Cc: Marco Elver Cc: Vincenzo Frascino Cc: Signed-off-by: Andrew Morton List of files: /linux-6.15/arch/arm64/kernel/traps.c

8cd076a6 - arm64: mops: handle single stepping after MOPS exception

Tue, 09 May 2023 14:22:32 +0000

arm64: mops: handle single stepping after MOPS exceptionWhen a MOPS main or epilogue instruction is being executed, the task mayget scheduled on a different CPU and restart execution from the prologueinstruction. If the main or epilogue instruction is being single steppedthen it makes sense to finish the step and take the step exceptionbefore starting to execute the next (prologue) instruction. Sofast-forward the single step state machine when taking a MOPS exception.This means that if a main or epilogue instruction is single stepped withptrace, the debugger will sometimes observe the PC moving back to theprologue instruction. (As already mentioned, this should be rare as itonly happens when the task is scheduled to another CPU during the step.)This also ensures that perf breakpoints count prologue instructionsconsistently (i.e. every time they are executed), rather than skippingthem when there also happens to be a breakpoint on a main or epilogueinstruction.Acked-by: Catalin Marinas Signed-off-by: Kristina Martsenko Link: https://lore.kernel.org/r/20230509142235.3284028-9-kristina.martsenko@arm.comSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

8536ceaa - arm64: mops: handle MOPS exceptions

Tue, 09 May 2023 14:22:31 +0000

arm64: mops: handle MOPS exceptionsThe memory copy/set instructions added as part of FEAT_MOPS can take anexception (e.g. page fault) part-way through their execution and resumeexecution afterwards.If however the task is re-scheduled and execution resumes on a differentCPU, then the CPU may take a new type of exception to indicate this.This is because the architecture allows two options (Option A and OptionB) to implement the instructions and a heterogeneous system can havedifferent implementations between CPUs.In this case the OS has to reset the registers and restart executionfrom the prologue instruction. The algorithm for doing this is providedas part of the Arm ARM.Add an exception handler for the new exception and wire it up foruserspace tasks.Reviewed-by: Catalin Marinas Signed-off-by: Kristina Martsenko Link: https://lore.kernel.org/r/20230509142235.3284028-8-kristina.martsenko@arm.comSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

b925b431 - arm64: hide unused is_valid_bugaddr()

Tue, 16 May 2023 16:06:36 +0000

arm64: hide unused is_valid_bugaddr()When generic BUG() support is disabled, this function has no declarationand no callers but causes a W=1 warning:arch/arm64/kernel/traps.c:950:5: error: no previous prototype for 'is_valid_bugaddr' [-Werror=missing-prototypes]Add an #ifdef that matches the one around the declaration.Signed-off-by: Arnd Bergmann Reviewed-by: Kees Cook Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20230516160642.523862-10-arnd@kernel.orgSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

5ab6876c - arm64/cpu: Mark cpu_park_loop() and friends __noreturn

Wed, 12 Apr 2023 23:49:34 +0000

arm64/cpu: Mark cpu_park_loop() and friends __noreturnIn preparation for marking panic_smp_self_stop() __noreturn across thekernel, first mark the arm64 implementation of cpu_park_loop() andrelated functions __noreturn.Signed-off-by: Josh Poimboeuf Signed-off-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/r/55787d3193ea3e295ccbb097abfab0a10ae49d45.1681342859.git.jpoimboe@kernel.org List of files: /linux-6.15/arch/arm64/kernel/traps.c

b61b82f8 - arm64: pass ESR_ELx to die() of cfi_handler

Mon, 20 Feb 2023 07:34:41 +0000

arm64: pass ESR_ELx to die() of cfi_handlerCommit 0f2cb928a154 ("arm64: consistently pass ESR_ELx to die()") causedall callers to pass the ESR_ELx value to die().For consistency, this patch also adds esr to die() call of cfi_handler.Also, when CFI error occurs, die handlers can use ESR_ELx value.Signed-off-by: Sangmoon Kim Acked-by: Mark Rutland Reviewed-by: Mark Brown Link: https://lore.kernel.org/r/20230220073441.2753-1-sangmoon.kim@samsung.comSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

1d959312 - efi: arm64: Wire up BTI annotation in memory attributes table

Wed, 01 Feb 2023 08:50:07 +0000

efi: arm64: Wire up BTI annotation in memory attributes tableUEFI v2.10 extends the EFI memory attributes table with a flag thatindicates whether or not all RuntimeServicesCode regions wereconstructed with BTI landing pads, permitting the OS to map theseregions with BTI restrictions enabled.So let's take this into account on arm64.Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook Acked-by: Mark Rutland Reviewed-by: Will Deacon List of files: /linux-6.15/arch/arm64/kernel/traps.c

25b84002 - arm64: Support Clang UBSAN trap codes for better reporting

Thu, 02 Feb 2023 22:07:49 +0000

arm64: Support Clang UBSAN trap codes for better reportingWhen building with CONFIG_UBSAN_TRAP=y on arm64, Clang encodes the UBSANcheck (handler) type in the esr. Extract this and actually report thesetraps as coming from the specific UBSAN check that tripped.Before: Internal error: BRK handler: 00000000f20003e8 [#1] PREEMPT SMPAfter: Internal error: UBSAN: shift out of bounds: 00000000f2005514 [#1] PREEMPT SMPAcked-by: Mark Rutland Reviewed-by: Ard Biesheuvel Acked-by: Mukesh Ojha Reviewed-by: Fangrui Song Cc: Catalin Marinas Cc: Will Deacon Cc: John Stultz Cc: Yongqin Liu Cc: Sami Tolvanen Cc: Yury Norov Cc: Andrey Konovalov Cc: Marco Elver Cc: linux-arm-kernel@lists.infradead.orgCc: llvm@lists.linux.devSigned-off-by: Kees Cook List of files: /linux-6.15/arch/arm64/kernel/traps.c

a873bb49 - arm64: traps: attempt to dump all instructions

Fri, 27 Jan 2023 12:12:56 +0000

arm64: traps: attempt to dump all instructionsCurrently dump_kernel_instr() dumps a few instructions around thept_regs::pc value, dumping 4 instructions before the PC before dumpingthe instruction at the PC. If an attempt to read an instruction fails,it gives up and does not attempt to dump any subsequent instructions.This is unfortunate when the pt_regs::pc value points to the start of apage with a leading guard page, where the instruction at the PC can beread, but prior instructions cannot.This patch makes dump_kernel_instr() attempt to dump each instructionregardless of whether reading a prior instruction could be read, whichgives a more useful code dump in such cases. When an instruction cannotbe read, it is reported as "????????", which cannot be confused with ahex value,For example, with a `UDF #0` (AKA 0x00000000) early in the kexec controlpage, we'll now get the following code dump:| Internal error: Oops - Undefined instruction: 0000000002000000 [#1] SMP| Modules linked in:| CPU: 0 PID: 261 Comm: kexec Not tainted 6.2.0-rc5+ #26| Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015| pstate: 604003c5 (nZCv DAIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)| pc : 0x48c00000| lr : machine_kexec+0x190/0x200| sp : ffff80000d36ba80| x29: ffff80000d36ba80 x28: ffff000002dfc380 x27: 0000000000000000| x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000| x23: ffff80000a9f7858 x22: 000000004c460000 x21: 0000000000000010| x20: 00000000ad821000 x19: ffff000000aa0000 x18: 0000000000000006| x17: ffff8000758a2000 x16: ffff800008000000 x15: ffff80000d36b568| x14: 0000000000000000 x13: ffff80000d36b707 x12: ffff80000a9bf6e0| x11: 00000000ffffdfff x10: ffff80000aaaf8e0 x9 : ffff80000815eff8| x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 00000000000affa8| x5 : 0000000000001fff x4 : 0000000000000001 x3 : ffff80000a263008| x2 : ffff80000a9e20f8 x1 : 0000000048c00000 x0 : ffff000000aa0000| Call trace:| 0x48c00000| kernel_kexec+0x88/0x138| __do_sys_reboot+0x108/0x288| __arm64_sys_reboot+0x2c/0x40| invoke_syscall+0x78/0x140| el0_svc_common.constprop.0+0x4c/0x100| do_el0_svc+0x34/0x80| el0_svc+0x34/0x140| el0t_64_sync_handler+0xf4/0x140| el0t_64_sync+0x194/0x1c0| Code: ???????? ???????? ???????? ???????? (00000000)| ---[ end trace 0000000000000000 ]---| Kernel panic - not syncing: Oops - Undefined instruction: Fatal exception| Kernel Offset: disabled| CPU features: 0x002000,00050108,c8004203| Memory Limit: noneSigned-off-by: Mark Rutland Cc: Ard Biesheuvel Cc: James Morse Cc: Will Deacon Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20230127121256.2141368-1-mark.rutland@arm.comSigned-off-by: Catalin Marinas List of files: /linux-6.15/arch/arm64/kernel/traps.c

124c49b1 - arm64: armv8_deprecated: rework deprected instruction handling

Wed, 19 Oct 2022 14:41:23 +0000

arm64: armv8_deprecated: rework deprected instruction handlingSupport for deprecated instructions can be enabled or disabled atruntime. To handle this, the code in armv8_deprecated.c registers andunregisters undef_hooks, and makes cross CPU calls to configure HWsupport. This is rather complicated, and the synchronization required tomake this safe ends up serializing the handling of instructions whichhave been trapped.This patch simplifies the deprecated instruction handling by removingthe dynamic registration and unregistration, and changing the traphandling code to determine whether a handler should be invoked. Thisremoves the need for dynamic list management, and simplifies the lockingrequirements, making it possible to handle trapped instructions entirelyin parallel.Where changing the emulation state requires a cross-call, this isserialized by locally disabling interrupts, ensuring that the CPU is notleft in an inconsistent state.To simplify sysctl management, each insn_emulation is given a separatesysctl table, permitting these to be registered separately. The coresysctl code will iterate over all of these when walking sysfs.I've tested this with userspace programs which use each of thedeprecated instructions, and I've concurrently modified the supportlevel for each of the features back-and-forth between HW and emulated tocheck that there are no spurious SIGILLs sent to userspace when thesupport level is changed.Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Joey Gouly Cc: Peter Zijlstra Cc: Will Deacon Link: https://lore.kernel.org/r/20221019144123.612388-10-mark.rutland@arm.comSigned-off-by: Will Deacon List of files: /linux-6.15/arch/arm64/kernel/traps.c

f5962add - arm64: rework EL0 MRS emulation

Wed, 19 Oct 2022 14:41:19 +0000

arm64: rework EL0 MRS emulationOn CPUs without FEAT_IDST, ID register emulation is slower than it needsto be, as all threads contend for the same lock to perform theemulation. This patch reworks the emulation to avoid this unnecessarycontention.On CPUs with FEAT_IDST (which is mandatory from ARMv8.4 onwards), EL0accesses to ID registers result in a SYS trap, and emulation of these ishandled with a sys64_hook. These hooks are statically allocated, and nolocking is required to iterate through the hooks and perform theemulation, allowing emulation to occur in parallel with no contention.On CPUs without FEAT_IDST, EL0 accesses to ID registers result in anUNDEFINED exception, and emulation of these accesses is handled with anundef_hook. When an EL0 MRS instruction is trapped to EL1, the kernelfinds the relevant handler by iterating through all of the undef_hooks,requiring undef_lock to be held during this lookup.This locking is only required to safely traverse the list of undef_hooks(as it can be concurrently modified), and the actual emulation of theMRS does not require any mutual exclusion. This locking is anunfortunate bottleneck, especially given that MRS emulation is enabledunconditionally and is never disabled.This patch reworks the non-FEAT_IDST MRS emulation logic so that it canbe invoked directly from do_el0_undef(). This removes the bottleneck,allowing MRS traps to be handled entirely in parallel, and is a steppingstone to making all of the undef_hooks lock-free.I've tested this in a 64-vCPU VM on a 64-CPU ThunderX2 host, with abenchmark which spawns a number of threads which each try to readID_AA64ISAR0_EL1 1000000 times. This is vastly more contention than willever be seen in realistic usage, but clearly demonstrates the removal ofthe bottleneck: | Threads || Time (seconds) | | || Before || After | | || Real | System || Real | System | |---------++--------+---------++--------+---------| | 1 || 0.29 | 0.20 || 0.24 | 0.12 | | 2 || 0.35 | 0.51 || 0.23 | 0.27 | | 4 || 1.08 | 3.87 || 0.24 | 0.56 | | 8 || 4.31 | 33.60 || 0.24 | 1.11 | | 16 || 9.47 | 149.39 || 0.23 | 2.15 | | 32 || 19.07 | 605.27 || 0.24 | 4.38 | | 64 || 65.40 | 3609.09 || 0.33 | 11.27 |Aside from the speedup, there should be no functional change as a resultof this patch.Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: James Morse Cc: Joey Gouly Cc: Peter Zijlstra Cc: Will Deacon Link: https://lore.kernel.org/r/20221019144123.612388-6-mark.rutland@arm.comSigned-off-by: Will Deacon List of files: /linux-6.15/arch/arm64/kernel/traps.c