arm64/arch_timer: include <linux/percpu.h>

arch_timer.h includes linux/smp.h since the commit:

6acc71ccac7187fc ("arm64: arch_timer: Allows a CPU-specific erratum to only affect a subset of CPUs"

arm64/arch_timer: include <linux/percpu.h>

arch_timer.h includes linux/smp.h since the commit:

6acc71ccac7187fc ("arm64: arch_timer: Allows a CPU-specific erratum to only affect a subset of CPUs")

It was included to use DEFINE_PER_CPU(), etc. But It should have
included <linux/percpu.h> rather than <linux/smp.h>. It worked because
smp.h includes percpu.h.

The next commit will remove percpu.h from smp.h and it will break this
usage.

Explicitly include percpu.h and remove smp.h

Signed-off-by: Puranjay Mohan <[email protected]>
Acked-by: Mark Rutland <[email protected]>
Reviewed-by: Stephen Boyd <[email protected]>
Reviewed-by: Anshuman Khandual <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Catalin Marinas <[email protected]>

show more ...

arm64/arch_timer: Provide noinstr sched_clock_read() functions

With the intent to provide local_clock_noinstr(), a variant of
local_clock() that's safe to be called from noinstr code (with the
assum

arm64/arch_timer: Provide noinstr sched_clock_read() functions

With the intent to provide local_clock_noinstr(), a variant of
local_clock() that's safe to be called from noinstr code (with the
assumption that any such code will already be non-preemptible),
prepare for things by providing a noinstr sched_clock_read() function.

Specifically, preempt_enable_*() calls out to schedule(), which upsets
noinstr validation efforts.

Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Tested-by: Michael Kelley <[email protected]> # Hyper-V
Link: https://lore.kernel.org/r/[email protected]

show more ...

arm64: Add CNT{P,V}CTSS_EL0 alternatives to cnt{p,v}ct_el0

CNTPCTSS_EL0 and CNTVCTSS_EL0 are alternatives to the usual
CNTPCT_EL0 and CNTVCT_EL0 that do not require a previous ISB
to be synchronised

arm64: Add CNT{P,V}CTSS_EL0 alternatives to cnt{p,v}ct_el0

CNTPCTSS_EL0 and CNTVCTSS_EL0 are alternatives to the usual
CNTPCT_EL0 and CNTVCT_EL0 that do not require a previous ISB
to be synchronised (SS stands for Self-Synchronising).

Use the ARM64_HAS_ECV capability to control alternative sequences
that switch to these low(er)-cost primitives. Note that the
counter access in the VDSO is for now left alone until we decide
whether we want to allow this.

Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/drivers/arch_arm_timer: Move workaround synchronisation around

We currently handle synchronisation when workarounds are enabled
by having an ISB in the __arch_counter_get_cnt?ct_stable()

clocksource/drivers/arch_arm_timer: Move workaround synchronisation around

We currently handle synchronisation when workarounds are enabled
by having an ISB in the __arch_counter_get_cnt?ct_stable() helpers.

While this works, this prevents us from relaxing this synchronisation.

Instead, move it closer to the point where the synchronisation is
actually needed. Further patches will subsequently relax this.

Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Drop unnecessary ISB on CVAL programming

Switching from TVAL to CVAL has a small drawback: we need an ISB
before reading the counter. We cannot get rid of it, but

clocksource/drivers/arm_arch_timer: Drop unnecessary ISB on CVAL programming

Switching from TVAL to CVAL has a small drawback: we need an ISB
before reading the counter. We cannot get rid of it, but we can
instead remove the one that comes just after writing to CVAL.

This reduces the number of ISBs from 3 to 2 when programming
the timer.

Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Move system register timer programming over to CVAL

In order to cope better with high frequency counters, move the
programming of the timers from the countdown ti

clocksource/drivers/arm_arch_timer: Move system register timer programming over to CVAL

In order to cope better with high frequency counters, move the
programming of the timers from the countdown timer (TVAL) over
to the comparator (CVAL).

The programming model is slightly different, as we now need to
read the current counter value to have an absolute deadline
instead of a relative one.

There is a small overhead to this change, which we will address
in the following patches.

Reviewed-by: Oliver Upton <[email protected]>
Reviewed-by: Mark Rutland <[email protected]>
Tested-by: Mark Rutland <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Extend write side of timer register accessors to u64

The various accessors for the timer sysreg and MMIO registers are
currently hardwired to 32bit. However, we a

clocksource/drivers/arm_arch_timer: Extend write side of timer register accessors to u64

The various accessors for the timer sysreg and MMIO registers are
currently hardwired to 32bit. However, we are about to introduce
the use of the CVAL registers, which require a 64bit access.

Upgrade the write side of the accessors to take a 64bit value
(the read side is left untouched as we don't plan to ever read
back any of these registers).

No functional change expected.

Reviewed-by: Oliver Upton <[email protected]>
Reviewed-by: Mark Rutland <[email protected]>
Tested-by: Mark Rutland <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Drop CNT*_TVAL read accessors

The arch timer driver never reads the various TVAL registers, only
writes to them. It is thus pointless to provide accessors
for the

clocksource/drivers/arm_arch_timer: Drop CNT*_TVAL read accessors

The arch timer driver never reads the various TVAL registers, only
writes to them. It is thus pointless to provide accessors
for them and to implement errata workarounds.

Drop these read-side accessors, and add a couple of BUG() statements
for the time being. These statements will be removed further down
the line.

Reviewed-by: Oliver Upton <[email protected]>
Reviewed-by: Mark Rutland <[email protected]>
Tested-by: Mark Rutland <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

clocksource/arm_arch_timer: Add build-time guards for unhandled register accesses

As we are about to change the registers that are used by the driver,
start by adding build-time checks to ensure tha

clocksource/arm_arch_timer: Add build-time guards for unhandled register accesses

As we are about to change the registers that are used by the driver,
start by adding build-time checks to ensure that we always handle
all registers and access modes.

Suggested-by: Mark Rutland <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

arm64: vdso: Avoid ISB after reading from cntvct_el0

We can avoid the expensive ISB instruction after reading the counter in
the vDSO gettime functions by creating a fake address hazard against a
du

arm64: vdso: Avoid ISB after reading from cntvct_el0

We can avoid the expensive ISB instruction after reading the counter in
the vDSO gettime functions by creating a fake address hazard against a
dummy stack read, just like we do inside the kernel.

Signed-off-by: Will Deacon <[email protected]>
Reviewed-by: Vincenzo Frascino <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Catalin Marinas <[email protected]>

show more ...

arm64: arch_timer: Allow an workaround descriptor to disable compat vdso

As we are about to disable the vdso for compat tasks in some circumstances,
let's allow a workaround descriptor to express ex

arm64: arch_timer: Allow an workaround descriptor to disable compat vdso

As we are about to disable the vdso for compat tasks in some circumstances,
let's allow a workaround descriptor to express exactly that.

Signed-off-by: Marc Zyngier <[email protected]>
Acked-by: Mark Rutland <[email protected]>
Cc: [email protected]
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Extract elf_hwcap use to arch-helper

Different mechanisms are used to test and set elf_hwcaps between ARM
and ARM64, this results in the use of ifdeferry in this

clocksource/drivers/arm_arch_timer: Extract elf_hwcap use to arch-helper

Different mechanisms are used to test and set elf_hwcaps between ARM
and ARM64, this results in the use of ifdeferry in this file when
setting/testing for the EVTSTRM hwcap.

Let's improve readability by extracting this to an arch helper.

Signed-off-by: Andrew Murray <[email protected]>
Acked-by: Mark Rutland <[email protected]>
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 234

Based on 1 normalized pattern(s):

this program is free software you can redistribute it and or modify
it under the terms of th

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 234

Based on 1 normalized pattern(s):

this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation this program is
distributed in the hope that it will be useful but without any
warranty without even the implied warranty of merchantability or
fitness for a particular purpose see the gnu general public license
for more details you should have received a copy of the gnu general
public license along with this program if not see http www gnu org
licenses

extracted by the scancode license scanner the SPDX license identifier

GPL-2.0-only

has been chosen to replace the boilerplate/reference in 503 file(s).

Signed-off-by: Thomas Gleixner <[email protected]>
Reviewed-by: Alexios Zavras <[email protected]>
Reviewed-by: Allison Randal <[email protected]>
Reviewed-by: Enrico Weigelt <[email protected]>
Cc: [email protected]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Greg Kroah-Hartman <[email protected]>

show more ...

arm64: arch_timer: mark functions as __always_inline

If CONFIG_FUNCTION_GRAPH_TRACER is enabled function
arch_counter_get_cntvct() is marked as notrace. However, function
__arch_counter_get_cntvct i

arm64: arch_timer: mark functions as __always_inline

If CONFIG_FUNCTION_GRAPH_TRACER is enabled function
arch_counter_get_cntvct() is marked as notrace. However, function
__arch_counter_get_cntvct is marked as inline. If
CONFIG_OPTIMIZE_INLINING is set that will make the two functions
tracable which they shouldn't.

Rework so that functions __arch_counter_get_* are marked with
__always_inline so they will be inlined even if CONFIG_OPTIMIZE_INLINING
is turned on.

Fixes: 0ea415390cd3 ("clocksource/arm_arch_timer: Use arch_timer_read_counter to access stable counters")
Acked-by: Marc Zyngier <[email protected]>
Signed-off-by: Anders Roxell <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/arm_arch_timer: Use arch_timer_read_counter to access stable counters

Instead of always going via arch_counter_get_cntvct_stable to access the
counter workaround, let's have arch_timer_r

clocksource/arm_arch_timer: Use arch_timer_read_counter to access stable counters

Instead of always going via arch_counter_get_cntvct_stable to access the
counter workaround, let's have arch_timer_read_counter point to the
right method.

For that, we need to track whether any CPU in the system has a
workaround for the counter. This is done by having an atomic variable
tracking this.

Acked-by: Mark Rutland <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/arm_arch_timer: Remove use of workaround static key

The use of a static key in a hotplug path has proved to be a real
nightmare, and makes it impossible to have scream-free lockdep
kerne

clocksource/arm_arch_timer: Remove use of workaround static key

The use of a static key in a hotplug path has proved to be a real
nightmare, and makes it impossible to have scream-free lockdep
kernel.

Let's remove the static key altogether, and focus on something saner.

Acked-by: Mark Rutland <[email protected]>
Acked-by: Daniel Lezcano <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/arm_arch_timer: Drop use of static key in arch_timer_reg_read_stable

Let's start with the removal of the arch_timer_read_ool_enabled
static key in arch_timer_reg_read_stable. It is not a

clocksource/arm_arch_timer: Drop use of static key in arch_timer_reg_read_stable

Let's start with the removal of the arch_timer_read_ool_enabled
static key in arch_timer_reg_read_stable. It is not a fast path,
and we can simplify things a bit.

Acked-by: Mark Rutland <[email protected]>
Acked-by: Daniel Lezcano <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/arm_arch_timer: Direcly assign set_next_event workaround

When a given timer is affected by an erratum and requires an
alternative implementation of set_next_event, we do a rather
complic

clocksource/arm_arch_timer: Direcly assign set_next_event workaround

When a given timer is affected by an erratum and requires an
alternative implementation of set_next_event, we do a rather
complicated dance to detect and call the workaround on each
set_next_event call.

This is clearly idiotic, as we can perfectly detect whether
this CPU requires a workaround while setting up the clock event
device.

This only requires the CPU-specific detection to be done a bit
earlier, and we can then safely override the set_next_event pointer
if we have a workaround associated to that CPU.

Acked-by: Mark Rutland <[email protected]>
Acked-by; Daniel Lezcano <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

arm64: arch_timer: Ensure counter register reads occur with seqlock held

When executing clock_gettime(), either in the vDSO or via a system call,
we need to ensure that the read of the counter regis

arm64: arch_timer: Ensure counter register reads occur with seqlock held

When executing clock_gettime(), either in the vDSO or via a system call,
we need to ensure that the read of the counter register occurs within
the seqlock reader critical section. This ensures that updates to the
clocksource parameters (e.g. the multiplier) are consistent with the
counter value and therefore avoids the situation where time appears to
go backwards across multiple reads.

Extend the vDSO logic so that the seqlock critical section covers the
read of the counter register as well as accesses to the data page. Since
reads of the counter system registers are not ordered by memory barrier
instructions, introduce dependency ordering from the counter read to a
subsequent memory access so that the seqlock memory barriers apply to
the counter access in both the vDSO and the system call paths.

Cc: <[email protected]>
Cc: Marc Zyngier <[email protected]>
Tested-by: Vincenzo Frascino <[email protected]>
Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
Reported-by: Thomas Gleixner <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

arm64: Implement arch_counter_get_cntpct to read the physical counter

As we are about to use the physical counter on arm64 systems that have
KVM support, implement arch_counter_get_cntpct() and the

arm64: Implement arch_counter_get_cntpct to read the physical counter

As we are about to use the physical counter on arm64 systems that have
KVM support, implement arch_counter_get_cntpct() and the associated
errata workaround functionality for stable timer reads.

Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Signed-off-by: Christoffer Dall <[email protected]>

show more ...

arm_arch_timer: Expose event stream status

The arch timer configuration for a CPU might get reset after suspending
said CPU.

In order to reliably use the event stream in the kernel (e.g. for delays

arm_arch_timer: Expose event stream status

The arch timer configuration for a CPU might get reset after suspending
said CPU.

In order to reliably use the event stream in the kernel (e.g. for delays),
we keep track of the state where we can safely consider the event stream as
properly configured. After writing to cntkctl, we issue an ISB to ensure
that subsequent delay loops can rely on the event stream being enabled.

Signed-off-by: Julien Thierry <[email protected]>
Acked-by: Mark Rutland <[email protected]>
Cc: Marc Zyngier <[email protected]>
Cc: Russell King <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Will Deacon <[email protected]>

show more ...

clocksource/drivers/arm_arch_timer: Avoid infinite recursion when ftrace is enabled

On platforms with an arch timer erratum workaround, it's possible for
arch_timer_reg_read_stable() to recurse into

clocksource/drivers/arm_arch_timer: Avoid infinite recursion when ftrace is enabled

On platforms with an arch timer erratum workaround, it's possible for
arch_timer_reg_read_stable() to recurse into itself when certain
tracing options are enabled, leading to stack overflows and related
problems.

For example, when PREEMPT_TRACER and FUNCTION_GRAPH_TRACER are
selected, it's possible to trigger this with:

$ mount -t debugfs nodev /sys/kernel/debug/
$ echo function_graph > /sys/kernel/debug/tracing/current_tracer

The problem is that in such cases, preempt_disable() instrumentation
attempts to acquire a timestamp via trace_clock(), resulting in a call
back to arch_timer_reg_read_stable(), and hence recursion.

This patch changes arch_timer_reg_read_stable() to use
preempt_{disable,enable}_notrace(), which avoids this.

This problem is similar to the fixed by upstream commit 96b3d28bf4
("sched/clock: Prevent tracing recursion in sched_clock_cpu()").

Fixes: 6acc71ccac71 ("arm64: arch_timer: Allows a CPU-specific erratum to only affect a subset of CPUs")
Signed-off-by: Ding Tianhong <[email protected]>
Acked-by: Mark Rutland <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Signed-off-by: Daniel Lezcano <[email protected]>

show more ...

arm64: arch_timer: Allow erratum matching with ACPI OEM information

Just as we're able to identify a broken platform using some DT
information, let's enable a way to spot the offenders with ACPI.

T

arm64: arch_timer: Allow erratum matching with ACPI OEM information

Just as we're able to identify a broken platform using some DT
information, let's enable a way to spot the offenders with ACPI.

The difference is that we can only match on some OEM info instead
of implementation-specific properties. So in order to avoid the
insane multiplication of errata structures, we allow an array
of OEM descriptions to be attached to an erratum structure.

Acked-by: Thomas Gleixner <[email protected]>
Tested-by: dann frazier <[email protected]>
Tested-by: Hanjun Guo <[email protected]>
Reviewed-by: Hanjun Guo <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>

show more ...

arm64: arch_timer: Allows a CPU-specific erratum to only affect a subset of CPUs

Instead of applying a CPU-specific workaround to all CPUs in the system,
allow it to only affect a subset of them (ty

arm64: arch_timer: Allows a CPU-specific erratum to only affect a subset of CPUs

Instead of applying a CPU-specific workaround to all CPUs in the system,
allow it to only affect a subset of them (typical big-little case).

This is done by turning the erratum pointer into a per-CPU variable.

Acked-by: Thomas Gleixner <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>

show more ...

arm64: arch_timer: Make workaround methods optional

Not all errata need to workaround all access types. Allow them to
be optional.

Acked-by: Thomas Gleixner <[email protected]>
Signed-off-by: Marc

arm64: arch_timer: Make workaround methods optional

Not all errata need to workaround all access types. Allow them to
be optional.

Acked-by: Thomas Gleixner <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>

show more ...