locking/atomic: make atomic*_{cmp,}xchg optional

Most architectures define the atomic/atomic64 xchg and cmpxchg
operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for the

locking/atomic: make atomic*_{cmp,}xchg optional

Most architectures define the atomic/atomic64 xchg and cmpxchg
operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch
code. On some architectures the existing definitions are kept as these
are used to build other arch_atomic*() operations.

Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Reviewed-by: Kees Cook <[email protected]>
Link: https://lore.kernel.org/r/[email protected]

show more ...

MIPS: retire "asm/llsc.h"

all that "asm/llsc.h" does is just to help inline asm, which can be
stringifyed from "asm/asm.h"

+. Since "asm/asm.h" has all we need, retire "asm/llsc.h"

+. remove unuse

MIPS: retire "asm/llsc.h"

all that "asm/llsc.h" does is just to help inline asm, which can be
stringifyed from "asm/asm.h"

+. Since "asm/asm.h" has all we need, retire "asm/llsc.h"

+. remove unused header file

Inspired-by: Maciej W. Rozycki <[email protected]>
Signed-off-by: Huang Pei <[email protected]>
Signed-off-by: Thomas Bogendoerfer <[email protected]>

show more ...

MIPS: locking/atomic: Fix atomic{_64,}_sub_if_positive

This looks like a typo and that caused atomic64 test failed.

Signed-off-by: Rui Wang <[email protected]>
Signed-off-by: Thomas Bogendoerfer

MIPS: locking/atomic: Fix atomic{_64,}_sub_if_positive

This looks like a typo and that caused atomic64 test failed.

Signed-off-by: Rui Wang <[email protected]>
Signed-off-by: Thomas Bogendoerfer <[email protected]>

show more ...

locking/atomic: mips: move to ARCH_ATOMIC

We'd like all architectures to convert to ARCH_ATOMIC, as once all
architectures are converted it will be possible to make significant
cleanups to the atomi

locking/atomic: mips: move to ARCH_ATOMIC

We'd like all architectures to convert to ARCH_ATOMIC, as once all
architectures are converted it will be possible to make significant
cleanups to the atomics headers, and this will make it much easier to
generically enable atomic functionality (e.g. debug logic in the
instrumented wrappers).

As a step towards that, this patch migrates mips to ARCH_ATOMIC. The
arch code provides arch_{atomic,atomic64,xchg,cmpxchg}*(), and common
code wraps these with optional instrumentation to provide the regular
functions.

Signed-off-by: Mark Rutland <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Bogendoerfer <[email protected]>
Cc: Will Deacon <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Link: https://lore.kernel.org/r/[email protected]

show more ...

MIPS: Compare __SYNC_loongson3_war against 0

When building with clang when CONFIG_CPU_LOONGSON3_WORKAROUNDS is
enabled:

In file included from lib/errseq.c:4:
In file included from ./include/linux

MIPS: Compare __SYNC_loongson3_war against 0

When building with clang when CONFIG_CPU_LOONGSON3_WORKAROUNDS is
enabled:

In file included from lib/errseq.c:4:
In file included from ./include/linux/atomic.h:7:
./arch/mips/include/asm/atomic.h:52:1: warning: converting the result of
'<<' to a boolean always evaluates to true
[-Wtautological-constant-compare]
ATOMIC_OPS(atomic64, s64)
^
./arch/mips/include/asm/atomic.h:40:9: note: expanded from macro
'ATOMIC_OPS'
return cmpxchg(&v->counter, o, n);
^
./arch/mips/include/asm/cmpxchg.h:194:7: note: expanded from macro
'cmpxchg'
if (!__SYNC_loongson3_war)
^
./arch/mips/include/asm/sync.h:147:34: note: expanded from macro
'__SYNC_loongson3_war'
# define __SYNC_loongson3_war (1 << 31)
^

While it is not wrong that the result of this shift is always true in a
boolean context, it is not a problem here. Regardless, the warning is
really noisy so rather than making the shift a boolean implicitly, use
it in an equality comparison so the shift is used as an integer value.

Fixes: 4d1dbfe6cbec ("MIPS: atomic: Emit Loongson3 sync workarounds within asm")
Fixes: a91f2a1dba44 ("MIPS: cmpxchg: Omit redundant barriers for Loongson3")
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Nathan Chancellor <[email protected]>
Acked-by: Nick Desaulniers <[email protected]>
Signed-off-by: Thomas Bogendoerfer <[email protected]>

show more ...

locking/atomic: Move ATOMIC_INIT into linux/types.h

This patch moves ATOMIC_INIT from asm/atomic.h into linux/types.h.
This allows users of atomic_t to use ATOMIC_INIT without having to
include atom

locking/atomic: Move ATOMIC_INIT into linux/types.h

This patch moves ATOMIC_INIT from asm/atomic.h into linux/types.h.
This allows users of atomic_t to use ATOMIC_INIT without having to
include atomic.h as that way may lead to header loops.

Signed-off-by: Herbert Xu <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Acked-by: Waiman Long <[email protected]>
Link: https://lkml.kernel.org/r/[email protected]

show more ...

MIPS: atomic: Deduplicate 32b & 64b read, set, xchg, cmpxchg

Remove the remaining duplication between 32b & 64b in asm/atomic.h by
making use of an ATOMIC_OPS() macro to generate:

- atomic_read()

MIPS: atomic: Deduplicate 32b & 64b read, set, xchg, cmpxchg

Remove the remaining duplication between 32b & 64b in asm/atomic.h by
making use of an ATOMIC_OPS() macro to generate:

- atomic_read()/atomic64_read()
- atomic_set()/atomic64_set()
- atomic_cmpxchg()/atomic64_cmpxchg()
- atomic_xchg()/atomic64_xchg()

This is consistent with the way all other functions in asm/atomic.h are
generated, and ensures consistency between the 32b & 64b functions.

Of note is that this results in the above now being static inline
functions rather than macros.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Unify 32b & 64b sub_if_positive

Unify the definitions of atomic_sub_if_positive() &
atomic64_sub_if_positive() using a macro like we do for most other
atomic functions. This allows us

MIPS: atomic: Unify 32b & 64b sub_if_positive

Unify the definitions of atomic_sub_if_positive() &
atomic64_sub_if_positive() using a macro like we do for most other
atomic functions. This allows us to share the implementation ensuring
consistency between the two. Notably this provides the appropriate
loongson3_war barriers in the atomic64_sub_if_positive() case which were
previously missing.

The code is rearranged a little to handle the !kernel_uses_llsc case
first in order to de-indent the LL/SC case & allow us not to go over 80
characters per line.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Use _atomic barriers in atomic_sub_if_positive()

Use smp_mb__before_atomic() & smp_mb__after_atomic() in
atomic_sub_if_positive() rather than the equivalent
smp_mb__before_llsc() & smp

MIPS: atomic: Use _atomic barriers in atomic_sub_if_positive()

Use smp_mb__before_atomic() & smp_mb__after_atomic() in
atomic_sub_if_positive() rather than the equivalent
smp_mb__before_llsc() & smp_llsc_mb(). The former are more standard &
this preps us for avoiding redundant duplicate barriers on Loongson3 in
a later patch.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Emit Loongson3 sync workarounds within asm

Generate the sync instructions required to workaround Loongson3 LL/SC
errata within inline asm blocks, which feels a little safer than doing

MIPS: atomic: Emit Loongson3 sync workarounds within asm

Generate the sync instructions required to workaround Loongson3 LL/SC
errata within inline asm blocks, which feels a little safer than doing
it from C where strictly speaking the compiler would be well within its
rights to insert a memory access between the separate asm statements we
previously had, containing sync & ll instructions respectively.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Use one macro to generate 32b & 64b functions

Cut down on duplication by generalizing the ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros to work for both 32b &
64b atomics,

MIPS: atomic: Use one macro to generate 32b & 64b functions

Cut down on duplication by generalizing the ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros to work for both 32b &
64b atomics, and removing the ATOMIC64_ variants. This ensures
consistency between our atomic_* & atomic64_* functions.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Handle !kernel_uses_llsc first

Handle the !kernel_uses_llsc path first in our ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros & return from within the
block. This allows us

MIPS: atomic: Handle !kernel_uses_llsc first

Handle the !kernel_uses_llsc path first in our ATOMIC_OP(),
ATOMIC_OP_RETURN() & ATOMIC_FETCH_OP() macros & return from within the
block. This allows us to de-indent the kernel_uses_llsc path by one
level which will be useful when making further changes.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: atomic: Fix whitespace in ATOMIC_OP macros

We define macros in asm/atomic.h which end each line with space
characters before a backslash to continue on the next line. Remove the
space characte

MIPS: atomic: Fix whitespace in ATOMIC_OP macros

We define macros in asm/atomic.h which end each line with space
characters before a backslash to continue on the next line. Remove the
space characters leaving tabs as the whitespace used for conformity with
coding convention.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

MIPS: Unify sc beqz definition

We currently duplicate the definition of __scbeqz in asm/atomic.h &
asm/cmpxchg.h. Move it to asm/llsc.h & rename it to __SC_BEQZ to fit
better with the existing __SC

MIPS: Unify sc beqz definition

We currently duplicate the definition of __scbeqz in asm/atomic.h &
asm/cmpxchg.h. Move it to asm/llsc.h & rename it to __SC_BEQZ to fit
better with the existing __SC macro provided there.

We include a tab in the string in order to avoid the need for users to
indent code any further to include whitespace of their own after the
instruction mnemonic.

Signed-off-by: Paul Burton <[email protected]>
Cc: [email protected]
Cc: Huacai Chen <[email protected]>
Cc: Jiaxun Yang <[email protected]>
Cc: [email protected]

show more ...

mips/atomic: Fix smp_mb__{before,after}_atomic()

Recent probing at the Linux Kernel Memory Model uncovered a
'surprise'. Strongly ordered architectures where the atomic RmW
primitive implies full me

mips/atomic: Fix smp_mb__{before,after}_atomic()

Recent probing at the Linux Kernel Memory Model uncovered a
'surprise'. Strongly ordered architectures where the atomic RmW
primitive implies full memory ordering and
smp_mb__{before,after}_atomic() are a simple barrier() (such as MIPS
without WEAK_REORDERING_BEYOND_LLSC) fail for:

*x = 1;
atomic_inc(u);
smp_mb__after_atomic();
r0 = *y;

Because, while the atomic_inc() implies memory order, it
(surprisingly) does not provide a compiler barrier. This then allows
the compiler to re-order like so:

atomic_inc(u);
*x = 1;
smp_mb__after_atomic();
r0 = *y;

Which the CPU is then allowed to re-order (under TSO rules) like:

atomic_inc(u);
r0 = *y;
*x = 1;

And this very much was not intended. Therefore strengthen the atomic
RmW ops to include a compiler barrier.

Reported-by: Andrea Parri <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Signed-off-by: Paul Burton <[email protected]>

show more ...

mips/atomic: Fix loongson_llsc_mb() wreckage

The comment describing the loongson_llsc_mb() reorder case doesn't
make any sense what so ever. Instruction re-ordering is not an SMP
artifact, but rathe

mips/atomic: Fix loongson_llsc_mb() wreckage

The comment describing the loongson_llsc_mb() reorder case doesn't
make any sense what so ever. Instruction re-ordering is not an SMP
artifact, but rather a CPU local phenomenon. Clarify the comment by
explaining that these issue cause a coherence fail.

For the branch speculation case; if futex_atomic_cmpxchg_inatomic()
needs one at the bne branch target, then surely the normal
__cmpxch_asm() implementation does too. We cannot rely on the
barriers from cmpxchg() because cmpxchg_local() is implemented with
the same macro, and branch prediction and speculation are, too, CPU
local.

Fixes: e02e07e3127d ("MIPS: Loongson: Introduce and use loongson_llsc_mb()")
Cc: Huacai Chen <[email protected]>
Cc: Huang Pei <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Signed-off-by: Paul Burton <[email protected]>

show more ...

locking/atomic, mips: Use s64 for atomic64

As a step towards making the atomic64 API use consistent types treewide,
let's have the mips atomic64 implementation use s64 as the underlying
type for ato

locking/atomic, mips: Use s64 for atomic64

As a step towards making the atomic64 API use consistent types treewide,
let's have the mips atomic64 implementation use s64 as the underlying
type for atomic64_t, rather than long or __s64, matching the generated
headers.

As atomic64_read() depends on the generic defintion of atomic64_t, this
still returns long on 64-bit. This will be converted in a subsequent
patch.

Otherwise, there should be no functional change as a result of this
patch.

Signed-off-by: Mark Rutland <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Paul Burton <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>

show more ...

MIPS: Loongson: Introduce and use loongson_llsc_mb()

On the Loongson-2G/2H/3A/3B there is a hardware flaw that ll/sc and
lld/scd is very weak ordering. We should add sync instructions "before
each l

MIPS: Loongson: Introduce and use loongson_llsc_mb()

On the Loongson-2G/2H/3A/3B there is a hardware flaw that ll/sc and
lld/scd is very weak ordering. We should add sync instructions "before
each ll/lld" and "at the branch-target between ll/sc" to workaround.
Otherwise, this flaw will cause deadlock occasionally (e.g. when doing
heavy load test with LTP).

Below is the explaination of CPU designer:

"For Loongson 3 family, when a memory access instruction (load, store,
or prefetch)'s executing occurs between the execution of LL and SC, the
success or failure of SC is not predictable. Although programmer would
not insert memory access instructions between LL and SC, the memory
instructions before LL in program-order, may dynamically executed
between the execution of LL/SC, so a memory fence (SYNC) is needed
before LL/LLD to avoid this situation.

Since Loongson-3A R2 (3A2000), we have improved our hardware design to
handle this case. But we later deduce a rarely circumstance that some
speculatively executed memory instructions due to branch misprediction
between LL/SC still fall into the above case, so a memory fence (SYNC)
at branch-target (if its target is not between LL/SC) is needed for
Loongson 3A1000, 3B1500, 3A2000 and 3A3000.

Our processor is continually evolving and we aim to to remove all these
workaround-SYNCs around LL/SC for new-come processor."

Here is an example:

Both cpu1 and cpu2 simutaneously run atomic_add by 1 on same atomic var,
this bug cause both 'sc' run by two cpus (in atomic_add) succeed at same
time('sc' return 1), and the variable is only *added by 1*, sometimes,
which is wrong and unacceptable(it should be added by 2).

Why disable fix-loongson3-llsc in compiler?
Because compiler fix will cause problems in kernel's __ex_table section.

This patch fix all the cases in kernel, but:

+. the fix at the end of futex_atomic_cmpxchg_inatomic is for branch-target
of 'bne', there other cases which smp_mb__before_llsc() and smp_llsc_mb() fix
the ll and branch-target coincidently such as atomic_sub_if_positive/
cmpxchg/xchg, just like this one.

+. Loongson 3 does support CONFIG_EDAC_ATOMIC_SCRUB, so no need to touch
edac.h

+. local_ops and cmpxchg_local should not be affected by this bug since
only the owner can write.

+. mips_atomic_set for syscall.c is deprecated and rarely used, just let
it go

Signed-off-by: Huacai Chen <[email protected]>
Signed-off-by: Huang Pei <[email protected]>
[[email protected]:
- Simplify the addition of -mno-fix-loongson3-llsc to cflags, and add
a comment describing why it's there.
- Make loongson_llsc_mb() a no-op when
CONFIG_CPU_LOONGSON3_WORKAROUNDS=n, rather than a compiler memory
barrier.
- Add a comment describing the bug & how loongson_llsc_mb() helps
in asm/barrier.h.]
Signed-off-by: Paul Burton <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: [email protected]
Cc: Steven J . Hill <[email protected]>
Cc: [email protected]
Cc: Fuxin Zhang <[email protected]>
Cc: Zhangjin Wu <[email protected]>
Cc: Li Xuefeng <[email protected]>
Cc: Xu Chenghua <[email protected]>

show more ...

MIPS: Fix a R10000_LLSC_WAR logic in atomic.h

Commit 4936084c2ee2 ("MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h")
introduce a mistake in atomic64_fetch_##op##_relaxed(), because it
forget to del

MIPS: Fix a R10000_LLSC_WAR logic in atomic.h

Commit 4936084c2ee2 ("MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h")
introduce a mistake in atomic64_fetch_##op##_relaxed(), because it
forget to delete R10000_LLSC_WAR in the if-condition. So fix it.

Fixes: 4936084c2ee2 ("MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h")
Signed-off-by: Huacai Chen <[email protected]>
Signed-off-by: Paul Burton <[email protected]>
Cc: Joshua Kinard <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Steven J . Hill <[email protected]>
Cc: Fuxin Zhang <[email protected]>
Cc: Zhangjin Wu <[email protected]>
Cc: [email protected]
Cc: [email protected] # 4.19+

show more ...

MIPS: Avoid using .set mips0 to restore ISA

We currently have 2 commonly used methods for switching ISA within
assembly code, then restoring the original ISA.

1) Using a pair of .set push & .set

MIPS: Avoid using .set mips0 to restore ISA

We currently have 2 commonly used methods for switching ISA within
assembly code, then restoring the original ISA.

1) Using a pair of .set push & .set pop directives. For example:

.set push
.set mips32r2
<some_insn>
.set pop

2) Using .set mips0 to restore the ISA originally specified on the
command line. For example:

.set mips32r2
<some_insn>
.set mips0

Unfortunately method 2 does not work with nanoMIPS toolchains, where the
assembler rejects the .set mips0 directive like so:

Error: cannot change ISA from nanoMIPS to mips0

In preparation for supporting nanoMIPS builds, switch all instances of
method 2 in generic non-platform-specific code to use push & pop as in
method 1 instead. The .set push & .set pop is arguably cleaner anyway,
and if nothing else it's good to consistently use one method.

Signed-off-by: Paul Burton <[email protected]>
Patchwork: https://patchwork.linux-mips.org/patch/21037/
Cc: [email protected]

show more ...

MIPS: Avoid move psuedo-instruction whilst using MIPS_ISA_LEVEL

MIPS_ISA_LEVEL is always defined as the 64 bit ISA that is a compatible
superset of the ISA that the kernel build is targeting, and is

MIPS: Avoid move psuedo-instruction whilst using MIPS_ISA_LEVEL

MIPS_ISA_LEVEL is always defined as the 64 bit ISA that is a compatible
superset of the ISA that the kernel build is targeting, and is used to
allow us to emit instructions that we may detect support for at runtime.

When we use a .set MIPS_ISA_LEVEL directive & are building a 32-bit
kernel, we therefore are temporarily allowing the assembler to generate
MIPS64 instructions. Using the move pseudo-instruction whilst this is
the case is problematic because the assembler is likely to emit a daddu
instruction which will generate a reserved instruction exception when
executed on a MIPS32 machine.

Unfortunately the combination of commit a0a5ac3ce8fe ("MIPS: Fix delay
slot bug in `atomic*_sub_if_positive' for R10000_LLSC_WAR") and commit
4936084c2ee2 ("MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h") causes
us to do exactly this in atomic_sub_if_positive(), and the result is
MIPS64 daddu instructions in 32-bit kernels.

Fix this by using .set mips0 to restore the default ISA after the ll
instruction, and use .set MIPS_ISA_LEVEL again prior to the sc. This
ensures everything but the ll & sc are assembled using the default ISA
for the kernel build & the move pseudo-instruction is emitted as a
MIPS32 addu instruction.

We appear to have another pre-existing instance of the same issue in our
atomic_fetch_*_relaxed() functions, and fix that up too by moving our
.set move0 such that it occurs prior to use of the move
pseudo-instruction.

Signed-off-by: Paul Burton <[email protected]>
Fixes: a0a5ac3ce8fe ("MIPS: Fix delay slot bug in `atomic*_sub_if_positive' for R10000_LLSC_WAR")
Fixes: 4936084c2ee2 ("MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h")
Patchwork: https://patchwork.linux-mips.org/patch/20253/
Cc: James Hogan <[email protected]>
Cc: Joshua Kinard <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: [email protected]

show more ...

MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h

This patch reduces down the conditionals in MIPS atomic code that deal
with a silicon bug in early R10000 cpus that required a workaround of
a branch-

MIPS: Cleanup R10000_LLSC_WAR logic in atomic.h

This patch reduces down the conditionals in MIPS atomic code that deal
with a silicon bug in early R10000 cpus that required a workaround of
a branch-likely instruction following a store-conditional in order to
to guarantee the whole ll/sc sequence is atomic. As the only real
difference is a branch-likely instruction (beqzl) over a standard
branch (beqz), the conditional is reduced down to a single preprocessor
check at the top to pick the required instruction.

This requires writing the uses in assembler, thus we discard the
non-R10000 case that uses a mixture of a C do...while loop with
embedded assembler that was added back in commit 7837314d141c ("MIPS:
Get rid of branches to .subsections."). A note found in the git log
for commit 5999eca25c1f ("[MIPS] Improve branch prediction in ll/sc
atomic operations.") is also addressed.

The macro definition for the branch instruction and the code comment
derives from a patch sent in earlier by Paul Burton for various cmpxchg
cleanups.

[[email protected]:
- Minor whitespace fix for checkpatch.]

Signed-off-by: Joshua Kinard <[email protected]>
Signed-off-by: Paul Burton <[email protected]>
Patchwork: https://patchwork.linux-mips.org/patch/17736/
Cc: Ralf Baechle <[email protected]>
Cc: James Hogan <[email protected]>
Cc: "Maciej W. Rozycki" <[email protected]>
Cc: [email protected]

show more ...

MIPS: Fix delay slot bug in `atomic*_sub_if_positive' for R10000_LLSC_WAR

This patch fixes an old bug in MIPS ll/sc atomics, in the
`atomic_sub_if_positive' and `atomic64_sub_if_positive' functions,

MIPS: Fix delay slot bug in `atomic*_sub_if_positive' for R10000_LLSC_WAR

This patch fixes an old bug in MIPS ll/sc atomics, in the
`atomic_sub_if_positive' and `atomic64_sub_if_positive' functions, for
the R10000_LLSC_WAR case where the result of the subu/dsubu instruction
would potentially not be made available to the sc/scd instruction due
to being in the delay-slot of the branch-likely (beqzl) instruction.

This also removes the need for the `noreorder' directive, allowing GAS
to use delay slot scheduling as needed.

The same fix is also applied to the standard branch (beqz) case in
preparation for a follow-up patch that will cleanup/merge the
R10000_LLSC_WAR and non-R10K sections together.

Signed-off-by: Joshua Kinard <[email protected]>
Signed-off-by: Paul Burton <[email protected]>
Tested-by: Joshua Kinard <[email protected]>
Patchwork: https://patchwork.linux-mips.org/patch/17735/
Cc: Ralf Baechle <[email protected]>
Cc: James Hogan <[email protected]>
Cc: "Maciej W. Rozycki" <[email protected]>
Cc: [email protected]

show more ...

atomics/treewide: Make unconditional inc/dec ops optional

Many of the inc/dec ops are mandatory, but for most architectures inc/dec are
simply trivial wrappers around their corresponding add/sub ops

atomics/treewide: Make unconditional inc/dec ops optional

Many of the inc/dec ops are mandatory, but for most architectures inc/dec are
simply trivial wrappers around their corresponding add/sub ops.

Let's make all the inc/dec ops optional, so that we can get rid of these
boilerplate wrappers.

The instrumented atomics are updated accordingly.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Acked-by: Peter Zijlstra (Intel) <[email protected]>
Acked-by: Palmer Dabbelt <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Link: https://lore.kernel.org/lkml/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>

show more ...

atomics/treewide: Make test ops optional

Some of the atomics return the result of a test applied after the atomic
operation, and almost all architectures implement these as trivial
wrappers around t

atomics/treewide: Make test ops optional

Some of the atomics return the result of a test applied after the atomic
operation, and almost all architectures implement these as trivial
wrappers around the underlying atomic. Specifically:

* <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0)
* <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0)
* <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0)
* <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0)

Rather than have these definitions duplicated in all architectures, with
minor inconsistencies in formatting and documentation, let's make these
operations optional, with default fallbacks as above. Implementations
must now provide a preprocessor symbol.

The instrumented atomics are updated accordingly.

Both x86 and m68k have custom implementations, which are left as-is,
given preprocessor symbols to avoid being overridden.

There should be no functional change as a result of this patch.

Signed-off-by: Mark Rutland <[email protected]>
Reviewed-by: Will Deacon <[email protected]>
Acked-by: Geert Uytterhoeven <[email protected]>
Acked-by: Peter Zijlstra (Intel) <[email protected]>
Acked-by: Palmer Dabbelt <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Link: https://lore.kernel.org/lkml/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>

show more ...