mm: shrinker: avoid memleak in alloc_shrinker_info

A memleak was found as below:

unreferenced object 0xffff8881010d2a80 (size 32):
comm "mkdir", pid 1559, jiffies 4294932666
hex dump (first 32

mm: shrinker: avoid memleak in alloc_shrinker_info

A memleak was found as below:

unreferenced object 0xffff8881010d2a80 (size 32):
comm "mkdir", pid 1559, jiffies 4294932666
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 @...............
backtrace (crc 2e7ef6fa):
[<ffffffff81372754>] __kmalloc_node_noprof+0x394/0x470
[<ffffffff813024ab>] alloc_shrinker_info+0x7b/0x1a0
[<ffffffff813b526a>] mem_cgroup_css_online+0x11a/0x3b0
[<ffffffff81198dd9>] online_css+0x29/0xa0
[<ffffffff811a243d>] cgroup_apply_control_enable+0x20d/0x360
[<ffffffff811a5728>] cgroup_mkdir+0x168/0x5f0
[<ffffffff8148543e>] kernfs_iop_mkdir+0x5e/0x90
[<ffffffff813dbb24>] vfs_mkdir+0x144/0x220
[<ffffffff813e1c97>] do_mkdirat+0x87/0x130
[<ffffffff813e1de9>] __x64_sys_mkdir+0x49/0x70
[<ffffffff81f8c928>] do_syscall_64+0x68/0x140
[<ffffffff8200012f>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

alloc_shrinker_info(), when shrinker_unit_alloc() returns an errer, the
info won't be freed. Just fix it.

Link: https://lkml.kernel.org/r/[email protected]
Fixes: 307bececcd12 ("mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}")
Signed-off-by: Chen Ridong <[email protected]>
Acked-by: Qi Zheng <[email protected]>
Acked-by: Roman Gushchin <[email protected]>
Acked-by: Vlastimil Babka <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
Reviewed-by: Dave Chinner <[email protected]>
Cc: Anshuman Khandual <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Wang Weiyang <[email protected]>
Cc: <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: use kvzalloc_node() from expand_one_shrinker_info()

syzbot is reporting uninit-value at shrinker_alloc(), for commit
307bececcd12 ("mm: shrinker: add a secondary array for
shrinker_inf

mm: shrinker: use kvzalloc_node() from expand_one_shrinker_info()

syzbot is reporting uninit-value at shrinker_alloc(), for commit
307bececcd12 ("mm: shrinker: add a secondary array for
shrinker_info::{map, nr_deferred}") which assumed that the ->unit was
allocated with __GFP_ZERO forgot to replace kvmalloc_node() in
expand_one_shrinker_info() with kvzalloc_node().

Link: https://lkml.kernel.org/r/[email protected]
Reported-by: syzbot <[email protected]>
Closes: https://syzkaller.appspot.com/bug?extid=1e0ed05798af62917464
Fixes: 307bececcd12 ("mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}")
Signed-off-by: Tetsuo Handa <[email protected]>
Acked-by: Qi Zheng <[email protected]>
Cc: Muchun Song <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: convert shrinker_rwsem to mutex

Now there are no readers of shrinker_rwsem, so we can simply replace it
with mutex lock.

[[email protected]: update the fix to alloc_shrinker_i

mm: shrinker: convert shrinker_rwsem to mutex

Now there are no readers of shrinker_rwsem, so we can simply replace it
with mutex lock.

[[email protected]: update the fix to alloc_shrinker_info()]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: hold write lock to reparent shrinker nr_deferred

For now, reparent_shrinker_deferred() is the only holder of read lock of
shrinker_rwsem. And it already holds the global cgroup_mutex,

mm: shrinker: hold write lock to reparent shrinker nr_deferred

For now, reparent_shrinker_deferred() is the only holder of read lock of
shrinker_rwsem. And it already holds the global cgroup_mutex, so it will
not be called in parallel.

Therefore, in order to convert shrinker_rwsem to shrinker_mutex later,
here we change to hold the write lock of shrinker_rwsem to reparent.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: make memcg slab shrink lockless

Like global slab shrink, this commit also uses refcount+RCU method to make
memcg slab shrink lockless.

Use the following script to do slab shrink stres

mm: shrinker: make memcg slab shrink lockless

Like global slab shrink, this commit also uses refcount+RCU method to make
memcg slab shrink lockless.

Use the following script to do slab shrink stress test:

```

DIR="/root/shrinker/memcg/mnt"

do_create()
{
mkdir -p /sys/fs/cgroup/memory/test
echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes
for i in `seq 0 $1`;
do
mkdir -p /sys/fs/cgroup/memory/test/$i;
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
mkdir -p $DIR/$i;
done
}

do_mount()
{
for i in `seq $1 $2`;
do
mount -t tmpfs $i $DIR/$i;
done
}

do_touch()
{
for i in `seq $1 $2`;
do
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 &
done
}

case "$1" in
touch)
do_touch $2 $3
;;
test)
do_create 4000
do_mount 0 4000
do_touch 0 3000
;;
*)
exit 1
;;
esac
```

Save the above script, then run test and touch commands. Then we can use
the following perf command to view hotspots:

perf top -U -F 999

1) Before applying this patchset:

33.15% [kernel] [k] down_read_trylock
25.38% [kernel] [k] shrink_slab
21.75% [kernel] [k] up_read
4.45% [kernel] [k] _find_next_bit
2.27% [kernel] [k] do_shrink_slab
1.80% [kernel] [k] intel_idle_irq
1.79% [kernel] [k] shrink_lruvec
0.67% [kernel] [k] xas_descend
0.41% [kernel] [k] mem_cgroup_iter
0.40% [kernel] [k] shrink_node
0.38% [kernel] [k] list_lru_count_one

2) After applying this patchset:

64.56% [kernel] [k] shrink_slab
12.18% [kernel] [k] do_shrink_slab
3.30% [kernel] [k] __rcu_read_unlock
2.61% [kernel] [k] shrink_lruvec
2.49% [kernel] [k] __rcu_read_lock
1.93% [kernel] [k] intel_idle_irq
0.89% [kernel] [k] shrink_node
0.81% [kernel] [k] mem_cgroup_iter
0.77% [kernel] [k] mem_cgroup_calculate_protection
0.66% [kernel] [k] list_lru_count_one

We can see that the first perf hotspot becomes shrink_slab, which is what
we expect.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: make global slab shrink lockless

The shrinker_rwsem is a global read-write lock in shrinkers subsystem,
which protects most operations such as slab shrink, registration and
unregistrat

mm: shrinker: make global slab shrink lockless

The shrinker_rwsem is a global read-write lock in shrinkers subsystem,
which protects most operations such as slab shrink, registration and
unregistration of shrinkers, etc. This can easily cause problems in the
following cases.

1) When the memory pressure is high and there are many filesystems
mounted or unmounted at the same time, slab shrink will be affected
(down_read_trylock() failed).

Such as the real workload mentioned by Kirill Tkhai:

```
One of the real workloads from my experience is start
of an overcommitted node containing many starting
containers after node crash (or many resuming containers
after reboot for kernel update). In these cases memory
pressure is huge, and the node goes round in long reclaim.
```

2) If a shrinker is blocked (such as the case mentioned
in [1]) and a writer comes in (such as mount a fs),
then this writer will be blocked and cause all
subsequent shrinker-related operations to be blocked.

Even if there is no competitor when shrinking slab, there may still be a
problem. The down_read_trylock() may become a perf hotspot with frequent
calls to shrink_slab(). Because of the poor multicore scalability of
atomic operations, this can lead to a significant drop in IPC
(instructions per cycle).

We used to implement the lockless slab shrink with SRCU [2], but then
kernel test robot reported -88.8% regression in
stress-ng.ramfs.ops_per_sec test case [3], so we reverted it [4].

This commit uses the refcount+RCU method [5] proposed by Dave Chinner
to re-implement the lockless global slab shrink. The memcg slab shrink is
handled in the subsequent patch.

For now, all shrinker instances are converted to dynamically allocated and
will be freed by call_rcu(). So we can use rcu_read_{lock,unlock}() to
ensure that the shrinker instance is valid.

And the shrinker instance will not be run again after unregistration. So
the structure that records the pointer of shrinker instance can be safely
freed without waiting for the RCU read-side critical section.

In this way, while we implement the lockless slab shrink, we don't need to
be blocked in unregister_shrinker().

The following are the test results:

stress-ng --timeout 60 --times --verify --metrics-brief --ramfs 9 &

1) Before applying this patchset:

setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 473062 60.00 8.00 279.13 7884.12 1647.59
for a 60.01s run time:
1440.34s available CPU time
7.99s user time ( 0.55%)
279.13s system time ( 19.38%)
287.12s total time ( 19.93%)
load average: 7.12 2.99 1.15
successful run completed in 60.01s (1 min, 0.01 secs)

2) After applying this patchset:

setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 477165 60.00 8.13 281.34 7952.55 1648.40
for a 60.01s run time:
1440.33s available CPU time
8.12s user time ( 0.56%)
281.34s system time ( 19.53%)
289.46s total time ( 20.10%)
load average: 6.98 3.03 1.19
successful run completed in 60.01s (1 min, 0.01 secs)

We can see that the ops/s has hardly changed.

[1]. https://lore.kernel.org/lkml/[email protected]/
[2]. https://lore.kernel.org/lkml/[email protected]/
[3]. https://lore.kernel.org/lkml/[email protected]/
[4]. https://lore.kernel.org/all/[email protected]/
[5]. https://lore.kernel.org/lkml/[email protected]/

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: rename {prealloc|unregister}_memcg_shrinker() to shrinker_memcg_{alloc|remove}()

With the new shrinker APIs, there is no action such as prealloc, so rename
{prealloc|unregister}_memcg_

mm: shrinker: rename {prealloc|unregister}_memcg_shrinker() to shrinker_memcg_{alloc|remove}()

With the new shrinker APIs, there is no action such as prealloc, so rename
{prealloc|unregister}_memcg_shrinker() to shrinker_memcg_{alloc|remove}(),
which corresponds to the idr_{alloc|remove}() inside the function.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}

Currently, we maintain two linear arrays per node per memcg, which are
shrinker_info::map and shrinker_info::nr_deferred. An

mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred}

Currently, we maintain two linear arrays per node per memcg, which are
shrinker_info::map and shrinker_info::nr_deferred. And we need to resize
them when the shrinker_nr_max is exceeded, that is, allocate a new array,
and then copy the old array to the new array, and finally free the old
array by RCU.

For shrinker_info::map, we do set_bit() under the RCU lock, so we may set
the value into the old map which is about to be freed. This may cause the
value set to be lost. The current solution is not to copy the old map when
resizing, but to set all the corresponding bits in the new map to 1. This
solves the data loss problem, but bring the overhead of more pointless
loops while doing memcg slab shrink.

For shrinker_info::nr_deferred, we will only modify it under the read lock
of shrinker_rwsem, so it will not run concurrently with the resizing. But
after we make memcg slab shrink lockless, there will be the same data loss
problem as shrinker_info::map, and we can't work around it like the map.

For such resizable arrays, the most straightforward idea is to change it
to xarray, like we did for list_lru [1]. We need to do xa_store() in the
list_lru_add()-->set_shrinker_bit(), but this will cause memory
allocation, and the list_lru_add() doesn't accept failure. A possible
solution is to pre-allocate, but the location of pre-allocation is not
well determined (such as deferred_split_shrinker case).

Therefore, this commit chooses to introduce the following secondary array
for shrinker_info::{map, nr_deferred}:

+---------------+--------+--------+-----+
| shrinker_info | unit 0 | unit 1 | ... | (secondary array)
+---------------+--------+--------+-----+
|
v
+---------------+-----+
| nr_deferred[] | map | (leaf array)
+---------------+-----+
(shrinker_info_unit)

The leaf array is never freed unless the memcg is destroyed. The secondary
array will be resized every time the shrinker id exceeds shrinker_nr_max.
So the shrinker_info_unit can be indexed from both the old and the new
shrinker_info->unit[x]. Then even if we get the old secondary array under
the RCU lock, the found map and nr_deferred are also true, so the updated
nr_deferred and map will not be lost.

[1]. https://lore.kernel.org/all/[email protected]/

[[email protected]: unlock the &shrinker_rwsem before the call to free_shrinker_info()]
Link: https://lkml.kernel.org/r/[email protected]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: remove old APIs

Now no users are using the old APIs, just remove them.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <zhen

mm: shrinker: remove old APIs

Now no users are using the old APIs, just remove them.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: "Darrick J. Wong" <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Steven Price <[email protected]>
Cc: "Theodore Ts'o" <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: shrinker: add infrastructure for dynamically allocating shrinker

Patch series "use refcount+RCU method to implement lockless slab shrink",
v6.

1. Background
=============

We used to implement

mm: shrinker: add infrastructure for dynamically allocating shrinker

Patch series "use refcount+RCU method to implement lockless slab shrink",
v6.

1. Background
=============

We used to implement the lockless slab shrink with SRCU [1], but then kernel
test robot reported -88.8% regression in stress-ng.ramfs.ops_per_sec test
case [2], so we reverted it [3].

This patch series aims to re-implement the lockless slab shrink using the
refcount+RCU method proposed by Dave Chinner [4].

[1]. https://lore.kernel.org/lkml/[email protected]/
[2]. https://lore.kernel.org/lkml/[email protected]/
[3]. https://lore.kernel.org/all/[email protected]/
[4]. https://lore.kernel.org/lkml/[email protected]/

2. Implementation
=================

Currently, the shrinker instances can be divided into the following three types:

a) global shrinker instance statically defined in the kernel, such as
workingset_shadow_shrinker.

b) global shrinker instance statically defined in the kernel modules, such as
mmu_shrinker in x86.

c) shrinker instance embedded in other structures.

For case a, the memory of shrinker instance is never freed. For case b, the
memory of shrinker instance will be freed after synchronize_rcu() when the
module is unloaded. For case c, the memory of shrinker instance will be freed
along with the structure it is embedded in.

In preparation for implementing lockless slab shrink, we need to dynamically
allocate those shrinker instances in case c, then the memory can be dynamically
freed alone by calling kfree_rcu().

This patchset adds the following new APIs for dynamically allocating shrinker,
and add a private_data field to struct shrinker to record and get the original
embedded structure.

1. shrinker_alloc()
2. shrinker_register()
3. shrinker_free()

In order to simplify shrinker-related APIs and make shrinker more independent of
other kernel mechanisms, this patchset uses the above APIs to convert all
shrinkers (including case a and b) to dynamically allocated, and then remove all
existing APIs. This will also have another advantage mentioned by Dave Chinner:

```
The other advantage of this is that it will break all the existing out of tree
code and third party modules using the old API and will no longer work with a
kernel using lockless slab shrinkers. They need to break (both at the source and
binary levels) to stop bad things from happening due to using uncoverted
shrinkers in the new setup.
```

Then we free the shrinker by calling call_rcu(), and use rcu_read_{lock,unlock}()
to ensure that the shrinker instance is valid. And the shrinker::refcount
mechanism ensures that the shrinker instance will not be run again after
unregistration. So the structure that records the pointer of shrinker instance
can be safely freed without waiting for the RCU read-side critical section.

In this way, while we implement the lockless slab shrink, we don't need to be
blocked in unregister_shrinker() to wait RCU read-side critical section.

PATCH 1: introduce new APIs
PATCH 2~38: convert all shrinnkers to use new APIs
PATCH 39: remove old APIs
PATCH 40~41: some cleanups and preparations
PATCH 42-43: implement the lockless slab shrink
PATCH 44~45: convert shrinker_rwsem to mutex

3. Testing
==========

3.1 slab shrink stress test
---------------------------

We can reproduce the down_read_trylock() hotspot through the following script:

```

DIR="/root/shrinker/memcg/mnt"

do_create()
{
mkdir -p /sys/fs/cgroup/memory/test
echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes
for i in `seq 0 $1`;
do
mkdir -p /sys/fs/cgroup/memory/test/$i;
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
mkdir -p $DIR/$i;
done
}

do_mount()
{
for i in `seq $1 $2`;
do
mount -t tmpfs $i $DIR/$i;
done
}

do_touch()
{
for i in `seq $1 $2`;
do
echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs;
dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 &
done
}

case "$1" in
touch)
do_touch $2 $3
;;
test)
do_create 4000
do_mount 0 4000
do_touch 0 3000
;;
*)
exit 1
;;
esac
```

Save the above script, then run test and touch commands. Then we can use the
following perf command to view hotspots:

perf top -U -F 999

1) Before applying this patchset:

33.15% [kernel] [k] down_read_trylock
25.38% [kernel] [k] shrink_slab
21.75% [kernel] [k] up_read
4.45% [kernel] [k] _find_next_bit
2.27% [kernel] [k] do_shrink_slab
1.80% [kernel] [k] intel_idle_irq
1.79% [kernel] [k] shrink_lruvec
0.67% [kernel] [k] xas_descend
0.41% [kernel] [k] mem_cgroup_iter
0.40% [kernel] [k] shrink_node
0.38% [kernel] [k] list_lru_count_one

2) After applying this patchset:

64.56% [kernel] [k] shrink_slab
12.18% [kernel] [k] do_shrink_slab
3.30% [kernel] [k] __rcu_read_unlock
2.61% [kernel] [k] shrink_lruvec
2.49% [kernel] [k] __rcu_read_lock
1.93% [kernel] [k] intel_idle_irq
0.89% [kernel] [k] shrink_node
0.81% [kernel] [k] mem_cgroup_iter
0.77% [kernel] [k] mem_cgroup_calculate_protection
0.66% [kernel] [k] list_lru_count_one

We can see that the first perf hotspot becomes shrink_slab, which is what we
expect.

3.2 registration and unregistration stress test
-----------------------------------------------

Run the command below to test:

stress-ng --timeout 60 --times --verify --metrics-brief --ramfs 9 &

1) Before applying this patchset:

setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 473062 60.00 8.00 279.13 7884.12 1647.59
for a 60.01s run time:
1440.34s available CPU time
7.99s user time ( 0.55%)
279.13s system time ( 19.38%)
287.12s total time ( 19.93%)
load average: 7.12 2.99 1.15
successful run completed in 60.01s (1 min, 0.01 secs)

2) After applying this patchset:

setting to a 60 second run per stressor
dispatching hogs: 9 ramfs
stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s
(secs) (secs) (secs) (real time) (usr+sys time)
ramfs 477165 60.00 8.13 281.34 7952.55 1648.40
for a 60.01s run time:
1440.33s available CPU time
8.12s user time ( 0.56%)
281.34s system time ( 19.53%)
289.46s total time ( 20.10%)
load average: 6.98 3.03 1.19
successful run completed in 60.01s (1 min, 0.01 secs)

We can see that the ops/s has hardly changed.


This patch (of 45):

Currently, the shrinker instances can be divided into the following three
types:

a) global shrinker instance statically defined in the kernel, such as
workingset_shadow_shrinker.

b) global shrinker instance statically defined in the kernel modules, such
as mmu_shrinker in x86.

c) shrinker instance embedded in other structures.

For case a, the memory of shrinker instance is never freed. For case b,
the memory of shrinker instance will be freed after synchronize_rcu() when
the module is unloaded. For case c, the memory of shrinker instance will
be freed along with the structure it is embedded in.

In preparation for implementing lockless slab shrink, we need to
dynamically allocate those shrinker instances in case c, then the memory
can be dynamically freed alone by calling kfree_rcu().

So this commit adds the following new APIs for dynamically allocating
shrinker, and add a private_data field to struct shrinker to record and
get the original embedded structure.

1. shrinker_alloc()

Used to allocate shrinker instance itself and related memory, it will
return a pointer to the shrinker instance on success and NULL on failure.

2. shrinker_register()

Used to register the shrinker instance, which is same as the current
register_shrinker_prepared().

3. shrinker_free()

Used to unregister (if needed) and free the shrinker instance.

In order to simplify shrinker-related APIs and make shrinker more
independent of other kernel mechanisms, subsequent submissions will use
the above API to convert all shrinkers (including case a and b) to
dynamically allocated, and then remove all existing APIs.

This will also have another advantage mentioned by Dave Chinner:

```
The other advantage of this is that it will break all the existing
out of tree code and third party modules using the old API and will
no longer work with a kernel using lockless slab shrinkers. They
need to break (both at the source and binary levels) to stop bad
things from happening due to using unconverted shrinkers in the new
setup.
```

[[email protected]: mm: shrinker: some cleanup]
Link: https://lkml.kernel.org/r/[email protected]
Link: https://lkml.kernel.org/r/[email protected]
Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Darrick J. Wong <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Christian Koenig <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joel Fernandes (Google) <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

drm/ttm: introduce pool_shrink_rwsem

Currently, synchronize_shrinkers() is only used by TTM pool. It only
requires that no shrinkers run in parallel.

After we use RCU+refcount method to implement

drm/ttm: introduce pool_shrink_rwsem

Currently, synchronize_shrinkers() is only used by TTM pool. It only
requires that no shrinkers run in parallel.

After we use RCU+refcount method to implement the lockless slab shrink, we
can not use shrinker_rwsem or synchronize_rcu() to guarantee that all
shrinker invocations have seen an update before freeing memory.

So we introduce a new pool_shrink_rwsem to implement a private
ttm_pool_synchronize_shrinkers(), so as to achieve the same purpose.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Reviewed-by: Christian König <[email protected]>
Acked-by: Daniel Vetter <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Darrick J. Wong <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...

mm: vmscan: move shrinker-related code into a separate file

The mm/vmscan.c file is too large, so separate the shrinker-related code
from it into a separate file. No functional changes.

Link: http

mm: vmscan: move shrinker-related code into a separate file

The mm/vmscan.c file is too large, so separate the shrinker-related code
from it into a separate file. No functional changes.

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Qi Zheng <[email protected]>
Reviewed-by: Muchun Song <[email protected]>
Cc: Christian Brauner <[email protected]>
Cc: Christian König <[email protected]>
Cc: Chuck Lever <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Daniel Vetter <[email protected]>
Cc: Darrick J. Wong <[email protected]>
Cc: Dave Chinner <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Joel Fernandes <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: Sergey Senozhatsky <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Theodore Ts'o <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Abhinav Kumar <[email protected]>
Cc: Alasdair Kergon <[email protected]>
Cc: Alexander Viro <[email protected]>
Cc: Alyssa Rosenzweig <[email protected]>
Cc: Andreas Dilger <[email protected]>
Cc: Andreas Gruenbacher <[email protected]>
Cc: Anna Schumaker <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Bob Peterson <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Carlos Llamas <[email protected]>
Cc: Chandan Babu R <[email protected]>
Cc: Chao Yu <[email protected]>
Cc: Chris Mason <[email protected]>
Cc: Coly Li <[email protected]>
Cc: Dai Ngo <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: David Airlie <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: David Sterba <[email protected]>
Cc: Dmitry Baryshkov <[email protected]>
Cc: Gao Xiang <[email protected]>
Cc: Huang Rui <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jaegeuk Kim <[email protected]>
Cc: Jani Nikula <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jason Wang <[email protected]>
Cc: Jeff Layton <[email protected]>
Cc: Jeffle Xu <[email protected]>
Cc: Joonas Lahtinen <[email protected]>
Cc: Josef Bacik <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: Kent Overstreet <[email protected]>
Cc: Marijn Suijten <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Cc: Mike Snitzer <[email protected]>
Cc: Minchan Kim <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Nadav Amit <[email protected]>
Cc: Neil Brown <[email protected]>
Cc: Oleksandr Tyshchenko <[email protected]>
Cc: Olga Kornievskaia <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: Rob Clark <[email protected]>
Cc: Rob Herring <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Sean Paul <[email protected]>
Cc: Song Liu <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Tomeu Vizoso <[email protected]>
Cc: Tom Talpey <[email protected]>
Cc: Trond Myklebust <[email protected]>
Cc: Tvrtko Ursulin <[email protected]>
Cc: Xuan Zhuo <[email protected]>
Cc: Yue Hu <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>

show more ...