Changes in percpu-vm.c

2ccd48ce - percpu: clean up all mappings when pcpu_map_pages() fails

Mon, 11 Mar 2024 19:43:46 +0000

percpu: clean up all mappings when pcpu_map_pages() failsIn pcpu_map_pages(), if __pcpu_map_pages() fails on a CPU, we call__pcpu_unmap_pages() to clean up mappings on all CPUs where mappings werecreated, but not on the CPU where __pcpu_map_pages() fails.__pcpu_map_pages() and __pcpu_unmap_pages() are wrappers aroundvmap_pages_range_noflush() and vunmap_range_noflush(). All other callersof vmap_pages_range_noflush() call vunmap_range_noflush() when mappingfails, except pcpu_map_pages(). The reason could be that partial mappingsmay be left behind from a failed mapping attempt.Call __pcpu_unmap_pages() for the failed CPU as well in pcpu_map_pages().This was found by code inspection, no failures or bugs were observed.Link: https://lkml.kernel.org/r/20240311194346.2291333-1-yosryahmed@google.comSigned-off-by: Yosry Ahmed Acked-by: Dennis Zhou Cc: Christoph Lameter (Ampere) Cc: Tejun Heo Signed-off-by: Andrew Morton List of files: /linux-6.15/mm/percpu-vm.c

93274f1d - percpu: flush tlb in pcpu_reclaim_populated()

Sat, 03 Jul 2021 03:49:57 +0000

percpu: flush tlb in pcpu_reclaim_populated()Prior to "percpu: implement partial chunk depopulation",pcpu_depopulate_chunk() was called only on the destruction path. Thismeant the virtual address range was on its way back to vmalloc whichwill handle flushing the tlbs for us.However, with pcpu_reclaim_populated(), we are now callingpcpu_depopulate_chunk() during the active lifecycle of a chunk.Therefore, we need to flush the tlb as well otherwise we can end upaccessing the wrong page through an invalid tlb mapping as reported in[1].[1] https://lore.kernel.org/lkml/20210702191140.GA3166599@roeck-us.net/Fixes: f183324133ea ("percpu: implement partial chunk depopulation")Reported-and-tested-by: Guenter Roeck Signed-off-by: Dennis Zhou List of files: /linux-6.15/mm/percpu-vm.c

faf65dde - percpu: rework memcg accounting

Thu, 03 Jun 2021 01:09:31 +0000

percpu: rework memcg accountingThe current implementation of the memcg accounting of the percpumemory is based on the idea of having two separate sets of chunks foraccounted and non-accounted memory. This approach has an advantageof not wasting any extra memory for memcg data for non-accountedchunks, however it complicates the code and leads to a higher chunksnumber due to a lower chunk utilization.Instead of having two chunk types it's possible to declare all* chunksmemcg-aware unless the kernel memory accounting is disabled globallyby a boot option. The size of objcg_array is usually small incomparison to chunks themselves (it obviously depends on the number ofCPUs), so even if some chunk will have no accounted allocations, thememory waste isn't significant and will likely be compensated bya higher chunk utilization. Also, with time more and more percpuallocations will likely become accounted.* The first chunk is initialized before the memory cgroup subsystem, so we don't know for sure whether we need to allocate obj_cgroups. Because it's small, let's make it free for use. Then we don't need to allocate obj_cgroups for it.Signed-off-by: Roman Gushchin Signed-off-by: Dennis Zhou List of files: /linux-6.15/mm/percpu-vm.c

f1833241 - percpu: implement partial chunk depopulation

Thu, 08 Apr 2021 03:57:36 +0000

percpu: implement partial chunk depopulationFrom Roman ("percpu: partial chunk depopulation"):In our [Facebook] production experience the percpu memory allocator issometimes struggling with returning the memory to the system. A typicalexample is a creation of several thousands memory cgroups (each hasseveral chunks of the percpu data used for vmstats, vmevents,ref counters etc). Deletion and complete releasing of these cgroupsdoesn't always lead to a shrinkage of the percpu memory, so thatsometimes there are several GB's of memory wasted.The underlying problem is the fragmentation: to release an underlyingchunk all percpu allocations should be released first. The percpuallocator tends to top up chunks to improve the utilization. It meansnew small-ish allocations (e.g. percpu ref counters) are placed ontoalmost filled old-ish chunks, effectively pinning them in memory.This patchset solves this problem by implementing a partial depopulationof percpu chunks: chunks with many empty pages are being asynchronouslydepopulated and the pages are returned to the system.To illustrate the problem the following script can be used:--cd /sys/fs/cgroupmkdir percpu_testecho "+memory" > percpu_test/cgroup.subtree_controlcat /proc/meminfo | grep Percpufor i in `seq 1 1000`; do mkdir percpu_test/cg_"${i}" for j in `seq 1 10`; do mkdir percpu_test/cg_"${i}"_"${j}" donedonecat /proc/meminfo | grep Percpufor i in `seq 1 1000`; do for j in `seq 1 10`; do rmdir percpu_test/cg_"${i}"_"${j}" donedonesleep 10cat /proc/meminfo | grep Percpufor i in `seq 1 1000`; do rmdir percpu_test/cg_"${i}"donermdir percpu_test--It creates 11000 memory cgroups and removes every 10 out of 11.It prints the initial size of the percpu memory, the size aftercreating all cgroups and the size after deleting most of them.Results: vanilla: ./percpu_test.sh Percpu: 7488 kB Percpu: 481152 kB Percpu: 481152 kB with this patchset applied: ./percpu_test.sh Percpu: 7488 kB Percpu: 481408 kB Percpu: 135552 kBThe total size of the percpu memory was reduced by more than 3.5 times.This patch:This patch implements partial depopulation of percpu chunks.As of now, a chunk can be depopulated only as a part of the finaldestruction, if there are no more outstanding allocations. Howeverto minimize a memory waste it might be useful to depopulate apartially filed chunk, if a small number of outstanding allocationsprevents the chunk from being fully reclaimed.This patch implements the following depopulation process: it scansover the chunk pages, looks for a range of empty and populated pagesand performs the depopulation. To avoid races with new allocations,the chunk is previously isolated. After the depopulation the chunk issidelined to a special list or freed. New allocations prefer usingactive chunks to sidelined chunks. If a sidelined chunk is used, it isreintegrated to the active lists.The depopulation is scheduled on the free path if the chunk is all ofthe following: 1) has more than 1/4 of total pages free and populated 2) the system has enough free percpu pages aside of this chunk 3) isn't the reserved chunk 4) isn't the first chunkIf it's already depopulated but got free populated pages, it's a goodtarget too. The chunk is moved to a special slot,pcpu_to_depopulate_slot, chunk->isolated is set, and the balance workitem is scheduled. On isolation, these pages are removed from thepcpu_nr_empty_pop_pages. It is constantly replaced to theto_depopulate_slot when it meets these qualifications.pcpu_reclaim_populated() iterates over the to_depopulate_slot until itbecomes empty. The depopulation is performed in the reverse direction tokeep populated pages close to the beginning. Depopulated chunks aresidelined to preferentially avoid them for new allocations. When noactive chunk can suffice a new allocation, sidelined chunks are firstchecked before creating a new chunk.Signed-off-by: Roman Gushchin Co-developed-by: Dennis Zhou Signed-off-by: Dennis Zhou Tested-by: Pratik Sampat Signed-off-by: Dennis Zhou List of files: /linux-6.15/mm/percpu-vm.c

4ad0ae8c - mm/vmalloc: remove unmap_kernel_range

Fri, 30 Apr 2021 05:59:01 +0000

mm/vmalloc: remove unmap_kernel_rangeThis is a shim around vunmap_range, get rid of it.Move the main API comment from the _noflush variant to the normalvariant, and make _noflush internal to mm/.[npiggin@gmail.com: fix nommu builds and a comment bug per sfr] Link: https://lkml.kernel.org/r/1617292598.m6g0knx24s.astroid@bobo.none[akpm@linux-foundation.org: move vunmap_range_noflush() stub inside !CONFIG_MMU, not !CONFIG_NUMA][npiggin@gmail.com: fix nommu builds] Link: https://lkml.kernel.org/r/1617292497.o1uhq5ipxp.astroid@bobo.noneLink: https://lkml.kernel.org/r/20210322021806.892164-5-npiggin@gmail.comSigned-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig Cc: Cédric Le Goater Cc: Uladzislau Rezki Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds List of files: /linux-6.15/mm/percpu-vm.c

b67177ec - mm/vmalloc: remove map_kernel_range

Fri, 30 Apr 2021 05:58:53 +0000

mm/vmalloc: remove map_kernel_rangePatch series "mm/vmalloc: cleanup after hugepage series", v2.Christoph pointed out some overdue cleanups required after the hugevmalloc series, and I had another failure error message improvement aswell.This patch (of 5):This is a shim around vmap_pages_range, get rid of it.Move the main API comment from the _noflush variant to the normal variant,and make _noflush internal to mm/.Link: https://lkml.kernel.org/r/20210322021806.892164-1-npiggin@gmail.comLink: https://lkml.kernel.org/r/20210322021806.892164-2-npiggin@gmail.comSigned-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig Cc: Uladzislau Rezki Cc: Cédric Le Goater Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds List of files: /linux-6.15/mm/percpu-vm.c

3c7be18a - mm: memcg/percpu: account percpu memory to memory cgroups

Wed, 12 Aug 2020 01:30:17 +0000

mm: memcg/percpu: account percpu memory to memory cgroupsPercpu memory is becoming more and more widely used by various subsystems,and the total amount of memory controlled by the percpu allocator can makea good part of the total memory.As an example, bpf maps can consume a lot of percpu memory, and they arecreated by a user. Also, some cgroup internals (e.g. memory controllerstatistics) can be quite large. On a machine with many CPUs and bignumber of cgroups they can consume hundreds of megabytes.So the lack of memcg accounting is creating a breach in the memoryisolation. Similar to the slab memory, percpu memory should be accountedby default.To implement the perpcu accounting it's possible to take the slab memoryaccounting as a model to follow. Let's introduce two types of percpuchunks: root and memcg. What makes memcg chunks different is anadditional space allocated to store memcg membership information. If__GFP_ACCOUNT is passed on allocation, a memcg chunk should be be used.If it's possible to charge the corresponding size to the target memorycgroup, allocation is performed, and the memcg ownership data is recorded.System-wide allocations are performed using root chunks, so there is noadditional memory overhead.To implement a fast reparenting of percpu memory on memcg removal, wedon't store mem_cgroup pointers directly: instead we use obj_cgroup API,introduced for slab accounting.[akpm@linux-foundation.org: fix CONFIG_MEMCG_KMEM=n build errors and warning][akpm@linux-foundation.org: move unreachable code, per Roman][cuibixuan@huawei.com: mm/percpu: fix 'defined but not used' warning] Link: http://lkml.kernel.org/r/6d41b939-a741-b521-a7a2-e7296ec16219@huawei.comSigned-off-by: Roman Gushchin Signed-off-by: Bixuan Cui Signed-off-by: Andrew Morton Reviewed-by: Shakeel Butt Acked-by: Dennis Zhou Cc: Christoph Lameter Cc: David Rientjes Cc: Johannes Weiner Cc: Joonsoo Kim Cc: Mel Gorman Cc: Michal Hocko Cc: Pekka Enberg Cc: Tejun Heo Cc: Tobin C. Harding Cc: Vlastimil Babka Cc: Waiman Long Cc: Bixuan Cui Cc: Michal Koutný Cc: Stephen Rothwell Link: http://lkml.kernel.org/r/20200623184515.4132564-3-guro@fb.comSigned-off-by: Linus Torvalds List of files: /linux-6.15/mm/percpu-vm.c

55716d26 - treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428

Sat, 01 Jun 2019 08:08:42 +0000

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428Based on 1 normalized pattern(s): this file is released under the gplv2extracted by the scancode license scanner the SPDX license identifier GPL-2.0-onlyhas been chosen to replace the boilerplate/reference in 68 file(s).Signed-off-by: Thomas Gleixner Reviewed-by: Armijn Hemel Reviewed-by: Allison Randal Cc: linux-spdx@vger.kernel.orgLink: https://lkml.kernel.org/r/20190531190114.292346262@linutronix.deSigned-off-by: Greg Kroah-Hartman List of files: /linux-6.15/mm/percpu-vm.c

554fef1c - percpu: allow select gfp to be passed to underlying allocators

Fri, 16 Feb 2018 18:09:58 +0000

percpu: allow select gfp to be passed to underlying allocatorsThe prior patch added support for passing gfp flags through to theunderlying allocators. This patch allows users to pass along gfp flags(currently only __GFP_NORETRY and __GFP_NOWARN) to the underlyingallocators. This should allow users to decide if they are ok withfailing allocations recovering in a more graceful way.Additionally, gfp passing was done as additional flags in the previouspatch. Instead, change this to caller passed semantics. GFP_KERNEL isalso removed as the default flag. It continues to be used for internallycaused underlying percpu allocations.V2:Removed gfp_percpu_mask in favor of doing it inline.Removed GFP_KERNEL as a default flag for __alloc_percpu_gfp.Signed-off-by: Dennis Zhou Suggested-by: Daniel Borkmann Acked-by: Christoph Lameter Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

47504ee0 - percpu: add __GFP_NORETRY semantics to the percpu balancing path

Fri, 16 Feb 2018 18:07:19 +0000

percpu: add __GFP_NORETRY semantics to the percpu balancing pathPercpu memory using the vmalloc area based chunk allocator lazilypopulates chunks by first requesting the full virtual address spacerequired for the chunk and subsequently adding pages as allocations comethrough. To ensure atomic allocations can succeed, a workqueue item isused to maintain a minimum number of empty pages. In certain scenarios,such as reported in [1], it is possible that physical memory becomesquite scarce which can result in either a rather long time spent tryingto find free pages or worse, a kernel panic.This patch adds support for __GFP_NORETRY and __GFP_NOWARN passing themthrough to the underlying allocators. This should prevent anyunnecessary panics potentially caused by the workqueue item. The passingof gfp around is as additional flags rather than a full set of flags.The next patch will change these to caller passed semantics.V2:Added const modifier to gfp flags in the balance path.Removed an extra whitespace.[1] https://lkml.org/lkml/2018/2/12/551Signed-off-by: Dennis Zhou Suggested-by: Daniel Borkmann Reported-by: syzbot+adb03f3f0bb57ce3acda@syzkaller.appspotmail.comAcked-by: Christoph Lameter Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

453f85d4 - mm: remove __GFP_COLD

Thu, 16 Nov 2017 01:38:03 +0000

mm: remove __GFP_COLDAs the page free path makes no distinction between cache hot and coldpages, there is no real useful ordering of pages in the free list thatallocation requests can take advantage of. Juding from the users of__GFP_COLD, it is likely that a number of them are the result of copyingother sites instead of actually measuring the impact. Remove the__GFP_COLD parameter which simplifies a number of paths in the pageallocator.This is potentially controversial but bear in mind that the size of theper-cpu pagelists versus modern cache sizes means that the whole per-cpulist can often fit in the L3 cache. Hence, there is only a potentialbenefit for microbenchmarks that alloc/free pages in a tight loop. It'seven worse when THP is taken into account which has little or no chanceof getting a cache-hot page as the per-cpu list is bypassed and thezeroing of multiple pages will thrash the cache anyway.The truncate microbenchmarks are not shown as this patch affects theallocation path and not the free path. A page fault microbenchmark wastested but it showed no sigificant difference which is not surprisinggiven that the __GFP_COLD branches are a miniscule percentage of thefault path.Link: http://lkml.kernel.org/r/20171018075952.10627-9-mgorman@techsingularity.netSigned-off-by: Mel Gorman Acked-by: Vlastimil Babka Cc: Andi Kleen Cc: Dave Chinner Cc: Dave Hansen Cc: Jan Kara Cc: Johannes Weiner Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds List of files: /linux-6.15/mm/percpu-vm.c

e3efe3db - percpu: fix static checker warnings in pcpu_destroy_chunk

Thu, 29 Jun 2017 14:56:26 +0000

percpu: fix static checker warnings in pcpu_destroy_chunkFrom 5021b97f4026334d2c8dfad80797dd1028cddd73 Mon Sep 17 00:00:00 2001From: Dennis Zhou Date: Thu, 29 Jun 2017 07:11:41 -0700Add NULL check in pcpu_destroy_chunk to correct static checker warnings.Signed-off-by: Dennis Zhou Reported-by: Dan Carpenter Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

df95e795 - percpu: add tracepoint support for percpu memory

Mon, 19 Jun 2017 23:28:32 +0000

percpu: add tracepoint support for percpu memoryAdd support for tracepoints to the following events: chunk allocation,chunk free, area allocation, area free, and area allocation failure.This should let us replay percpu memory requests and evaluatecorresponding decisions.Signed-off-by: Dennis Zhou Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

30a5b536 - percpu: expose statistics about percpu memory via debugfs

Mon, 19 Jun 2017 23:28:31 +0000

percpu: expose statistics about percpu memory via debugfsThere is limited visibility into the use of percpu memory leaving usunable to reason about correctness of parameters and overall use ofpercpu memory. These counters and statistics aim to help understandbasic statistics about percpu memory such as number of allocations overthe lifetime, allocation sizes, and fragmentation.New Config: PERCPU_STATSSigned-off-by: Dennis Zhou Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

8a1df543 - percpu: remove unused chunk_alloc parameter from pcpu_get_pages()

Sat, 25 Feb 2017 20:59:26 +0000

percpu: remove unused chunk_alloc parameter from pcpu_get_pages()pcpu_get_pages() doesn't use chunk_alloc parameter, remove it.Fixes: fbbb7f4e149f ("percpu: remove the usage of separate populated bitmap in percpu-vm")Signed-off-by: Tahsin Erdogan Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

a93ace48 - percpu: move region iterations out of pcpu_[de]populate_chunk()

Tue, 02 Sep 2014 18:46:02 +0000

percpu: move region iterations out of pcpu_[de]populate_chunk()Previously, pcpu_[de]populate_chunk() were called with the range whichmay contain multiple target regions in it andpcpu_[de]populate_chunk() iterated over the regions. This has thebenefit of batching up cache flushes for all the regions; however,we're planning to add more bookkeeping logic around [de]population tosupport atomic allocations and this delegation of iterations gets inthe way.This patch moves the region iterations out ofpcpu_[de]populate_chunk() into its callers - pcpu_alloc() andpcpu_reclaim() - so that we can later add logic to track more statesaround them. This change may make cache and tlb flushes more frequentbut multi-region [de]populations are rare anyway and if this actuallybecomes a problem, it's not difficult to factor out cache flushes asseparate callbacks which are directly invoked from percpu.c.Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

dca49645 - percpu: move common parts out of pcpu_[de]populate_chunk()

Tue, 02 Sep 2014 18:46:01 +0000

percpu: move common parts out of pcpu_[de]populate_chunk()percpu-vm and percpu-km implement separate versions ofpcpu_[de]populate_chunk() and some part which is or should be commonare currently in the specific implementations. Make the followingchanges.* Allocate area clearing is moved from the pcpu_populate_chunk() implementations to pcpu_alloc(). This makes percpu-km's version noop.* Quick exit tests in pcpu_[de]populate_chunk() of percpu-vm are moved to their respective callers so that they are applied to percpu-km too. This doesn't make any meaningful difference as both functions are noop for percpu-km; however, this is more consistent and will help implementing atomic allocation support.Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

cdb4cba5 - percpu: remove @may_alloc from pcpu_get_pages()

Tue, 02 Sep 2014 18:46:01 +0000

percpu: remove @may_alloc from pcpu_get_pages()pcpu_get_pages() creates the temp pages array if not already allocatedand returns the pointer to it. As the function is called from both[de]population paths and depopulation can only happen after at leastone successful population, the param doesn't make any difference - theallocation will always happen on the population path anyway.Remove @may_alloc from pcpu_get_pages(). Also, add an lockdepassertion pcpu_alloc_mutex instead of vaguely stating that theexclusion is the caller's responsibility.Signed-off-by: Tejun Heo List of files: /linux-6.15/mm/percpu-vm.c

fbbb7f4e - percpu: remove the usage of separate populated bitmap in percpu-vm

Tue, 02 Sep 2014 18:46:01 +0000

percpu: remove the usage of separate populated bitmap in percpu-vmpercpu-vm uses pcpu_get_pages_and_bitmap() to acquire temp pages arrayand populated bitmap and uses the two during [de]population. The tempbitmap is used only to build the new bitmap that is copied tochunk->populated after the operation succeeds; however, the new bitmapcan be trivially set after success without using the temp bitmap.This patch removes the temp populated bitmap usage from percpu-vm.c.* pcpu_get_pages_and_bitmap() is renamed to pcpu_get_pages() and no longer hands out the temp bitmap.* @populated arugment is dropped from all the related functions. @populated updates in pcpu_[un]map_pages() are dropped.* Two loops in pcpu_map_pages() are merged.* pcpu_[de]populated_chunk() modify chunk->populated bitmap directly from @page_start and @page_end after success.Signed-off-by: Tejun Heo Acked-by: Christoph Lameter List of files: /linux-6.15/mm/percpu-vm.c

849f5169 - percpu: perform tlb flush after pcpu_map_pages() failure

Fri, 15 Aug 2014 20:06:10 +0000

percpu: perform tlb flush after pcpu_map_pages() failureIf pcpu_map_pages() fails midway, it unmaps the already mapped pages.Currently, it doesn't flush tlb after the partial unmapping. This maybe okay in most cases as the established mapping hasn't been used atthat point but it can go wrong and when it goes wrong it'd beextremely difficult to track down.Flush tlb after the partial unmapping.Signed-off-by: Tejun Heo Cc: stable@vger.kernel.org List of files: /linux-6.15/mm/percpu-vm.c