Changes in task_mmu.c

6dd55dd1 - fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)

Mon, 03 Mar 2025 16:30:12 +0000

fs/proc/task_mmu: remove per-page mapcount dependency for smaps/smaps_rollup (CONFIG_NO_PAGE_MAPCOUNT)Let's implement an alternative when per-page mapcounts in large folios areno longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.When computing the output for smaps / smaps_rollups, in particular whencalculating the USS (Unique Set Size) and the PSS (Proportional Set Size),we still rely on per-page mapcounts.To determine private vs. shared, we'll use folio_likely_mapped_shared(),similar to how we handle PM_MMAP_EXCLUSIVE. Similarly, we might nowunder-estimate the USS and count pages towards "shared" that are actually"private" ("exclusively mapped").When calculating the PSS, we'll now also use the average per-page mapcountfor large folios: this can result in both, an over-estimation and anunder-estimation of the PSS. The difference is not expected to mattermuch in practice, but we'll have to learn as we go.We can now provide folio_precise_page_mapcount() only withCONFIG_PAGE_MAPCOUNT, and remove one of the last users of per-pagemapcounts when CONFIG_NO_PAGE_MAPCOUNT is enabled.Document the new behavior.Link: https://lkml.kernel.org/r/20250303163014.1128035-20-david@redhat.comSigned-off-by: David Hildenbrand Cc: Andy Lutomirks^H^Hski Cc: Borislav Betkov Cc: Dave Hansen Cc: Ingo Molnar Cc: Jann Horn Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Kirill A. Shutemov Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcow (Oracle) Cc: Michal Koutn Cc: Muchun Song Cc: tejun heo Cc: Thomas Gleixner Cc: Vlastimil Babka Cc: Zefan Li Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

7a34ae14 - fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT)

Mon, 03 Mar 2025 16:30:11 +0000

fs/proc/task_mmu: remove per-page mapcount dependency for "mapmax" (CONFIG_NO_PAGE_MAPCOUNT)Let's implement an alternative when per-page mapcounts in large folios areno longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.For calculating "mapmax", we now use the average per-page mapcount in alarge folio instead of the per-page mapcount.For hugetlb folios and folios that are not partially mapped into MMs,there is no change.Likely, this change will not matter much in practice, and an alternativemight be to simple remove this stat with CONFIG_NO_PAGE_MAPCOUNT. However, there might be value to it, so let's keep it like that anddocument the behavior.Link: https://lkml.kernel.org/r/20250303163014.1128035-19-david@redhat.comSigned-off-by: David Hildenbrand Cc: Andy Lutomirks^H^Hski Cc: Borislav Betkov Cc: Dave Hansen Cc: Ingo Molnar Cc: Jann Horn Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Kirill A. Shutemov Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcow (Oracle) Cc: Michal Koutn Cc: Muchun Song Cc: tejun heo Cc: Thomas Gleixner Cc: Vlastimil Babka Cc: Zefan Li Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

eb168769 - fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT)

Mon, 03 Mar 2025 16:30:10 +0000

fs/proc/task_mmu: remove per-page mapcount dependency for PM_MMAP_EXCLUSIVE (CONFIG_NO_PAGE_MAPCOUNT)Let's implement an alternative when per-page mapcounts in large folios areno longer maintained -- soon with CONFIG_NO_PAGE_MAPCOUNT.PM_MMAP_EXCLUSIVE will now be set if folio_likely_mapped_shared() is true-- when the folio is considered "mapped shared", including when it oncewas "mapped shared" but no longer is, as documented.This might result in and under-indication of "exclusively mapped", whichis considered better than over-indicating it: under-estimating the USS(Unique Set Size) is better than over-estimating it.As an alternative, we could simply remove that flag withCONFIG_NO_PAGE_MAPCOUNT completely, but there might be value to it. So,let's keep it like that and document the behavior.Link: https://lkml.kernel.org/r/20250303163014.1128035-18-david@redhat.comSigned-off-by: David Hildenbrand Cc: Andy Lutomirks^H^Hski Cc: Borislav Betkov Cc: Dave Hansen Cc: Ingo Molnar Cc: Jann Horn Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Kirill A. Shutemov Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcow (Oracle) Cc: Michal Koutn Cc: Muchun Song Cc: tejun heo Cc: Thomas Gleixner Cc: Vlastimil Babka Cc: Zefan Li Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

003fde44 - mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared()

Mon, 03 Mar 2025 16:30:07 +0000

mm: convert folio_likely_mapped_shared() to folio_maybe_mapped_shared()Let's reuse our new MM ownership tracking infrastructure for large foliosto make folio_likely_mapped_shared() never return false negatives -- neverindicating "not mapped shared" although the folio *is* mapped shared. With that, we can rename it to folio_maybe_mapped_shared() and get rid ofthe dependency on the mapcount of the first folio page.The semantics are now arguably clearer: no mixture of "false negatives"and "false positives", only the remaining possibility for "falsepositives".Thoroughly document the new semantics. We might now detect that a largefolio is "maybe mapped shared" although it *no longer* is -- but once was.Now, if more than two MMs mapped a folio at the same time, and the MMmapping the folio exclusively at the end is not one tracked in the twofolio MM slots, we will detect the folio as "maybe mapped shared".For anonymous folios, usually (except weird corner cases) all PTEs thattarget a "maybe mapped shared" folio are R/O. As soon as a child processwould write to them (iow, actively use them), we would CoW and effectivelyreplace these PTEs. Most cases (below) are not expected to really matterwith large anonymous folios for this reason.Most importantly, there will be no change at all for:* small folios* hugetlb folios* PMD-mapped PMD-sized THPs (single mapping)This change has the potential to affect existing callers offolio_likely_mapped_shared() -> folio_maybe_mapped_shared():(1) fs/proc/task_mmu.c: no change (hugetlb)(2) khugepaged counts PTEs that target shared folios towards max_ptes_shared (default: HPAGE_PMD_NR / 2), meaning we could skip a collapse where we would have previously collapsed. This only applies to anonymous folios and is not expected to matter in practice. Worth noting that this change sorts out case (A) documented in commit 1bafe96e89f0 ("mm/khugepaged: replace page_mapcount() check by folio_likely_mapped_shared()") by removing the possibility for "false negatives".(3) MADV_COLD / MADV_PAGEOUT / MADV_FREE will not try splitting PTE-mapped THPs that are considered shared but not fully covered by the requested range, consequently not processing them. PMD-mapped PMD-sized THP are not affected, or when all PTEs are covered. These functions are usually only called on anon/file folios that are exclusively mapped most of the time (no other file mappings or no fork()), so the "false negatives" are not expected to matter in practice.(4) mbind() / migrate_pages() / move_pages() will refuse to migrate shared folios unless MPOL_MF_MOVE_ALL is effective (requires CAP_SYS_NICE). We will now reject some folios that could be migrated. Similar to (3), especially with MPOL_MF_MOVE_ALL, so this is not expected to matter in practice. Note that cpuset_migrate_mm_workfn() calls do_migrate_pages() with MPOL_MF_MOVE_ALL.(5) NUMA hinting mm/migrate.c:migrate_misplaced_folio_prepare() will skip file folios that are probably shared libraries (-> "mapped shared" and executable). This check would have detected it as a shared library at some point (at least 3 MMs mapping it), so detecting it afterwards does not sound wrong (still a shared library). Not expected to matter. mm/memory.c:numa_migrate_check() will indicate TNF_SHARED in MAP_SHARED file mappings when encountering a shared folio. Similar reasoning, not expected to matter. mm/mprotect.c:change_pte_range() will skip folios detected as shared in CoW mappings. Similarly, this is not expected to matter in practice, but if it would ever be a problem we could relax that check a bit (e.g., basing it on the average page-mapcount in a folio), because it was only an optimization when many (e.g., 288) processes were mapping the same folios -- see commit 859d4adc3415 ("mm: numa: do not trap faults on shared data section pages.")(6) mm/rmap.c:folio_referenced_one() will skip exclusive swapbacked folios in dying processes. Applies to anonymous folios only. Without "false negatives", we'll now skip all actually shared ones. Skipping ones that are actually exclusive won't really matter, it's a pure optimization, and is not expected to matter in practice.In theory, one can detect the problematic scenario: folio_mapcount() > 0and no folio MM slot is occupied ("state unknown"). One could reset theMM slots while doing an rmap walk, which migration / folio split alreadydo when setting everything up. Further, when batching PTEs we mightnaturally learn about a owner (e.g., folio_mapcount() == nr_ptes) andcould update the owner. However, we'll defer that until the scenarioswhere it would really matter are clear.Link: https://lkml.kernel.org/r/20250303163014.1128035-15-david@redhat.comSigned-off-by: David Hildenbrand Cc: Andy Lutomirks^H^Hski Cc: Borislav Betkov Cc: Dave Hansen Cc: Ingo Molnar Cc: Jann Horn Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Kirill A. Shutemov Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcow (Oracle) Cc: Michal Koutn Cc: Muchun Song Cc: tejun heo Cc: Thomas Gleixner Cc: Vlastimil Babka Cc: Zefan Li Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

ad449d85 - fs/proc/task_mmu: reduce scope of lazy mmu region

Mon, 03 Mar 2025 14:15:36 +0000

fs/proc/task_mmu: reduce scope of lazy mmu regionUpdate the way arch_[enter|leave]_lazy_mmu_mode() is called inpagemap_scan_pmd_entry() to follow the normal pattern of holding the ptlfor user space mappings. As a result the scope is reduced to only the ptetable, but that's where most of the performance win is.While I believe there wasn't technically a bug here, the original scopemade it easier to accidentally nest or, worse, accidentally call somethinglike kmap() which would expect an immediate mode pte modification but itwould end up deferred.Link: https://lkml.kernel.org/r/20250303141542.3371656-3-ryan.roberts@arm.comSigned-off-by: Ryan Roberts Acked-by: David Hildenbrand Acked-by: Juergen Gross Cc: Andreas Larsson Cc: Borislav Betkov Cc: Boris Ostrovsky Cc: Catalin Marinas Cc: Dave Hansen Cc: David S. Miller Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Juegren Gross Cc: Matthew Wilcow (Oracle) Cc: Thomas Gleinxer Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

8e2f2aeb - fs/proc/task_mmu: add guard region bit to pagemap

Fri, 21 Feb 2025 12:05:22 +0000

fs/proc/task_mmu: add guard region bit to pagemapPatch series "fs/proc/task_mmu: add guard region bit to pagemap".Currently there is no means of determining whether a given page in amapping range is designated a guard region (as installed via madvise()using the MADV_GUARD_INSTALL flag).This is generally not an issue, but in some instances users may wish todetermine whether this is the case.This series adds this ability via /proc/$pid/pagemap, updates thedocumentation and adds a self test to assert that this functionscorrectly.This patch (of 2):Currently there is no means by which users can determine whether a givenpage in memory is in fact a guard region, that is having had theMADV_GUARD_INSTALL madvise() flag applied to it.This is intentional, as to provide this information in VMA metadata wouldcontradict the intent of the feature (providing a means to change faultbehaviour at a page table level rather than a VMA level), and wouldrequire VMA metadata operations to scan page tables, which isunacceptable.In many cases, users have no need to reflect and determine what regionshave been designated guard regions, as it is the user who has establishedthem in the first place.But in some instances, such as monitoring software, or software thatrelies upon being able to ascertain the nature of mappings within a remoteprocess for instance, it becomes useful to be able to determine whichpages have the guard region marker applied.This patch makes use of an unused pagemap bit (58) to provide thisinformation.This patch updates the documentation at the same time as making the changesuch that the implementation of the feature and the documentation of itare tied together.Link: https://lkml.kernel.org/r/cover.1740139449.git.lorenzo.stoakes@oracle.comLink: https://lkml.kernel.org/r/521d99c08b975fb06a1e7201e971cc24d68196d1.1740139449.git.lorenzo.stoakes@oracle.comSigned-off-by: Lorenzo Stoakes Acked-by: David Hildenbrand Cc: Jann Horn Cc: Jonathan Corbet Cc: Kalesh Singh Cc: Liam Howlett Cc: Matthew Wilcow (Oracle) Cc: "Paul E . McKenney" Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

3754137d - fs/proc/task_mmu: fix pagemap flags with PMD THP entries on 32bit

Tue, 17 Dec 2024 19:50:00 +0000

fs/proc/task_mmu: fix pagemap flags with PMD THP entries on 32bitEntries (including flags) are u64, even on 32bit. So right now we arecutting of the flags on 32bit. This way, for example the cow selftestcomplains about: # ./cow ... Bail Out! read and ioctl return unmatched results for populated: 0 1Link: https://lkml.kernel.org/r/20241217195000.1734039-1-david@redhat.comFixes: 2c1f057e5be6 ("fs/proc/task_mmu: properly detect PM_MMAP_EXCLUSIVE per page of PMD-mapped THPs")Signed-off-by: David Hildenbrand Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

669b0cb8 - fs/proc/task_mmu: prevent integer overflow in pagemap_scan_get_args()

Thu, 14 Nov 2024 08:59:32 +0000

fs/proc/task_mmu: prevent integer overflow in pagemap_scan_get_args()The "arg->vec_len" variable is a u64 that comes from the user at the startof the function. The "arg->vec_len * sizeof(struct page_region))"multiplication can lead to integer wrapping. Use size_mul() to avoidthat.Also the size_add/mul() functions work on unsigned long so for 32bitsystems we need to ensure that "arg->vec_len" fits in an unsigned long.Link: https://lkml.kernel.org/r/39d41335-dd4d-48ed-8a7f-402c57d8ea84@stanley.mountainFixes: 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")Signed-off-by: Dan Carpenter Cc: Andrei Vagin Cc: Andrii Nakryiko Cc: Arnd Bergmann Cc: David Hildenbrand Cc: Matthew Wilcox Cc: Michał Mirosław Cc: Muhammad Usama Anjum Cc: Oscar Salvador Cc: Peter Xu Cc: Ryan Roberts Cc: Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

5778ace0 - fs/proc: fix build with GCC 15 due to -Werror=unterminated-string-initialization

Sat, 05 Oct 2024 06:37:00 +0000

fs/proc: fix build with GCC 15 due to -Werror=unterminated-string-initializationshow show_smap_vma_flags() has been a using misspelled initializer inmnemonics[] - it needed to initialize 2 element array of char and it usedNUL-padded 2 character string literals (i.e. 3-element initializer).This has been spotted by gcc-15[*]; prior to that gcc quietly dropped the3rd eleemnt of initializers. To fix this we are increasing the size ofmnemonics[] (from mnemonics[BITS_PER_LONG][2] tomnemonics[BITS_PER_LONG][3]) to accomodate the NUL-padded string literals.This also helps us in simplyfying the logic for printing of the flags asinstead of printing each character from the mnemonics[], we can just printthe mnemonics[] using seq_printf.[*]: fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization] 917 | [0 ... (BITS_PER_LONG-1)] = "??", | ^~~~fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization]fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization]fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization]fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization]fs/proc/task_mmu.c:917:49: error: initializer-string for array of `char' is too long [-Werror=unterminate d-string-initialization]...Stephen pointed out:: The C standard explicitly allows for a string initializer to be too long: due to the NUL byte at the end ... so this warning may be overzealous.but let's make the warning go away anwyay.Link: https://lkml.kernel.org/r/20241005063700.2241027-1-brahmajit.xyz@gmail.comLink: https://lkml.kernel.org/r/20241003093040.47c08382@canb.auug.org.auSigned-off-by: Brahmajit Das Cc: Cyrill Gorcunov Cc: David Hildenbrand Cc: Stephen Rothwell Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

bcc9d04e - mm: Introduce ARCH_HAS_USER_SHADOW_STACK

Tue, 01 Oct 2024 22:58:40 +0000

mm: Introduce ARCH_HAS_USER_SHADOW_STACKSince multiple architectures have support for shadow stacks and we need toselect support for this feature in several places in the generic codeprovide a generic config option that the architectures can select.Suggested-by: David Hildenbrand Acked-by: David Hildenbrand Reviewed-by: Deepak Gupta Reviewed-by: Rick Edgecombe Reviewed-by: Mike Rapoport (IBM) Reviewed-by: Catalin Marinas Reviewed-by: Kees Cook Tested-by: Kees Cook Acked-by: Shuah Khan Reviewed-by: Thiago Jung Bauermann Signed-off-by: Mark Brown Link: https://lore.kernel.org/r/20241001-arm64-gcs-v13-1-222b78d87eee@kernel.orgSigned-off-by: Catalin Marinas List of files: /linux-6.15/fs/proc/task_mmu.c

d175ee98 - mm: Define VM_DROPPABLE for powerpc/32

Mon, 02 Sep 2024 19:17:18 +0000

mm: Define VM_DROPPABLE for powerpc/32Commit 9651fcedf7b9 ("mm: add MAP_DROPPABLE for designating alwayslazily freeable mappings") only adds VM_DROPPABLE for 64 bitsarchitectures.In order to also use the getrandom vDSO implementation on powerpc/32,use VM_ARCH_1 for VM_DROPPABLE on powerpc/32. This is possible becauseVM_ARCH_1 is used for VM_SAO on powerpc and VM_SAO is only forpowerpc/64. It is used in combination with PROT_SAO in some parts ofcode that are restricted to CONFIG_PPC64 through #ifdefs, it istherefore possible to define VM_SAO for CONFIG_PPC64 only.Signed-off-by: Christophe Leroy Acked-by: Michael Ellerman Signed-off-by: Jason A. Donenfeld List of files: /linux-6.15/fs/proc/task_mmu.c

9f82f15d - mm: use ARCH_PKEY_BITS to define VM_PKEY_BITN

Thu, 22 Aug 2024 15:10:46 +0000

mm: use ARCH_PKEY_BITS to define VM_PKEY_BITNUse the new CONFIG_ARCH_PKEY_BITS to simplify setting these bitsfor different architectures.Signed-off-by: Joey Gouly Cc: Andrew Morton Cc: linux-fsdevel@vger.kernel.orgCc: linux-mm@kvack.orgAcked-by: Dave Hansen Reviewed-by: Anshuman Khandual Link: https://lore.kernel.org/r/20240822151113.1479789-4-joey.gouly@arm.comSigned-off-by: Will Deacon List of files: /linux-6.15/fs/proc/task_mmu.c

00bd8ec2 - fs/procfs: remove build ID-related code duplication in PROCMAP_QUERY

Mon, 29 Jul 2024 17:40:44 +0000

fs/procfs: remove build ID-related code duplication in PROCMAP_QUERYA piece of build ID handling code in PROCMAP_QUERY ioctl() wasaccidentally duplicated. It wasn't meant to be part of ed5d583a88a9("fs/procfs: implement efficient VMA querying API for /proc//maps")commit, which is what introduced duplication.It has no correctness implications, but we unnecessarily perform the samework twice, if build ID parsing is requested. Drop the duplication.Link: https://lkml.kernel.org/r/20240729174044.4008399-1-andrii@kernel.orgFixes: ed5d583a88a9 ("fs/procfs: implement efficient VMA querying API for /proc//maps")Signed-off-by: Andrii Nakryiko Reported-by: Jann Horn Cc: Alexey Dobriyan Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

9651fced - mm: add MAP_DROPPABLE for designating always lazily freeable mappings

Thu, 08 Dec 2022 16:55:04 +0000

mm: add MAP_DROPPABLE for designating always lazily freeable mappingsThe vDSO getrandom() implementation works with a buffer allocated with anew system call that has certain requirements:- It shouldn't be written to core dumps. * Easy: VM_DONTDUMP.- It should be zeroed on fork. * Easy: VM_WIPEONFORK.- It shouldn't be written to swap. * Uh-oh: mlock is rlimited. * Uh-oh: mlock isn't inherited by forks.- It shouldn't reserve actual memory, but it also shouldn't crash when page faulting in memory if none is available * Uh-oh: VM_NORESERVE means segfaults.It turns out that the vDSO getrandom() function has three really nicecharacteristics that we can exploit to solve this problem:1) Due to being wiped during fork(), the vDSO code is already robust to having the contents of the pages it reads zeroed out midway through the function's execution.2) In the absolute worst case of whatever contingency we're coding for, we have the option to fallback to the getrandom() syscall, and everything is fine.3) The buffers the function uses are only ever useful for a maximum of 60 seconds -- a sort of cache, rather than a long term allocation.These characteristics mean that we can introduce VM_DROPPABLE, whichhas the following semantics:a) It never is written out to swap.b) Under memory pressure, mm can just drop the pages (so that they're zero when read back again).c) It is inherited by fork.d) It doesn't count against the mlock budget, since nothing is locked.e) If there's not enough memory to service a page fault, it's not fatal, and no signal is sent.This way, allocations used by vDSO getrandom() can use: VM_DROPPABLE | VM_DONTDUMP | VM_WIPEONFORK | VM_NORESERVEAnd there will be no problem with OOMing, crashing on overcommitment,using memory when not in use, not wiping on fork(), coredumps, orwriting out to swap.In order to let vDSO getrandom() use this, expose these via mmap(2) asMAP_DROPPABLE.Note that this involves removing the MADV_FREE special case fromsort_folio(), which according to Yu Zhao is unnecessary and will simplyresult in an extra call to shrink_folio_list() in the worst case. Thechunk removed reenables the swapbacked flag, which we don't want forVM_DROPPABLE, and we can't conditionalize it here because there isn't avma reference available.Finally, the provided self test ensures that this is working as desired.Cc: linux-mm@kvack.orgAcked-by: David Hildenbrand Signed-off-by: Jason A. Donenfeld List of files: /linux-6.15/fs/proc/task_mmu.c

e6c0c032 - mm: provide mm_struct and address to huge_ptep_get()

Tue, 02 Jul 2024 13:51:20 +0000

mm: provide mm_struct and address to huge_ptep_get()On powerpc 8xx huge_ptep_get() will need to know whether the given ptep isa PTE entry or a PMD entry. This cannot be known with the PMD entryitself because there is no easy way to know it from the content of theentry.So huge_ptep_get() will need to know either the size of the page or getthe pmd.In order to be consistent with huge_ptep_get_and_clear(), give mm andaddress to huge_ptep_get().Link: https://lkml.kernel.org/r/cc00c70dd384298796a4e1b25d6c4eb306d3af85.1719928057.git.christophe.leroy@csgroup.euSigned-off-by: Christophe Leroy Reviewed-by: Oscar Salvador Cc: Jason Gunthorpe Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Peter Xu Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

bfc69fd0 - fs/procfs: add build ID fetching to PROCMAP_QUERY API

Thu, 27 Jun 2024 17:08:55 +0000

fs/procfs: add build ID fetching to PROCMAP_QUERY APIThe need to get ELF build ID reliably is an important aspect when dealingwith profiling and stack trace symbolization, and /proc//maps textualrepresentation doesn't help with this.To get backing file's ELF build ID, application has to first resolve VMA,then use it's start/end address range to follow a special/proc//map_files/- symlink to open the ELF file (this isnecessary because backing file might have been removed from the disk orwas already replaced with another binary in the same file path.Such approach, beyond just adding complexity of having to do a bunch ofextra work, has extra security implications. Because application opensunderlying ELF file and needs read access to its entire contents (as faras kernel is concerned), kernel puts additional capable() checks onfollowing /proc//map_files/- symlink. And that makessense in general.But in the case of build ID, profiler/symbolizer doesn't need the contentsof ELF file, per se. It's only build ID that is of interest, and ELFbuild ID itself doesn't provide any sensitive information.So this patch adds a way to request backing file's ELF build ID along therest of VMA information in the same API. User has control over whetherthis piece of information is requested or not by either settingbuild_id_size field to zero or non-zero maximum buffer size they providedthrough build_id_addr field (which encodes user pointer as __u64 field). This is a completely optional piece of information, and so has noperformance implications for user cases that don't care about build ID,while improving performance and simplifying the setup for thoseapplication that do need it.Kernel already implements build ID fetching, which is used from BPFsubsystem. We are reusing this code here, but plan a follow up changes tomake it work better under more relaxed assumption (compared to whatexisting code assumes) of being called from user process context, in whichpage faults are allowed. BPF-specific implementation currently bails outif necessary part of ELF file is not paged in, all due to extraBPF-specific restrictions (like the need to fetch build ID in restrictivecontexts such as NMI handler).[andrii@kernel.org: fix integer to pointer cast warning in do_procmap_query()] Link: https://lkml.kernel.org/r/20240701174805.1897344-1-andrii@kernel.orgLink: https://lkml.kernel.org/r/20240627170900.1672542-4-andrii@kernel.orgSigned-off-by: Andrii Nakryiko Acked-by: Liam R. Howlett Cc: Alexey Dobriyan Cc: Al Viro Cc: Christian Brauner Cc: Greg Kroah-Hartman Cc: Mike Rapoport (IBM) Cc: Suren Baghdasaryan Cc: Andi Kleen Cc: Arnd Bergmann Cc: Stephen Rothwell Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

ed5d583a - fs/procfs: implement efficient VMA querying API for /proc//maps

Thu, 27 Jun 2024 17:08:54 +0000

fs/procfs: implement efficient VMA querying API for /proc//maps/proc//maps file is extremely useful in practice for various tasksinvolving figuring out process memory layout, what files are backing anygiven memory range, etc. One important class of applications thatabsolutely rely on this are profilers/stack symbolizers (perf tool beingone of them). Patterns of use differ, but they generally would fall intotwo categories.In on-demand pattern, a profiler/symbolizer would normally capture stacktrace containing absolute memory addresses of some functions, and wouldthen use /proc//maps file to find corresponding backing ELF files(normally, only executable VMAs are of interest), file offsets withinthem, and then continue from there to get yet more information (ELFsymbols, DWARF information) to get human-readable symbolic information. This pattern is used by Meta's fleet-wide profiler, as one example.In preprocessing pattern, application doesn't know the set of addresses ofinterest, so it has to fetch all relevant VMAs (again, probably onlyexecutable ones), store or cache them, then proceed with profiling andstack trace capture. Once done, it would do symbolization based on storedVMA information. This can happen at much later point in time. Thispatterns is used by perf tool, as an example.In either case, there are both performance and correctness requirementinvolved. This address to VMA information translation has to be done asefficiently as possible, but also not miss any VMA (especially in the caseof loading/unloading shared libraries). In practice, correctness can't beguaranteed (due to process dying before VMA data can be captured, orshared library being unloaded, etc), but any effort to maximize the chanceof finding the VMA is appreciated.Unfortunately, for all the /proc//maps file universality andusefulness, it doesn't fit the above use cases 100%.First, it's main purpose is to emit all VMAs sequentially, but in practicecaptured addresses would fall only into a smaller subset of all process'VMAs, mainly containing executable text. Yet, library would need to parsemost or all of the contents to find needed VMAs, as there is no way toskip VMAs that are of no use. Efficient library can do the linear passand it is still relatively efficient, but it's definitely an overhead thatcan be avoided, if there was a way to do more targeted querying of therelevant VMA information.Second, it's a text based interface, which makes its programmatic use fromapplications and libraries more cumbersome and inefficient due to the needto handle text parsing to get necessary pieces of information. Theoverhead is actually payed both by kernel, formatting originally binaryVMA data into text, and then by user space application, parsing it backinto binary data for further use.For the on-demand pattern of usage, described above, another problem whenwriting generic stack trace symbolization library is an unfortunateperformance-vs-correctness tradeoff that needs to be made. Library has tomake a decision to either cache parsed contents of /proc//maps (afterinitial processing) to service future requests (if application requests tosymbolize another set of addresses (for the same process), captured atsome later time, which is typical for periodic/continuous profiling cases)to avoid higher costs of re-parsing this file. Or it has to choose tocache the contents in memory to speed up future requests. In the formercase, more memory is used for the cache and there is a risk of gettingstale data if application loads or unloads shared libraries, or otherwisechanged its set of VMAs somehow, e.g., through additional mmap() calls. In the latter case, it's the performance hit that comes from re-openingthe file and re-parsing its contents all over again.This patch aims to solve this problem by providing a new API built on topof /proc//maps. It's meant to address both non-selectiveness andtext nature of /proc//maps, by giving user more control of what sortof VMA(s) needs to be queried, and being binary-based interface eliminatesthe overhead of text formatting (on kernel side) and parsing (on userspace side).It's also designed to be extensible and forward/backward compatible byincluding required struct size field, which user has to provide. We useestablished copy_struct_from_user() approach to handle extensibility.User has a choice to pick either getting VMA that covers provided addressor -ENOENT if none is found (exact, least surprising, case). Or, with anextra query flag (PROCMAP_QUERY_COVERING_OR_NEXT_VMA), they can get eitherVMA that covers the address (if there is one), or the closest next VMA(i.e., VMA with the smallest vm_start > addr). The latter allows moreefficient use, but, given it could be a surprising behavior, requires anexplicit opt-in.There is another query flag that is useful for some use cases. PROCMAP_QUERY_FILE_BACKED_VMA instructs this API to only returnfile-backed VMAs. Combining this with PROCMAP_QUERY_COVERING_OR_NEXT_VMAmakes it possible to efficiently iterate only file-backed VMAs of theprocess, which is what profilers/symbolizers are normally interested in.All the above querying flags can be combined with (also optional) set ofdesired VMA permissions flags. This allows to, for example, iterate onlyan executable subset of VMAs, which is what preprocessing pattern, used byperf tool, would benefit from, as the assumption is that captured stacktraces would have addresses of executable code. This saves time byskipping non-executable VMAs altogether efficienty.All these querying flags (modifiers) are orthogonal and can be combined ina semantically meaningful and natural way.Basing this ioctl()-based API on top of /proc//maps's FD makes sensegiven it's querying the same set of VMA data. It's also benefitialbecause permission checks for /proc//maps is performed at open timeonce, and the actual data read of text contents of /proc//maps isdone without further permission checks. We piggyback on this pattern withioctl()-based API as well, as that's a desired property. Both forperformance reasons, but also for security and flexibility reasons.Allowing application to open an FD for /proc/self/maps without any extracapabilities, and then passing it to some sort of profiling agent throughUnix-domain socket, would allow such profiling agent to not require someof the capabilities that are otherwise expected when opening/proc//maps file for *another* process. This is a desirable propertyfor some more restricted setups.This new ioctl-based implementation doesn't interfere with seq_file-basedimplementation of /proc//maps textual interface, and so could be usedtogether or independently without paying any price for that.Note also, that fetching VMA name (e.g., backing file path, or specialhard-coded or user-provided names) is optional just like build ID. Ifuser sets vma_name_size to zero, kernel code won't attempt to retrieve it,saving resources.Earlier versions of this patch set were adding per-VMA locking, which iswhy we have a code structure that is ready for abstracting mmap_lock vsvm_lock differences (query_vma_setup(), query_vma_teardown(), andquery_vma_find_by_addr()), but given anon_vma_name() is not yet compatiblewith per-VMA locking, initial implementation sticks to using onlymmap_lock for now. It will be easy to add back per-VMA locking once allthe pieces are ready later on. Which is why we keep existing codestructure with setup/teardown/query helper functions.[andrii@kernel.org: improve PROCMAP_QUERY's compat mode handling] Link: https://lkml.kernel.org/r/20240701174805.1897344-2-andrii@kernel.orgLink: https://lkml.kernel.org/r/20240627170900.1672542-3-andrii@kernel.orgSigned-off-by: Andrii Nakryiko Acked-by: Liam R. Howlett Cc: Alexey Dobriyan Cc: Al Viro Cc: Christian Brauner Cc: Greg Kroah-Hartman Cc: Mike Rapoport (IBM) Cc: Suren Baghdasaryan Cc: Andi Kleen Cc: Arnd Bergmann Cc: Stephen Rothwell Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

acd4b2ec - fs/procfs: extract logic for getting VMA name constituents

Thu, 27 Jun 2024 17:08:53 +0000

fs/procfs: extract logic for getting VMA name constituentsPatch series "ioctl()-based API to query VMAs from /proc//maps", v6.Implement binary ioctl()-based interface to /proc//maps file to allowapplications to query VMA information more efficiently than reading *all*VMAs nonselectively through text-based interface of /proc//maps file.Patch #2 goes into a lot of details and background on some common patternsof using /proc//maps in the area of performance profiling andsubsequent symbolization of captured stack traces. As mentioned in thatpatch, patterns of VMA querying can differ depending on specific use case,but can generally be grouped into two main categories: the need to query asmall subset of VMAs covering a given batch of addresses, orreading/storing/caching all (typically, executable) VMAs upfront for laterprocessing.The new PROCMAP_QUERY ioctl() API added in this patch set was motivated bythe former pattern of usage. Earlier revisions had a patch adding a toolthat faithfully reproduces an efficient VMA matching pass of a symbolizer,collecting a subset of covering VMAs for a given set of addresses asefficiently as possible. This tool served both as a testing ground, aswell as a benchmarking tool. It implements everything both for currentlyexisting text-based /proc//maps interface, as well as for newly-addedPROCMAP_QUERY ioctl(). This revision dropped the tool from the patch setand, once the API lands upstream, this tool might be added separately onGithub as an example.Based on discussion on earlier revisions of this patch set, it turned outthat this ioctl() API is competitive with highly-optimized text-basedpre-processing pattern that perf tool is using. Based on perf discussion,this revision adds more flexibility in specifying a subset of VMAs thatare of interest. Now it's possible to specify desired permissions of VMAs(e.g., request only executable ones) and/or restrict to only a subset ofVMAs that have file backing. This further improves the efficiency whenusing this new API thanks to more selective (executable VMAs only)querying.In addition to a custom benchmarking tool, and experimental perfintegration (available at [0]), Daniel Mueller has since also implementedan experimental integration into blazesym (see [1]), a library used forstack trace symbolization by our server fleet-wide profiler and anotheron-device profiler agent that runs on weaker ARM devices. The latterARM-based device profiler is especially sensitive to performance, and sowe benchmarked and compared text-based /proc//maps solution to theequivalent one using PROCMAP_QUERY ioctl().Results are very encouraging, giving us 5x improvement for end-to-endso-called "address normalization" pass, which is the part of thesymbolization process that happens locally on ARM device, before beingsent out for further heavier-weight processing on more powerful remoteserver. Note that this is not an artificial microbenchmark. It's a fullend-to-end API call being measured with real-world data on real-worlddevice. TEXT-BASED ========== Benchmarking main/normalize_process_no_build_ids_uncached_maps main/normalize_process_no_build_ids_uncached_maps time: [49.777 µs 49.982 µs 50.250 µs] IOCTL-BASED =========== Benchmarking main/normalize_process_no_build_ids_uncached_maps main/normalize_process_no_build_ids_uncached_maps time: [10.328 µs 10.391 µs 10.457 µs] change: [−79.453% −79.304% −79.166%] (p = 0.00 < 0.02) Performance has improved.You can see above that we see the drop from 50µs down to 10µs forexactly the same amount of work, with the same data and target process.With the aforementioned custom tool, we see about ~40x improvement (itmight vary a bit, depending on a specific captured set of addresses). Andeven for perf-based benchmark it's on par or slightly ahead when usingpermission-based filtering (fetching only executable VMAs).Earlier revisions attempted to use per-VMA locking, if kernel was compiledwith CONFIG_PER_VMA_LOCK=y, but it turned out that anon_vma_name() is notyet compatible with per-VMA locking and assumes mmap_lock to be taken,which makes the use of per-VMA locking for this API premature. It wasagreed ([2]) to continue for now with just mmap_lock, but the codestructure is such that it should be easy to add per-VMA locking supportonce all the pieces are ready.One thing that did not change was basing this new API as an ioctl()command on /proc//maps file. An ioctl-based API on top of pidfd wasconsidered, but has its own downsides. Implementing ioctl() directly onpidfd will cause access permission checks on every single ioctl(), whichleads to performance concerns and potential spam of capable() auditmessages. It also prevents a nice pattern, possible with/proc//maps, in which application opens /proc/self/maps FD (requiringno additional capabilities) and passed this FD to profiling agent forquerying. To achieve similar pattern, a new file would have to be createdfrom pidf just for VMA querying, which is considered to be inferior tojust querying /proc//maps FD as proposed in current approach. Theseaspects were discussed in the hallway track at recent LSF/MM/BPF 2024 andsticking to procfs ioctl() was the final agreement we arrived at. [0] https://github.com/anakryiko/linux/commits/procfs-proc-maps-ioctl-v2/ [1] https://github.com/libbpf/blazesym/pull/675 [2] https://lore.kernel.org/bpf/7rm3izyq2vjp5evdjc7c6z4crdd3oerpiknumdnmmemwyiwx7t@hleldw7iozi3/This patch (of 6):Extract generic logic to fetch relevant pieces of data to describe VMAname. This could be just some string (either special constant oruser-provided), or a string with some formatted wrapping text (e.g.,"[anon_shmem:]"), or, commonly, file path. seq_file-basedlogic has different methods to handle all three cases, but they arecurrently mixed in with extracting underlying sources of data.This patch splits this into data fetching and data formatting, so thatdata fetching can be reused later on.There should be no functional changes.Link: https://lkml.kernel.org/r/20240627170900.1672542-1-andrii@kernel.orgLink: https://lkml.kernel.org/r/20240627170900.1672542-2-andrii@kernel.orgSigned-off-by: Andrii Nakryiko Acked-by: Liam R. Howlett Cc: Alexey Dobriyan Cc: Al Viro Cc: Christian Brauner Cc: Greg Kroah-Hartman Cc: Mike Rapoport (IBM) Cc: Suren Baghdasaryan Cc: Andi Kleen Cc: Arnd Bergmann Cc: Stephen Rothwell Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

aca08acc - fs/proc/task_mmu: use folio API in pte_is_pinned()

Tue, 04 Jun 2024 11:48:19 +0000

fs/proc/task_mmu: use folio API in pte_is_pinned()Patch series "mm: remove page_maybe_dma_pinned() and page_mkclean()".Most page_maybe_dma_pinned() and page_mkclean() callers have beenconverted to the folio equivalents, after two more convertsions,remove them and update the comment and documention.This patch (of 4):Convert to use vm_normal_folio() and folio_maybe_dma_pinned() API, whichhelps to remove page_maybe_dma_pinned() in the subsequent change.Link: https://lkml.kernel.org/r/20240604114822.2089819-1-wangkefeng.wang@huawei.comLink: https://lkml.kernel.org/r/20240604114822.2089819-2-wangkefeng.wang@huawei.comSigned-off-by: Kefeng Wang Acked-by: David Hildenbrand Cc: Daniel Vetter Cc: Helge Deller Cc: Jonathan Corbet Cc: Matthew Wilcox (Oracle) Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c

cdd9a571 - fs/proc: move page_mapcount() to fs/proc/internal.h

Fri, 07 Jun 2024 12:23:56 +0000

fs/proc: move page_mapcount() to fs/proc/internal.h... and rename it to folio_precise_page_mapcount(). fs/proc is the lastremaining user, and that should stay that way.While at it, cleanup kpagecount_read() a bit: there are still some legacyleftovers -- when the interface was introduced it returned the pagerefcount, but was changed briefly afterwards to return the page mapcount. Further, some simple folio conversion.Once we stop using the per-page mapcounts of large folios, allfolio_precise_page_mapcount() users will have to implement an alternativeway to achieve what they are trying to achieve, possibly in a less preciseway.[dan.carpenter@linaro.org: fix uninitialized variable in pagemap_pmd_range()] Link: https://lkml.kernel.org/r/9d6eaba7-92f8-4a70-8765-38a519680a87@moroto.mountainLink: https://lkml.kernel.org/r/20240607122357.115423-6-david@redhat.comSigned-off-by: David Hildenbrand Signed-off-by: Dan Carpenter Cc: Alexey Dobriyan Cc: Jonathan Corbet Cc: Kirill A. Shutemov Cc: Lance Yang Cc: Oscar Salvador Signed-off-by: Andrew Morton List of files: /linux-6.15/fs/proc/task_mmu.c