Changes in tlb.h

02d9e1a2 - mm: pgtable: remove tlb_remove_page_ptdesc()

Tue, 25 Feb 2025 03:45:56 +0000

mm: pgtable: remove tlb_remove_page_ptdesc()The tlb_remove_ptdesc()/tlb_remove_table() is specially designed for pagetable pages, and now all architectures have been converted to use it toremove page table pages. So let's remove tlb_remove_page_ptdesc(), itcurrently has no users and should not be used for page table pages.Link: https://lkml.kernel.org/r/3df04c8494339073b71be4acb2d92e108ecd1b60.1740454179.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: Alexandre Ghiti Cc: "Aneesh Kumar K.V" Cc: Arnd Bergmann Cc: Dave Hansen Cc: David Hildenbrand Cc: Hugh Dickens Cc: Jann Horn Cc: Matthew Wilcow (Oracle) Cc: "Mike Rapoport (IBM)" Cc: Muchun Song Cc: Nicholas Piggin Cc: Rik van Riel Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

1a03c275 - mm: pgtable: change pt parameter of tlb_remove_ptdesc() to struct ptdesc*

Tue, 25 Feb 2025 03:45:52 +0000

mm: pgtable: change pt parameter of tlb_remove_ptdesc() to struct ptdesc*All callers of tlb_remove_ptdesc() pass it a pointer of struct ptdesc, solet's change the pt parameter from void * to struct ptdesc * to perform atype safety check.Link: https://lkml.kernel.org/r/60bb44299cf2d731df6592e446e7f694054d0dbe.1740454179.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Cc: Alexandre Ghiti Cc: "Aneesh Kumar K.V" Cc: Arnd Bergmann Cc: Dave Hansen Cc: David Hildenbrand Cc: Hugh Dickens Cc: Jann Horn Cc: Matthew Wilcow (Oracle) Cc: "Mike Rapoport (IBM)" Cc: Muchun Song Cc: Nicholas Piggin Cc: Rik van Riel Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

f21bb37a - mm: pgtable: make generic tlb_remove_table() use struct ptdesc

Tue, 25 Feb 2025 03:45:51 +0000

mm: pgtable: make generic tlb_remove_table() use struct ptdescPatch series "remove tlb_remove_page_ptdesc()", v2.As suggested by Peter Zijlstra below [1], this series aims to removetlb_remove_page_ptdesc().: Fundamentally tlb_remove_page() is about removing *pages* as from a PTE,: there should not be a page-table anywhere near here *ever*.:: Yes, some architectures use tlb_remove_page() for page-tables too, but: that is more or less an implementation detail that can be fixed.After this series, all architectures use tlb_remove_table() ortlb_remove_ptdesc() to remove the page table pages. In the future, onceall architectures using tlb_remove_table() have also converted to usingstruct ptdesc (eg. powerpc), it may be possible to use onlytlb_remove_ptdesc().[1] https://lore.kernel.org/linux-mm/20250103111457.GC22934@noisy.programming.kicks-ass.net/This patch (of 6):Now only arm will call tlb_remove_ptdesc()/tlb_remove_table() whenCONFIG_MMU_GATHER_TABLE_FREE is disabled. In this case, the type of thetable parameter is actually struct ptdesc * instead of struct page *.Since struct ptdesc still overlaps with struct page and has not beenseparated from it, forcing the table parameter to struct page * will notcause any problems at this time. But this is definitely incorrect andneeds to be fixed. So just like the generic __tlb_remove_table(), letgeneric tlb_remove_table() use struct ptdesc by default whenCONFIG_MMU_GATHER_TABLE_FREE is disabled.Link: https://lkml.kernel.org/r/cover.1740454179.git.zhengqi.arch@bytedance.comLink: https://lkml.kernel.org/r/5be8c3ab7bd68510bf0db4cf84010f4dfe372917.1740454179.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Reviewed-by: Kevin Brodsky Cc: Alexandre Ghiti Cc: "Aneesh Kumar K.V" Cc: Arnd Bergmann Cc: Dave Hansen Cc: David Hildenbrand Cc: Hugh Dickens Cc: Jann Horn Cc: Matthew Wilcow (Oracle) Cc: "Mike Rapoport (IBM)" Cc: Muchun Song Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Rik van Riel Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

7bd1fa0d - mm/mmu_gather: clean up the stale code comment

Tue, 11 Feb 2025 03:43:48 +0000

mm/mmu_gather: clean up the stale code commentIn commit d7f861b9c43a ("mm/mmu_gather: add __tlb_remove_folio_pages()"),helper function __tlb_remove_folio_pages_size() was added. And based onthe helper, wrapper functions __tlb_remove_folio_pages() and__tlb_remove_page_size() are created and used by upper level functions.So let's update the code comment to reflect the current code abouttlb_remove_page()/tlb_remove_page_size(), etc.Link: https://lkml.kernel.org/r/20250211034348.39531-2-bhe@redhat.comSigned-off-by: Baoquan He Cc: "Aneesh Kumar K.V" Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Qi Zheng Cc: Will Deacon Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

1d23b940 - mm/mmu_gather: remove unused __tlb_remove_page()

Tue, 11 Feb 2025 10:13:39 +0000

mm/mmu_gather: remove unused __tlb_remove_page()Nobody is using __tlb_remove_page() now, clean it up.And also remove the code comment above tlb_remove_page() because it's notmeaningful any more.Link: https://lkml.kernel.org/r/Z6si0/A/zzEF/bFJ@MiWiFi-R3L-srvSigned-off-by: Baoquan He Reviewed-by: Qi Zheng Cc: "Aneesh Kumar K.V" Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Will Deacon Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

553e7752 - mm: pgtable: introduce generic pagetable_dtor_free()

Wed, 08 Jan 2025 06:57:33 +0000

mm: pgtable: introduce generic pagetable_dtor_free()The pte_free(), pmd_free(), __pud_free() and __p4d_free() inasm-generic/pgalloc.h and the generic __tlb_remove_table() are basicallythe same, so let's introduce pagetable_dtor_free() to deduplicate them.In addition, the pagetable_dtor_free() in s390 does the same thing, solet's s390 also calls generic pagetable_dtor_free().Link: https://lkml.kernel.org/r/1663a0565aca881d1338ceb7d1db4aa9c333abd6.1736317725.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Suggested-by: Peter Zijlstra (Intel) Reviewed-by: Kevin Brodsky Acked-by: Alexander Gordeev [s390]Cc: Alexandre Ghiti Cc: Alexandre Ghiti Cc: Andreas Larsson Cc: Aneesh Kumar K.V (Arm) Cc: Arnd Bergmann Cc: Dave Hansen Cc: David Hildenbrand Cc: David Rientjes Cc: Hugh Dickins Cc: Jann Horn Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Mike Rapoport (Microsoft) Cc: Muchun Song Cc: Nicholas Piggin Cc: Palmer Dabbelt Cc: Ryan Roberts Cc: Thomas Gleixner Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

92ec7fd1 - mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table()

Wed, 08 Jan 2025 06:57:31 +0000

mm: pgtable: completely move pagetable_dtor() to generic tlb_remove_table()For the generic tlb_remove_table(), it is implemented in the following twoforms:1) CONFIG_MMU_GATHER_TABLE_FREE is enabledtlb_remove_table--> generic __tlb_remove_table()2) CONFIG_MMU_GATHER_TABLE_FREE is disabledtlb_remove_table--> tlb_remove_pageFor case 1), the pagetable_dtor() has already been moved to generic__tlb_remove_table().For case 2), now only arm will call tlb_remove_table()/tlb_remove_ptdesc()when CONFIG_MMU_GATHER_TABLE_FREE is disabled. Let's movepagetable_dtor() completely to generic tlb_remove_table(), so that thearchitectures can follow more easily.Link: https://lkml.kernel.org/r/0c733ac867b287ec08190676496d1decebf49da2.1736317725.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Suggested-by: Kevin Brodsky Reviewed-by: Kevin Brodsky Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Alexandre Ghiti Cc: Andreas Larsson Cc: Aneesh Kumar K.V (Arm) Cc: Arnd Bergmann Cc: Dave Hansen Cc: David Hildenbrand Cc: David Rientjes Cc: Hugh Dickins Cc: Jann Horn Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Mike Rapoport (Microsoft) Cc: Muchun Song Cc: Nicholas Piggin Cc: Palmer Dabbelt Cc: Peter Zijlstra (Intel) Cc: Ryan Roberts Cc: Thomas Gleixner Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

2dccdf70 - mm: pgtable: introduce generic __tlb_remove_table()

Wed, 08 Jan 2025 06:57:30 +0000

mm: pgtable: introduce generic __tlb_remove_table()Several architectures (arm, arm64, riscv and x86) define exactly the same__tlb_remove_table(), just introduce generic __tlb_remove_table() toeliminate these duplications.The s390 __tlb_remove_table() is nearly the same, so also make s390__tlb_remove_table() version generic.Link: https://lkml.kernel.org/r/ea372633d94f4d3f9f56a7ec5994bf050bf77e39.1736317725.git.zhengqi.arch@bytedance.comSigned-off-by: Qi Zheng Reviewed-by: Kevin Brodsky Acked-by: Andreas Larsson [sparc]Acked-by: Alexander Gordeev [s390]Acked-by: Arnd Bergmann [asm-generic]Cc: Alexandre Ghiti Cc: Alexandre Ghiti Cc: Aneesh Kumar K.V (Arm) Cc: Dave Hansen Cc: David Hildenbrand Cc: David Rientjes Cc: Hugh Dickins Cc: Jann Horn Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Mike Rapoport (Microsoft) Cc: Muchun Song Cc: Nicholas Piggin Cc: Palmer Dabbelt Cc: Peter Zijlstra (Intel) Cc: Ryan Roberts Cc: Thomas Gleixner Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yu Zhao Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

d7f861b9 - mm/mmu_gather: add __tlb_remove_folio_pages()

Wed, 14 Feb 2024 20:44:33 +0000

mm/mmu_gather: add __tlb_remove_folio_pages()Add __tlb_remove_folio_pages(), which will remove multiple consecutivepages that belong to the same large folio, instead of only a single page. We'll be using this function when optimizing unmapping/zapping of largefolios that are mapped by PTEs.We're using the remaining spare bit in an encoded_page to indicate thatthe next enoced page in an array contains actually shifted "nr_pages". Teach swap/freeing code about putting multiple folio references, anddelayed rmap handling to remove page ranges of a folio.This extension allows for still gathering almost as many small folios aswe used to (-1, because we have to prepare for a possibly bigger nextentry), but still allows for gathering consecutive pages that belong tothe same large folio.Note that we don't pass the folio pointer, because it is not required fornow. Further, we don't support page_size != PAGE_SIZE, it won't berequired for simple PTE batching.We have to provide a separate s390 implementation, but it's fairlystraight forward.Another, more invasive and likely more expensive, approach would be to usefolio+range or a PFN range instead of page+nr_pages. But, we should dothat consistently for the whole mmu_gather. For now, let's keep it simpleand add "nr_pages" only.Note that it is now possible to gather significantly more pages: In thepast, we were able to gather ~10000 pages, now we can also gather ~5000folio fragments that span multiple pages. A folio fragment on x86-64 canspan up to 512 pages (2 MiB THP) and on arm64 with 64k in theory 8192pages (512 MiB THP). Gathering more memory is not considered something weshould worry about, especially because these are already corner cases.While we can gather more total memory, we won't free more folio fragments.As long as page freeing time primarily only depends on the number ofinvolved folios, there is no effective change for !preempt configurations.However, we'll adjust tlb_batch_pages_flush() separately to handle cornercases where page freeing time grows proportionally with the actual memorysize.Link: https://lkml.kernel.org/r/20240214204435.167852-9-david@redhat.comSigned-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Cc: Alexander Gordeev Cc: Aneesh Kumar K.V Cc: Arnd Bergmann Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Christophe Leroy Cc: Heiko Carstens Cc: Matthew Wilcox (Oracle) Cc: Michael Ellerman Cc: Michal Hocko Cc: "Naveen N. Rao" Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Sven Schnelle Cc: Vasily Gorbik Cc: Will Deacon Cc: Yin Fengwei Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

4d5bf0b6 - mm/mmu_gather: add tlb_remove_tlb_entries()

Wed, 14 Feb 2024 20:44:32 +0000

mm/mmu_gather: add tlb_remove_tlb_entries()Let's add a helper that lets us batch-process multiple consecutive PTEs.Note that the loop will get optimized out on all architectures except onpowerpc. We have to add an early define of __tlb_remove_tlb_entry() onppc to make the compiler happy (and avoid making tlb_remove_tlb_entries()a macro).[arnd@kernel.org: change __tlb_remove_tlb_entry() to an inline function] Link: https://lkml.kernel.org/r/20240221154549.2026073-1-arnd@kernel.orgLink: https://lkml.kernel.org/r/20240214204435.167852-8-david@redhat.comSigned-off-by: David Hildenbrand Signed-off-by: Arnd Bergmann Reviewed-by: Ryan Roberts Cc: Alexander Gordeev Cc: Aneesh Kumar K.V Cc: Arnd Bergmann Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Christophe Leroy Cc: Heiko Carstens Cc: Matthew Wilcox (Oracle) Cc: Michael Ellerman Cc: Michal Hocko Cc: "Naveen N. Rao" Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Sven Schnelle Cc: Vasily Gorbik Cc: Will Deacon Cc: Yin Fengwei Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

c30d6bc8 - mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size()

Wed, 14 Feb 2024 20:44:30 +0000

mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size()We have two bits available in the encoded page pointer to store additionalinformation. Currently, we use one bit to request delay of the rmapremoval until after a TLB flush.We want to make use of the remaining bit internally for batching ofmultiple pages of the same folio, specifying that the next encoded pagepointer in an array is actually "nr_pages". So pass page + delay_rmapflag instead of an encoded page, to handle the encoding internally.Link: https://lkml.kernel.org/r/20240214204435.167852-6-david@redhat.comSigned-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Cc: Alexander Gordeev Cc: Aneesh Kumar K.V Cc: Arnd Bergmann Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Christophe Leroy Cc: Heiko Carstens Cc: Matthew Wilcox (Oracle) Cc: Michael Ellerman Cc: Michal Hocko Cc: "Naveen N. Rao" Cc: Nicholas Piggin Cc: Peter Zijlstra (Intel) Cc: Sven Schnelle Cc: Vasily Gorbik Cc: Will Deacon Cc: Yin Fengwei Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

bf2d4334 - mm: add utility functions for ptdesc

Mon, 07 Aug 2023 23:04:45 +0000

mm: add utility functions for ptdescIntroduce utility functions setting the foundation for ptdescs. Thesewill also assist in the splitting out of ptdesc from struct page.Functions that focus on the descriptor are prefixed with ptdesc_* whilefunctions that focus on the pagetable are prefixed with pagetable_*.pagetable_alloc() is defined to allocate new ptdesc pages as compoundpages. This is to standardize ptdescs by allowing for one allocation andone free function, in contrast to 2 allocation and 2 free functions.Link: https://lkml.kernel.org/r/20230807230513.102486-4-vishal.moola@gmail.comSigned-off-by: Vishal Moola (Oracle) Cc: Arnd Bergmann Cc: Catalin Marinas Cc: Christophe Leroy Cc: Claudio Imbrenda Cc: Dave Hansen Cc: David Hildenbrand Cc: "David S. Miller" Cc: Dinh Nguyen Cc: Geert Uytterhoeven Cc: Geert Uytterhoeven Cc: Guo Ren Cc: Huacai Chen Cc: Hugh Dickins Cc: John Paul Adrian Glaubitz Cc: Jonas Bonn Cc: Matthew Wilcox Cc: Mike Rapoport (IBM) Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: Richard Weinberger Cc: Thomas Bogendoerfer Cc: Yoshinori Sato Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

6bbd42e2 - mmu_notifiers: call invalidate_range() when invalidating TLBs

Tue, 25 Jul 2023 13:42:05 +0000

mmu_notifiers: call invalidate_range() when invalidating TLBsThe invalidate_range() is going to become an architecture specific mmunotifier used to keep the TLB of secondary MMUs such as an IOMMU in syncwith the CPU page tables. Currently it is called from separate code pathsto the main CPU TLB invalidations. This can lead to a secondary TLB notgetting invalidated when required and makes it hard to reason about whenexactly the secondary TLB is invalidated.To fix this move the notifier call to the architecture specific TLBmaintenance functions for architectures that have secondary MMUs requiringexplicit software invalidations.This fixes a SMMU bug on ARM64. On ARM64 PTE permission upgrades requirea TLB invalidation. This invalidation is done by the architecturespecific ptep_set_access_flags() which calls flush_tlb_page() if required.However this doesn't call the notifier resulting in infinite faults beinggenerated by devices using the SMMU if it has previously cached aread-only PTE in it's TLB.Moving the invalidations into the TLB invalidation functions ensures allinvalidations happen at the same time as the CPU invalidation. Thearchitecture specific flush_tlb_all() routines do not call the notifier asnone of the IOMMUs require this.Link: https://lkml.kernel.org/r/0287ae32d91393a582897d6c4db6f7456b1001f2.1690292440.git-series.apopple@nvidia.comSigned-off-by: Alistair Popple Suggested-by: Jason Gunthorpe Tested-by: SeongJae Park Acked-by: Catalin Marinas Reviewed-by: Jason Gunthorpe Tested-by: Luis Chamberlain Cc: Andrew Donnellan Cc: Chaitanya Kumar Borah Cc: Frederic Barrat Cc: John Hubbard Cc: Kevin Tian Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Nicolin Chen Cc: Robin Murphy Cc: Sean Christopherson Cc: Tvrtko Ursulin Cc: Will Deacon Cc: Zhi Wang Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

5df397de - mm: delay page_remove_rmap() until after the TLB has been flushed

Wed, 09 Nov 2022 20:30:51 +0000

mm: delay page_remove_rmap() until after the TLB has been flushedWhen we remove a page table entry, we are very careful to only free thepage after we have flushed the TLB, because other CPUs could still beusing the page through stale TLB entries until after the flush.However, we have removed the rmap entry for that page early, which meansthat functions like folio_mkclean() would end up not serializing with thepage table lock because the page had already been made invisible to rmap.And that is a problem, because while the TLB entry exists, we could end upwith the following situation: (a) one CPU could come in and clean it, never seeing our mapping of the page (b) another CPU could continue to use the stale and dirty TLB entry and continue to write to said pageresulting in a page that has been dirtied, but then marked clean again,all while another CPU might have dirtied it some more.End result: possibly lost dirty data.This extends our current TLB gather infrastructure to optionally track a"should I do a delayed page_remove_rmap() for this page after flushing theTLB". It uses the newly introduced 'encoded page pointer' to do thatwithout having to keep separate data around.Note, this is complicated by a couple of issues: - we want to delay the rmap removal, but not past the page table lock, because that simplifies the memcg accounting - only SMP configurations want to delay TLB flushing, since on UP there are obviously no remote TLBs to worry about, and the page table lock means there are no preemption issues either - s390 has its own mmu_gather model that doesn't delay TLB flushing, and as a result also does not want the delayed rmap. As such, we can treat S390 like the UP case and use a common fallback for the "no delays" case. - we can track an enormous number of pages in our mmu_gather structure, with MAX_GATHER_BATCH_COUNT batches of MAX_TABLE_BATCH pages each, all set up to be approximately 10k pending pages. We do not want to have a huge number of batched pages that we then need to check for delayed rmap handling inside the page table lock.Particularly that last point results in a noteworthy detail, where thenormal page batch gathering is limited once we have delayed rmaps pending,in such a way that only the last batch (the so-called "active batch") inthe mmu_gather structure can have any delayed entries.NOTE! While the "possibly lost dirty data" sounds catastrophic, for thisall to happen you need to have a user thread doing either madvise() withMADV_DONTNEED or a full re-mmap() of the area concurrently with anotherthread continuing to use said mapping.So arguably this is about user space doing crazy things, but from a VMconsistency standpoint it's better if we track the dirty bit properly evenwhen user space goes off the rails.[akpm@linux-foundation.org: fix UP build, per Linus]Link: https://lore.kernel.org/all/B88D3073-440A-41C7-95F4-895D3F657EF2@gmail.com/Link: https://lkml.kernel.org/r/20221109203051.1835763-4-torvalds@linux-foundation.orgSigned-off-by: Linus Torvalds Acked-by: Johannes Weiner Acked-by: Hugh Dickins Reported-by: Nadav Amit Tested-by: Nadav Amit Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

7cc8f9c7 - mm: mmu_gather: prepare to gather encoded page pointers with flags

Wed, 09 Nov 2022 20:30:50 +0000

mm: mmu_gather: prepare to gather encoded page pointers with flagsThis is purely a preparatory patch that makes all the data structuresready for encoding flags with the mmu_gather page pointers.The code currently always sets the flag to zero and doesn't use it yet,but now it's tracking the type state along. The next step will be toactually start using it.Link: https://lkml.kernel.org/r/20221109203051.1835763-3-torvalds@linux-foundation.orgSigned-off-by: Linus Torvalds Acked-by: Johannes Weiner Acked-by: Hugh Dickins Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

2ba99c5e - mm/khugepaged: fix GUP-fast interaction by sending IPI

Fri, 25 Nov 2022 21:37:13 +0000

mm/khugepaged: fix GUP-fast interaction by sending IPISince commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THPcollapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t toensure that the page table was not removed by khugepaged in between.However, lockless_pages_from_mm() still requires that the page table isnot concurrently freed. Fix it by sending IPIs (if the architecture usessemi-RCU-style page table freeing) before freeing/reusing page tables.Link: https://lkml.kernel.org/r/20221129154730.2274278-2-jannh@google.comLink: https://lkml.kernel.org/r/20221128180252.1684965-2-jannh@google.comLink: https://lkml.kernel.org/r/20221125213714.4115729-2-jannh@google.comFixes: ba76149f47d8 ("thp: khugepaged")Signed-off-by: Jann Horn Reviewed-by: Yang Shi Acked-by: David Hildenbrand Cc: John Hubbard Cc: Peter Xu Cc: Signed-off-by: Andrew Morton List of files: /linux-6.15/include/asm-generic/tlb.h

7fb5e508 - mmu_gather: fix the CONFIG_MMU_GATHER_NO_RANGE case

Fri, 22 Jul 2022 16:28:34 +0000

mmu_gather: fix the CONFIG_MMU_GATHER_NO_RANGE caseSudip reports that alpha doesn't build properly, with errors like include/asm-generic/tlb.h:401:1: error: redefinition of 'tlb_update_vma_flags' 401 | tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) | ^~~~~~~~~~~~~~~~~~~~ include/asm-generic/tlb.h:372:1: note: previous definition of 'tlb_update_vma_flags' with type 'void(struct mmu_gather *, struct vm_area_struct *)' 372 | tlb_update_vma_flags(struct mmu_gather *tlb, struct vm_area_struct *vma) { }the cause being that We have this odd situation where some architectureswere never converted to the newer TLB flushing interfaces that have arange for the flush. Instead people left them alone, and we have themselect the MMU_GATHER_NO_RANGE config option to make the tlb headerfiles account for this.Peter Zijlstra cleaned some of these nasty header file games up incommits 1e9fdf21a433 ("mmu_gather: Remove per arch tlb_{start,end}_vma()") 18ba064e42df ("mmu_gather: Let there be one tlb_{start,end}_vma() implementation")but tlb_update_vma_flags() was left alone, and then commit b67fbebd4cf9("mmu_gather: Force tlb-flush VM_PFNMAP vmas") ended up removing only_one_ of the two stale duplicate dummy inline functions.This removes the other stale one.Somebody braver than me should try to remove MMU_GATHER_NO_RANGEentirely, but it requires fixing up the oddball architectures that useit: alpha, m68k, microblaze, nios2 and openrisc.The fixups should be fairly straightforward ("fix the build errors itexposes by adding the appropriate range arguments"), but the reason thiswasn't done in the first place is that so few people end up working onthose architectures. But it could be done one architecture at a time,hint, hint.Reported-by: Sudip Mukherjee (Codethink) Fixes: b67fbebd4cf9 ("mmu_gather: Force tlb-flush VM_PFNMAP vmas")Link: https://lore.kernel.org/all/YtpXh0QHWwaEWVAY@debian/Cc: Peter Zijlstra Cc: Will Deacon Cc: "Aneesh Kumar K.V" Cc: Andrew Morton Cc: Nick Piggin Cc: Arnd Bergmann Signed-off-by: Linus Torvalds List of files: /linux-6.15/include/asm-generic/tlb.h

b67fbebd - mmu_gather: Force tlb-flush VM_PFNMAP vmas

Fri, 08 Jul 2022 07:18:06 +0000

mmu_gather: Force tlb-flush VM_PFNMAP vmasJann reported a race between munmap() and unmap_mapping_range(), whereunmap_mapping_range() will no-op once unmap_vmas() has unlinked theVMA; however munmap() will not yet have invalidated the TLBs.Therefore unmap_mapping_range() will complete while there are still(stale) TLB entries for the specified range.Mitigate this by force flushing TLBs for VM_PFNMAP ranges.Signed-off-by: Peter Zijlstra (Intel) Acked-by: Will Deacon Signed-off-by: Linus Torvalds List of files: /linux-6.15/include/asm-generic/tlb.h

18ba064e - mmu_gather: Let there be one tlb_{start,end}_vma() implementation

Fri, 08 Jul 2022 07:18:05 +0000

mmu_gather: Let there be one tlb_{start,end}_vma() implementationNow that architectures are no longer allowed to overridetlb_{start,end}_vma() re-arrange code so that there is only oneimplementation for each of these functions.This much simplifies trying to figure out what they actually do.Signed-off-by: Peter Zijlstra (Intel) Acked-by: Will Deacon Signed-off-by: Linus Torvalds List of files: /linux-6.15/include/asm-generic/tlb.h

1e9fdf21 - mmu_gather: Remove per arch tlb_{start,end}_vma()

Fri, 08 Jul 2022 07:18:03 +0000

mmu_gather: Remove per arch tlb_{start,end}_vma()Scattered across the archs are 3 basic forms of tlb_{start,end}_vma().Provide two new MMU_GATHER_knobs to enumerate them and remove the perarch tlb_{start,end}_vma() implementations. - MMU_GATHER_NO_FLUSH_CACHE indicates the arch has flush_cache_range() but does *NOT* want to call it for each VMA. - MMU_GATHER_MERGE_VMAS indicates the arch wants to merge the invalidate across multiple VMAs if possible.With these it is possible to capture the three forms: 1) empty stubs; select MMU_GATHER_NO_FLUSH_CACHE and MMU_GATHER_MERGE_VMAS 2) start: flush_cache_range(), end: empty; select MMU_GATHER_MERGE_VMAS 3) start: flush_cache_range(), end: flush_tlb_range(); defaultObviously, if the architecture does not have flush_cache_range() thenit also doesn't need to select MMU_GATHER_NO_FLUSH_CACHE.Signed-off-by: Peter Zijlstra (Intel) Acked-by: Will Deacon Cc: David Miller Signed-off-by: Linus Torvalds List of files: /linux-6.15/include/asm-generic/tlb.h