Convert vm_page_alloc() callers to use vm_page_alloc_noobj().Remove page zeroing code from consumers and stop specifyingVM_ALLOC_NOOBJ. In a few places, also convert an allocation loop tosimply
Convert vm_page_alloc() callers to use vm_page_alloc_noobj().Remove page zeroing code from consumers and stop specifyingVM_ALLOC_NOOBJ. In a few places, also convert an allocation loop tosimply use VM_ALLOC_WAITOK.Similarly, convert vm_page_alloc_domain() callers.Note that callers are now responsible for assigning the pindex.Reviewed by: alc, hselasky, kibSponsored by: The FreeBSD Foundation(cherry picked from commit a4667e09e6520dc2c4b0b988051f060fed695a91)
show more ...
Convert allocations of the phys pager to vm_pager_allocate().Future changes would require additional initialization of OBJT_PHYSobjects, and vm_object_allocate() is not suitable for it.Reviewed
Convert allocations of the phys pager to vm_pager_allocate().Future changes would require additional initialization of OBJT_PHYSobjects, and vm_object_allocate() is not suitable for it.Reviewed by: markjTested by: phoSponsored by: The FreeBSD FoundationMFC after: 1 weekDifferential revision: https://reviews.freebsd.org/D24652
dev/xen: clean up empty lines in .c and .h files
Make page busy state deterministic on free. Pages must be xbusy whenremoved from objects including calls to free. Pages must not be xbusywhen freed and not on an object. Strengthen assertions to
Make page busy state deterministic on free. Pages must be xbusy whenremoved from objects including calls to free. Pages must not be xbusywhen freed and not on an object. Strengthen assertions to match theseexpectations. In practice very little code had to change busy handlingto meet these rules but we can now make stronger guarantees to busyholders and avoid conditionally dropping busy in free.Refine vm_page_remove() and vm_page_replace() semantics now that we havestronger guarantees about busy state. This removes redundant andpotentially problematic code that has proliferated.Discussed with: markjReviewed by: kibDifferential Revision: https://reviews.freebsd.org/D22822
(4/6) Protect page valid with the busy lock.Atomics are used for page busy and valid state when the shared busy isheld. The details of the locking protocol and valid and dirtysynchronization are
(4/6) Protect page valid with the busy lock.Atomics are used for page busy and valid state when the shared busy isheld. The details of the locking protocol and valid and dirtysynchronization are in the updated vm_page.h comments.Reviewed by: kib, markjTested by: phoSponsored by: Netflix, IntelDifferential Revision: https://reviews.freebsd.org/D21594
(1/6) Replace busy checks with acquires where it is trival to do so.This is the first in a series of patches that promotes the page busy fieldto a first class lock that no longer requires the obje
(1/6) Replace busy checks with acquires where it is trival to do so.This is the first in a series of patches that promotes the page busy fieldto a first class lock that no longer requires the object lock forconsistency.Reviewed by: kib, markjTested by: phoSponsored by: Netflix, IntelDifferential Revision: https://reviews.freebsd.org/D21548
Replace redundant code with a few new vm_page_grab facilities: - VM_ALLOC_NOCREAT will grab without creating a page. - vm_page_grab_valid() will grab and page in if necessary. - vm_page_busy_acqui
Replace redundant code with a few new vm_page_grab facilities: - VM_ALLOC_NOCREAT will grab without creating a page. - vm_page_grab_valid() will grab and page in if necessary. - vm_page_busy_acquire() automates some busy acquire loops.Discussed with: alc, kib, markjTested by: pho (part of larger branch)Sponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D21546
Change synchonization rules for vm_page reference counting.There are several mechanisms by which a vm_page reference is held,preventing the page from being freed back to the page allocator. Inpa
Change synchonization rules for vm_page reference counting.There are several mechanisms by which a vm_page reference is held,preventing the page from being freed back to the page allocator. Inparticular, holding the page's object lock is sufficient to prevent thepage from being freed; holding the busy lock or a wiring is sufficent aswell. These references are protected by the page lock, which musttherefore be acquired for many per-page operations. This results infalse sharing since the page locks are external to the vm_pagestructures themselves and each lock protects multiple structures.Transition to using an atomically updated per-page reference counter.The object's reference is counted using a flag bit in the counter. Asecond flag bit is used to atomically block new references viapmap_extract_and_hold() while removing managed mappings of a page.Thus, the reference count of a page is guaranteed not to increase if thepage is unbusied, unmapped, and the object's write lock is held. Asa consequence of this, the page lock no longer protects a page'sidentity; operations which move pages between objects are nowsynchronized solely by the objects' locks.The vm_page_wire() and vm_page_unwire() KPIs are changed. The formerrequires that either the object lock or the busy lock is held. Thelatter no longer has a return value and may free the page if it releasesthe last reference to that page. vm_page_unwire_noq() behaves the sameas before; the caller is responsible for checking its return value andfreeing or enqueuing the page as appropriate. vm_page_wire_mapped() isintroduced for use in pmap_extract_and_hold(). It fails if the page isconcurrently being unmapped, typically triggering a fallback to thefault handler. vm_page_wire() no longer requires the page lock andvm_page_unwire() now internally acquires the page lock when releasingthe last wiring of a page (since the page lock still protects a page'squeue state). In particular, synchronization details are no longerleaked into the caller.The change excises the page lock from several frequently executed codepaths. In particular, vm_object_terminate() no longer bounces betweenpage locks as it releases an object's pages, and direct I/O andsendfile(SF_NOCACHE) completions no longer require the page lock. Inthese latter cases we now get linear scalability in the common scenariowhere different threads are operating on different files.__FreeBSD_version is bumped. The DRM ports have been updated toaccomodate the KPI changes.Reviewed by: jeff (earlier version)Tested by: gallatin (earlier version), phoSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D20486
Replace uses of vm_page_unwire(m, PQ_NONE) with vm_page_unwire_noq(m).These calls are not the same in general: the former will dequeue thepage if it is enqueued, while the latter will just leave i
Replace uses of vm_page_unwire(m, PQ_NONE) with vm_page_unwire_noq(m).These calls are not the same in general: the former will dequeue thepage if it is enqueued, while the latter will just leave it alone. But,all existing uses of the former apply to unmanaged pages, which arenever enqueued in the first place. No functional change intended.Reviewed by: kibMFC after: 1 weekSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D20470
Change the vm_ooffset_t type to unsigned.The type represents byte offset in the vm_object_t data space, whichdoes not span negative offsets in FreeBSD VM. The change matches byteoffset signess w
Change the vm_ooffset_t type to unsigned.The type represents byte offset in the vm_object_t data space, whichdoes not span negative offsets in FreeBSD VM. The change matches byteoffset signess with the unsignedness of the vm_pindex_t whichrepresents the type of the page indexes in the objects.This allows to remove the UOFF_TO_IDX() macro which was used when wehave to forcibly interpret the type as unsigned anyway. Also it fixesa lot of implicit bugs in the device drivers d_mmap methods.Reviewed by: alc, markj (previous version)Tested by: phoMFC after: 2 weeksSponsored by: The FreeBSD Foundation
xen: fix gntdevCurrent interface to the gntdev in FreeBSD is wrong, and mostly workedout of luck before the PTI FreeBSD fixes, when kernel and user-spacewhere sharing the same page tables.On Fr
xen: fix gntdevCurrent interface to the gntdev in FreeBSD is wrong, and mostly workedout of luck before the PTI FreeBSD fixes, when kernel and user-spacewhere sharing the same page tables.On FreeBSD ioctls have the size of the passed struct encoded in theioctl number, because the generic ioctl handler in the OS takes careof copying the data from user-space to kernel space, and then callsthe device specific ioctl handler. Thus using ioctl structs withvariable sizes is not possible.The fix is to turn the array of structs at the end ofioctl_gntdev_alloc_gref and ioctl_gntdev_map_grant_ref into pointers,that can be properly accessed from the kernel gntdev driver using thecopyin/copyout functions. Note that this is exactly how it's done forthe privcmd driver.Sponsored by: Citrix Systems R&D
Remove unneeded include of vm_phys.h.
xen/gntdev: prevent unsynchronized accesses to the map entryvm_map_lookup_done should only be called when the gntdev has finished poking atthe entry.Reported by: alcReviewed by: alcMFC after:
xen/gntdev: prevent unsynchronized accesses to the map entryvm_map_lookup_done should only be called when the gntdev has finished poking atthe entry.Reported by: alcReviewed by: alcMFC after: 1 weekSponsored by: Citrix Systems R&D
xen/gndev: use UOFF_TO_IDX instead of OFF_TO_IDXThe Xen grant table device treats the mmap offset parameter as an unsignedtype, and as so it must use the newly introduced UOFF_TO_IDX.Sponsored b
xen/gndev: use UOFF_TO_IDX instead of OFF_TO_IDXThe Xen grant table device treats the mmap offset parameter as an unsignedtype, and as so it must use the newly introduced UOFF_TO_IDX.Sponsored by: Citrix Systems R&DMFC after: 2 weeksX-MFC-with: r313690
xen: add a grant-table user-space deviceA grant-table user-space device will allow user-space applications to mapand share grants (Xen way to share memory) among Xen domains. This granttable user
xen: add a grant-table user-space deviceA grant-table user-space device will allow user-space applications to mapand share grants (Xen way to share memory) among Xen domains. This granttable user-space device has been tested with the QEMU Qdisk Xen backed.Submitted by: jaggiReviewed by: roygerDifferential review: https://reviews.freebsd.org/D7293