en Copyright 2015 Java http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#2ff63af9 sys: Remove $FreeBSD$: one-line .h patternRemove /^\s*\*+\s*\$FreeBSD\$.*$\n/ List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Wed, 16 Aug 2023 17:54:18 +0000 Warner Losh <imp@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#9cb6ba29 vm: centralize VM_BATCHQUEUE_SIZE definitionRemove the platform-specific definitions of VM_BATCHQUEUE_SIZEfor amd64 and powerpc64, and instead treat all 64-bit platformsidentically. This has the effect of increasing the arm64and riscv VM_BATCHQUEUE_SIZE to match that of other platforms.Reviewed by: jhb, markjSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D37707 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Sat, 21 Jan 2023 19:26:25 +0000 Andrew Gallatin <gallatin@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#1cac76c9 vm: reduce lock contention when processing vm batchqueuesRather than waiting until the batchqueue is full to acquire the lock &process the queue, we now start trying to acquire the lock using trylockswhen the batchqueue is 1/2 full. This removes almost all contention on thevm pagequeue mutex for for our busy sendfile() based web workload.It also greadly reduces the amount of time a network driver ithreadremains blocked on a mutex, and eliminates some packet drops underheavy load.So that the system does not loose the benefit of processing largebatchqueues, I've doubled the size of the batchqueues. This way, whenthere is no contention, we process the same batch size as before.This has been run for several months on a busy Netflix server, as wellas on my personal desktop.Reviewed by: markjSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D37305 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Wed, 14 Dec 2022 19:34:07 +0000 Andrew Gallatin <gallatin@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#a69511db amd64: -m32 support for machine/vmparam.hInstall the i386 vmparam.h under /usr/include/i386 on amd64 and includewhen targeting i386.Reviewed by: jhb, imp List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Mon, 13 Jun 2022 17:35:39 +0000 Brooks Davis <brooks@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#90d4da62 amd64: provide PHYS_IN_DMAP() and VIRT_IN_DMAP()It is useful for quickly checking an address against the DMAP region.These definitions exist already on arm64 and riscv.Reviewed by: kib, markjMFC after: 3 daysSponsored by: Juniper Networks, Inc.Sponsored by: Klara, Inc.Differential Revision: https://reviews.freebsd.org/D32962 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Wed, 17 Nov 2021 15:29:02 +0000 Mitchell Horne <mhorne@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#89786088 amd64: Populate the KMSAN shadow maps and integrate with the VM- During boot, allocate PDP pages for the shadow maps. The region above KERNBASE is currently not shadowed.- Create a dummy shadow for the vm page array. For now, this array is not protected by the shadow map to help reduce kernel memory usage.- Grow shadows when growing the kernel map.- Increase the default kernel stack size when KMSAN is enabled. As with KASAN, sanitizer instrumentation appears to create stack frames large enough that the default value is not sufficient.- Disable UMA's use of the direct map when KMSAN is configured. KMSAN cannot validate the direct map.- Disable unmapped I/O when KMSAN configured.- Lower the limit on paging buffers when KMSAN is configured. Each buffer has a static MAXPHYS-sized allocation of KVA, which in turn eats 2*MAXPHYS of space in the shadow map.Reviewed by: alc, kibSponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D31295 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 10 Aug 2021 20:52:36 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#f95f780e amd64: Define KVA regions for KMSAN shadow mapsKMSAN requires two shadow maps, each one-to-one with the kernel map.Allocate regions of the kernels PML4 page for them. Add functions tocreate mappings in the shadow map regions, these will be used by theKMSAN runtime.Reviewed by: alc, kibSponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D31295 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 10 Aug 2021 20:25:39 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#e18380e3 amd64: do not assume that kernel is loaded at 2M physicalAllow any 2M aligned contiguous location below 4G for the stagingarea location. It should still be mapped by loader at KERNBASE.The assumption kernel makes about loader->kernel handoff with regard tothe MMU programming are explicitly listed at the beginning of hammer_time(),where kernphys is calculated. Now kernphys is the variable instead ofsymbol designating the physical address.Reviewed by: markjSponsored by: The FreeBSD FoundationMFC after: 1 weekDifferential revision: https://reviews.freebsd.org/D31121 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Sat, 10 Jul 2021 19:48:02 +0000 Konstantin Belousov <kib@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#6faf45b3 amd64: Implement a KASAN shadow mapThe idea behind KASAN is to use a region of memory to track the validityof buffers in the kernel map. This region is the shadow map. Thecompiler inserts calls to the KASAN runtime for every emitted loadand store, and the runtime uses the shadow map to decide whether theaccess is valid. Various kernel allocators call kasan_mark() to updatethe shadow map.Since the shadow map tracks only accesses to the kernel map, accesses toother kernel maps are not validated by KASAN. UMA_MD_SMALL_ALLOC isdisabled when KASAN is configured to reduce usage of the direct map.Currently we have no mechanism to completely eliminate uses of thedirect map, so KASAN's coverage is not comprehensive.The shadow map uses one byte per eight bytes in the kernel map. Inpmap_bootstrap() we create an initial set of page tables for the kerneland preloaded data.When pmap_growkernel() is called, we call kasan_shadow_map() to extendthe shadow map. kasan_shadow_map() uses pmap_kasan_enter() to allocatememory for the shadow region and map it.Reviewed by: kibMFC after: 2 weeksSponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D29417 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 13 Apr 2021 20:30:05 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#fd2ef8ef Unobfuscate "KERNLOAD" parameter on amd64. This change lines-up amd64 with thei386 and the rest of supported architectures by defining KERNLOAD in thevmparam.h and getting rid of magic constant in the linker script, which albeitdocumented via comment but isn't programmatically accessible at a compile time.Use KERNLOAD to eliminate another (matching) magic constant 100 lines downinside unremarkable TU "copy.c" 3 levels deep in the EFI loader tree.Reviewed by: markjApproved by: markjMFC after: 1 monthDifferential Revision: https://reviews.freebsd.org/D27355 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Wed, 25 Nov 2020 23:19:01 +0000 Maxim Sobolev <sobomax@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#78257765 Add a vmparam.h constant indicating pmap support for large pages.Enable SHM_LARGEPAGE support on arm64.Reviewed by: alc, kibSponsored by: Juniper Networks, Inc., Klara, Inc.Differential Revision: https://reviews.freebsd.org/D26467 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Wed, 23 Sep 2020 19:34:21 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#ab041f71 Move vm_page_dump bitset array definition to MI codeThese definitions were repeated by all architectures, with smallvariations. Consolidate the common definitons in machineindependent code and use bitset(9) macros for manipulation. Manyopportunities for deduplication remain in the machine dependentminidump logic. The only intended functional change is increasingthe bit index type to vm_pindex_t, allowing the indexing of pageswith address of 8 TiB and greater.Reviewed by: kib, markjApproved by: scottl (implicit)MFC after: 1 weekSponsored by: Ampere Computing, Inc.Differential Revision: https://reviews.freebsd.org/D26129 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Mon, 21 Sep 2020 22:20:37 +0000 D Scott Phillips <scottph@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#543769bf amd64: clean up empty lines in .c and .h files List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 01 Sep 2020 21:16:54 +0000 Mateusz Guzik <mjg@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#9ce875d9 amd64 pmap: LA57 AKA 5-level pagingSince LA57 was moved to the main SDM document with revision 072, itseems that we should have a support for it, and silicons are coming.This patch makes pmap support both LA48 and LA57 hardware. Theselection of page table level is done at startup, kernel alwaysreceives control from loader with 4-level paging. It is not clear howUEFI spec would adapt LA57, for instance it could hand out control inLA57 mode sometimes.To switch from LA48 to LA57 requires turning off long mode, requestingLA57 in CR4, then re-entering long mode. This is somewhat delicateand done in pmap_bootstrap_la57(). AP startup in LA57 mode is mucheasier, we only need to toggle a bit in CR4 and load right value in CR3.I decided to not change kernel map for now. Single PML5 entry iscreated that points to the existing kernel_pml4 (KML4Phys) page, and apml5 entry to create our recursive mapping for vtopte()/vtopde().This decision is motivated by the fact that we cannot overcommit forKVA, so large space there is unusable until machines start providingwider physical memory addressing. Another reason is that I do notwant to break our fragile autotuning, so the KVA expansion is notincluded into this first step. Nice side effect is that minidumps arecompatible.On the other hand, (very) large address space is definitelyimmediately useful for some userspace applications.For userspace, numbering of pte entries (or page table pages) isalways done for 5-level structures even if we operate in 4-level mode.The pmap_is_la57() function is added to report the mode of thespecified pmap, this is done not to allow simultaneous 4-/5-levels(which is not allowed by hw), but to accomodate for EPT which hasseparate level control and in principle might not allow 5-leve EPTdespite x86 paging supports it. Anyway, it does not seems critical tohave 5-level EPT support now.Tested by: pho (LA48 hardware)Reviewed by: alcSponsored by: The FreeBSD FoundationDifferential revision: https://reviews.freebsd.org/D25273 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Sun, 23 Aug 2020 20:19:04 +0000 Konstantin Belousov <kib@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#f97bf604 Fix some nits in pmap_page_array_startup().- Use ptoa() instead of the archaic ctob().- Use pagezero() to zero a PDP page.- Remove PA_MIN_ADDRESS, orphaned by r351742.- Remove unneeded parens and an unnecessary control flow statement.Reported by: alcReviewed by: alc, kibSponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D21495 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 03 Sep 2019 22:26:01 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#9d75f0dc Map the vm_page array into KVA on amd64.r351198 allows the kernel to use domain-local memory to back the vm_pagearray (up to 2MB boundaries) and reserves a separate PML4 entry for thatpurpose. One consequence of that change is that the vm_page array is nolonger present in minidumps, which only adds pages mapped aboveVM_MIN_KERNEL_ADDRESS.To avoid the friction caused by having kernel data structures mappedbelow VM_MIN_KERNEL_ADDRESS, map the vm_page array starting atVM_MIN_KERNEL_ADDRESS instead of using a dedicated PML4 entry.Reviewed by: kibDiscussed with: jeffSponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D21491 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 03 Sep 2019 13:18:51 +0000 Mark Johnston <markj@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#3e5e1b51 Allocate amd64's page array using pages and page directory pages from theNUMA domain that the pages describe. Patch original from gallatin.Reviewed by: kibTested by: phoSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D21252 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Sun, 18 Aug 2019 23:07:56 +0000 Jeff Roberson <jeff@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#2fd0c8e7 Provide pmap_large_map() KPI on amd64.The KPI allows to map very large contigous physical memory regionsinto KVA, which are not covered by DMAP.I see both with QEMU and with some real hardware started shipping, theregions for NVDIMMs might be very far apart from the normal RAM, andwe expect that at least initial users of NVDIMM could install verylarge amount of such memory. IMO it is not reasonable to extend DMAPto cover that far-away regions both because it could overflow existing4T window for DMAP in KVA, and because it costs in page table pagesallocations, for gap and for possibly unused NV RAM.Also, KPI provides some special functionality for fast cache flushingbased on the knowledge of the NVRAM mapping use.Reviewed by: alc, markjSponsored by: The FreeBSD FoundationApproved by: re (gjb)MFC after: 1 weekDifferential revision: https://reviews.freebsd.org/D17070 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 16 Oct 2018 17:28:10 +0000 Konstantin Belousov <kib@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#67d33338 Rename VM_FREELIST_ISADMA to VM_FREELIST_LOWMEM.There's no differene between VM_FREELIST_ISADMA and VM_FREELIST_LOWMEMexcept for the default boundary (16MB on x86 and 256MB on MIPS, butthey are otherwise the same). We don't need both for any system wesupport (there were some really old ARC systems that did have ISA/EISAbus, but we never ran on them and they are too old to ever growsupport for).Differential Review: https://reviews.freebsd.org/D16290 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Fri, 27 Jul 2018 18:34:20 +0000 Warner Losh <imp@FreeBSD.org> http://172.16.0.5:8080/history/freebsd-14.2/sys/amd64/include/vmparam.h#5cd29d0f Improve VM page queue scalability.Currently both the page lock and a page queue lock must be held inorder to enqueue, dequeue or requeue a page in a given page queue.The queue locks are a scalability bottleneck in many workloads. Thischange reduces page queue lock contention by batching queue operations.To detangle the page and page queue locks, per-CPU batch queues areused to reference pages with pending queue operations. The requestedoperation is encoded in the page's aflags field with the page lockheld, after which the page is enqueued for a deferred batch operation.Page queue scans are similarly optimized to minimize the amount ofwork performed with a page queue lock held.Reviewed by: kib, jeff (previous versions)Tested by: phoSponsored by: Dell EMC IsilonDifferential Revision: https://reviews.freebsd.org/D14893 List of files: /freebsd-14.2/sys/amd64/include/vmparam.h Tue, 24 Apr 2018 21:15:54 +0000 Mark Johnston <markj@FreeBSD.org>