sys/bitset.h: reduce visibility of BIT_* macros(cherry picked from commit 5e04571cf3cf4024be926976a6abf19626df30be)
malloc: move malloc_type_internal into malloc_typeAccording to code comments the original motivation was to allow formalloc_type_internal changes without ABI breakage. This can be triviallyaccomp
malloc: move malloc_type_internal into malloc_typeAccording to code comments the original motivation was to allow formalloc_type_internal changes without ABI breakage. This can be triviallyaccomplished by providing spare fields and versioning the struct, asimplemented in the patch below.The upshots are one less memory indirection on each alloc and disappearanceof mt_zone.Reviewed by: markjDifferential Revision: https://reviews.freebsd.org/D27104
show more ...
malloc: export kernel zones instead of relying on them being power-of-2Reviewed by: markj (previous version)Differential Revision: https://reviews.freebsd.org/D27026
memstat_kvm_uma: fix reading of uma_zone_domain structuresCoverity flagged the scaling by sizeof(uzd). That is the typeof the pointer, so the scaling was already done by pointer arithmetic.Howev
memstat_kvm_uma: fix reading of uma_zone_domain structuresCoverity flagged the scaling by sizeof(uzd). That is the typeof the pointer, so the scaling was already done by pointer arithmetic.However, this was also passing a stack frame pointer to kvm_read,so it was doubly wrong.Move ZDOM_GET into the !_KERNEL section and use it in libmemstat.Reported by: CoverityReviewed by: markjMFC after: 2 weeksSponsored by: Dell EMC IsilonDifferential Revision: https://reviews.freebsd.org/D26213
Use per-domain locks for the bucket cache.This gives much better concurrency when there are a large number ofcores per-domain and multiple domains. Avoid taking the lock entirelyif it will not b
Use per-domain locks for the bucket cache.This gives much better concurrency when there are a large number ofcores per-domain and multiple domains. Avoid taking the lock entirelyif it will not be productive. ROUNDROBIN domains will have mixedmemory in each domain and will load balance to all domains.While here refactor the zone/domain separation and bucket limits tosimplify callers.Reviewed by: markjDifferential Revision: https://reviews.freebsd.org/D23673
libmemstat: Catch up with r357776.Reported by: O. Hartmann <[email protected]>
Fix libmemstat_uma build after r357485.Submitted by: cy
Use per-domain keg locks. This provides both a lock and separate spaceaccounting for each NUMA domain. Independent keg domain locks are importantwith cross-domain frees. Hashed zones are non-num
Use per-domain keg locks. This provides both a lock and separate spaceaccounting for each NUMA domain. Independent keg domain locks are importantwith cross-domain frees. Hashed zones are non-numa and use a single keglock to protect the hash table.Reviewed by: markj, rlibbyDifferential Revision: https://reviews.freebsd.org/D22829
Optimize fast path allocations by storing bucket headers in the per-cpucache area. This allows us to check on bucket space for all per-cpubuckets with a single cacheline access and fewer branches.
Optimize fast path allocations by storing bucket headers in the per-cpucache area. This allows us to check on bucket space for all per-cpubuckets with a single cacheline access and fewer branches.Reviewed by: markj, rlibbyDifferential Revision: https://reviews.freebsd.org/D22825
Revert r355706 & r355710The quick fix didn't work. I'll sort it out tomorrow.Revert r355710: "libmemstat: unbreak build"Revert r355706: "uma dbg: flexible size for slab debug bitset too"
libmemstat: unbreak buildr355706 added an instance of offsetof() to the UMA private kernel headerfile uma_int.h. Userspace memstat_uma.c includes that header, andchokes on offsetof() because app
libmemstat: unbreak buildr355706 added an instance of offsetof() to the UMA private kernel headerfile uma_int.h. Userspace memstat_uma.c includes that header, andchokes on offsetof() because apparently the definition in sys/types.h isifdef _KERNEL. Now, include sys/stddef.h which has an identicaldefinition.Pointyhat to: rlibbySponsored by: Dell EMC Isilon
Update Makefile.depend filesUpdate a bunch of Makefile.depend files asa result of adding Makefile.depend.options filesReviewed by: bdreweryMFC after: 1 weekSponsored by: Juniper NetworksDi
Update Makefile.depend filesUpdate a bunch of Makefile.depend files asa result of adding Makefile.depend.options filesReviewed by: bdreweryMFC after: 1 weekSponsored by: Juniper NetworksDifferential Revision: https://reviews.freebsd.org/D22494
pkgbase: Create a FreeBSD-utilities package and make it the default oneThe default package use to be FreeBSD-runtime but it should only containbinaries and libs enough to boot to single user and r
pkgbase: Create a FreeBSD-utilities package and make it the default oneThe default package use to be FreeBSD-runtime but it should only containbinaries and libs enough to boot to single user and repair the system, itis also very handy to have a package that can be tranform to a small mfsroot.So create a new package named FreeBSD-utilities and make it the default one.Also move a few binaries and lib into this package when it make sense.Reviewed by: bapt, gjbDifferential Revision: https://reviews.freebsd.org/D21506
Extend uma_reclaim() to permit different reclamation targets.The page daemon periodically invokes uma_reclaim() to reclaim cacheditems from each zone when the system is under memory pressure. Thi
Extend uma_reclaim() to permit different reclamation targets.The page daemon periodically invokes uma_reclaim() to reclaim cacheditems from each zone when the system is under memory pressure. Thisis important since the size of these caches is unbounded by default.However it also results in bursts of high latency when allocating fromheavily used zones as threads miss in the per-CPU caches and mustaccess the keg in order to allocate new items.With r340405 we maintain an estimate of each zone's usage of its(per-NUMA domain) cache of full buckets. Start making use of thisestimate to avoid reclaiming the entire cache when under memorypressure. In particular, introduce TRIM, DRAIN and DRAIN_CPUverbs for uma_reclaim() and uma_zone_reclaim(). When trimming, onlyitems in excess of the estimate are reclaimed. Draining a zonereclaims all of the cached full buckets (the previous behaviour ofuma_reclaim()), and may further drain the per-CPU caches in extremecases.Now, when under memory pressure, the page daemon will trim zonesrather than draining them. As a result, heavily used zones do not incurbursts of bucket cache misses following reclamation, but large, unusedcaches will be reclaimed as before.Reviewed by: jeffTested by: pho (an earlier version)MFC after: 2 monthsSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D16667
Add two new kernel options to control memory locality on NUMA hardware. - UMA_XDOMAIN enables an additional per-cpu bucket for freed memory that was freed on a different domain from where it was
Add two new kernel options to control memory locality on NUMA hardware. - UMA_XDOMAIN enables an additional per-cpu bucket for freed memory that was freed on a different domain from where it was allocated. This is only used for UMA_ZONE_NUMA (first-touch) zones. - UMA_FIRSTTOUCH sets the default UMA policy to be first-touch for all zones. This tries to maintain locality for kernel memory.Reviewed by: gallatin, alc, kibTested by: pho, gallatinSponsored by: NetflixDifferential Revision: https://reviews.freebsd.org/D20929
The KVM code also needs a fix similar to r344269.Reported by: pho
With r343051 UMA switched from atomic counts to counter(9) and now kernelreports snap counts of how much a zone alloced and how much it freed. Itmay happen that snap values doesn't match, e.g allo
With r343051 UMA switched from atomic counts to counter(9) and now kernelreports snap counts of how much a zone alloced and how much it freed. Itmay happen that snap values doesn't match, e.g alloced - freed < 0.Workaround that in memstat library.Reported by: pho
This was missed in r343051: make uz_allocs, uz_frees and uz_fails counter(9).
o Move zone limit from keg level up to zone level. This means that now two zones sharing a keg may have different limits. Now this is going to work: zone = uma_zcreate(); uma_zone_set_max(zo
o Move zone limit from keg level up to zone level. This means that now two zones sharing a keg may have different limits. Now this is going to work: zone = uma_zcreate(); uma_zone_set_max(zone, limit); zone2 = uma_zsecond_create(zone); uma_zone_set_max(zone2, limit2); Kegs no longer have uk_maxpages field, but zones have uz_items. When set, it may be rounded up to minimum possible CPU bucket cache size. For small limits bucket cache can also be reconfigured to be smaller. Counter uz_items is updated whenever items transition from keg to a bucket cache or directly to a consumer. If zone has uz_maxitems set and it is reached, then we are going to sleep.o Since new limits don't play well with multi-keg zones, remove them. The idea of multi-keg zones was introduced exactly 10 years ago, and never have had a practical usage. In discussion with Jeff we came to a wild agreement that if we ever want to reintroduce the idea of a smart allocator that would be able to choose between two (or more) totally different backing stores, that choice should be made one level higher than UMA, e.g. in malloc(9) or in mget(), or whatever and choice should be controlled by the caller.o Sleeping code is improved to account number of sleepers and wake them one by one, to avoid thundering herd problem.o Flag UMA_ZONE_NOBUCKETCACHE removed, instead uma_zone_set_maxcache() KPI added. Having no bucket cache basically means setting maxcache to 0.o Now with many fields added and many removed (no multi-keg zones!) make sure that struct uma_zone is perfectly aligned.Reviewed by: markj, jeffTested by: phoDifferential Revision: https://reviews.freebsd.org/D17773
libmemstat: adjust for per-cpu stats after r338899Reported by: yuripvReviewed by: kib, markjApproved by: re (gjb)Sponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freeb
libmemstat: adjust for per-cpu stats after r338899Reported by: yuripvReviewed by: kib, markjApproved by: re (gjb)Sponsored by: The FreeBSD FoundationDifferential Revision: https://reviews.freebsd.org/D17490
Reduce <sys/queue.h> pollution.While <sys/sysctl.h> includes <sys/queue.h> unconditionally, it is onlyactually used in code which is conditional on _KERNEL. Make the #includeitself conditional a
Reduce <sys/queue.h> pollution.While <sys/sysctl.h> includes <sys/queue.h> unconditionally, it is onlyactually used in code which is conditional on _KERNEL. Make the #includeitself conditional as well, and fix userland code that uses <sys/queue.h>for other purposes but relied on <sys/sysctl.h> to bring it in.MFC after: 1 week
Implement NUMA support in uma(9) and malloc(9). Allocations from specificdomains can be done by the _domain() API variants. UMA also supports afirst-touch policy via the NUMA zone flag.The slab
Implement NUMA support in uma(9) and malloc(9). Allocations from specificdomains can be done by the _domain() API variants. UMA also supports afirst-touch policy via the NUMA zone flag.The slab layer is now segregated by VM domains and is precise. It handlesiteration for round-robin directly. The per-cpu cache layer remainsa mix of domains according to where memory is allocated and freed. Wellbehaved clients can achieve perfect locality with no performance penalty.The direct domain allocation functions have to visit the slab layer andso require per-zone locks which come at some expense.Reviewed by: Attilio (a slightly older version)Tested by: phoSponsored by: Netflix, Dell/EMC Isilon
lib: further adoption of SPDX licensing ID tags.Mainly focus on files that use BSD 2-Clause license, however the tool Iwas using mis-identified many licenses so this was mostly a manual - errorpr
lib: further adoption of SPDX licensing ID tags.Mainly focus on files that use BSD 2-Clause license, however the tool Iwas using mis-identified many licenses so this was mostly a manual - errorprone - task.The Software Package Data Exchange (SPDX) group provides a specificationto make it easier for automated tools to detect and summarize well knownopensource licenses. We are gradually adopting the specification, notingthat the tags are considered only advisory and do not, in any way,superceed or replace the license texts.
DIRDEPS_BUILD: Update dependencies.Sponsored by: Dell EMC Isilon
Fix buildworld for powerpc.vmpage requires struct pmap to exist and contain a pm_stats field. As ofr308817, either AIM or BOOKE is required to be set in order to get theirrespective pmap structs
Fix buildworld for powerpc.vmpage requires struct pmap to exist and contain a pm_stats field. As ofr308817, either AIM or BOOKE is required to be set in order to get theirrespective pmap structs. Rather than expose them both, or try to unify themunnecessarily, add a third option which contains only a pm_stats field, andchange the two existing pmap structures to place the common fields at thebeginning of the struct. This actually fixes the stats collection by libkvm onAIM hardware, because before it was accessing a possibly different offset, whichwould cause it to read garbage.Bump __FreeBSD_version to denote this ABI change, so that ports which depend onlibkvm can be rebuilt.
1234