slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-ch

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-chunk
items.

also fixes a bug where the lru limit wasn't being utilized for HOT_LRU

also some cleanup from previous commits.

show more ...

cache_memlimit command for tuning runtime maxbytes

Allows dynamically increasing the memory limit of a running system, if memory
isn't being preallocated.

If `-o modern` is in use, can also dynamic

cache_memlimit command for tuning runtime maxbytes

Allows dynamically increasing the memory limit of a running system, if memory
isn't being preallocated.

If `-o modern` is in use, can also dynamically lower memory usage. pages are
free()'ed back to the OS via the slab rebalancer as memory is freed up. Does
not guarantee the OS will actually give the memory back for other applications
to use, that depends on how the OS handles memory.

show more ...

allow manually specifying slab class sizes

"-o slab_sizes=100-200-300-400-500" will create 5 slab classes of those
specified sizes, with the final class being item_max_size.

Using the new online st

allow manually specifying slab class sizes

"-o slab_sizes=100-200-300-400-500" will create 5 slab classes of those
specified sizes, with the final class being item_max_size.

Using the new online stats sizes command, it's possible to determine if the
typical factoral slab class growth rate doesn't align well with how items are
stored.

This is dangerous unless you really know what you're doing. If your items have
an exact or very predictable size this makes a lot of sense. If they do not,
the defaults are safer.

show more ...

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of memory.

show more ...

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destinatio

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destination of "0", which is a global
page pool. Pages can be re-assigned out of that pool during allocation.

Combined with item rescuing from the previous patch, we can safely shuffle
pages back to the reassignment pool as chunks free up naturally. This should
be a safe default going forward. Users should be able to decide to free or
move pages based on eviction pressure as well. This is coming up in another
commit.

This also fixes a calculation of the NOEXP LRU size, and completely removes
the old slab automover thread. Slab automove decisions will now be part of the
lru maintainer thread.

show more ...

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also remove

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also removes the condition wakeup for the background thread. Instead runs on a
timer, and meters its aggressiveness by how much shuffling is going on.

Also fixes a segfault in lru_pull_tail(), was unlinking `it` instead of
`search`.

show more ...

first pass at LRU maintainer thread

The basics work, but tests still do not pass.

A background thread wakes up once per second, or when signaled. It is signaled
if a slab class gets an allocation r

first pass at LRU maintainer thread

The basics work, but tests still do not pass.

A background thread wakes up once per second, or when signaled. It is signaled
if a slab class gets an allocation request and has fewer than N chunks free.

The background thread shuffles LRU's: HOT, WARM, COLD. HOT is where new items
exist. HOT and WARM flow into COLD. Active items in COLD flow back to WARM.
Evictions are pulled from COLD.

item_update's no longer do anything (and need to be fixed to tick it->time).
Items are reshuffled within or around LRU's as they reach the bottom.

Ratios of HOT/WARM memory are hardcoded, as are the low/high watermarks.
Thread is not fast enough right now, sets cannot block on it.

show more ...

item locks now lock hash table buckets

expansion requires switching to a global lock temporarily, so all buckets have
a covered read lock.

slab rebalancer is paused during hash table expansion.

in

item locks now lock hash table buckets

expansion requires switching to a global lock temporarily, so all buckets have
a covered read lock.

slab rebalancer is paused during hash table expansion.

internal item "trylocks" are always issued, and tracked as the hash power
variable can change out from under it.

show more ...

remove end_page_ptr business from slabs

slab memory assignment used to lazily split a new page into chunks as memory
was requested. now it doesn't, so drop all the related code.

Cuts the memory ass

remove end_page_ptr business from slabs

slab memory assignment used to lazily split a new page into chunks as memory
was requested. now it doesn't, so drop all the related code.

Cuts the memory assignment hotpath a tiny bit, so that's exciting.

show more ...

no same-class reassignment, better errors

Add human parseable strings to the errors for slabs ressign. Also prevent
reassigning memory to the same source and destination.

slab reassignment

Adds a "slabs reassign src dst" manual command, and a thread to safely process
slab moves in the background.

- slab freelist is now a linked list, reusing the item structure
- is

slab reassignment

Adds a "slabs reassign src dst" manual command, and a thread to safely process
slab moves in the background.

- slab freelist is now a linked list, reusing the item structure
- is -o slab_reassign is enabled, an extra background thread is started
- thread attempts to safely free up items when it's been told to move a page
from one slab to another.

-o slab_automove is stubbed.

There are some limitations. Most notable is that you cannot repeatedly move
pages around without first having items use up the memory. Slabs with newly
assigned memory work off of a pointer, handing out chunks individually. We
would need to change that to quickly split chunks for all newly assigned pages
into that slabs freelist.

Further testing is required to ensure such is possible without impacting
performance.

show more ...

Issue 163: Buggy mem_requested values

remove old unfinished slab reassignment code

Old code was unfinished, had no test coverage, and not quite what we'll end up
with in the future.

Slab reassignment will happen in earnest soon, but fo

remove old unfinished slab reassignment code

Old code was unfinished, had no test coverage, and not quite what we'll end up
with in the future.

Slab reassignment will happen in earnest soon, but for now we should stop
confusing users.

show more ...

Don't expose the protocol used to the client api of the stats

(dustin) I made some changes to the original growth code to pass in
the required size.

Refactor: move the slabs mutex (and all usage of it) into slabs.c

Fix for binary stats with subcommand.

The subcommand is not necessarily a null terminated string (in the
case of the binary protocol, it certainly isn't). The subcommand
would not be recognized due

Fix for binary stats with subcommand.

The subcommand is not necessarily a null terminated string (in the
case of the binary protocol, it certainly isn't). The subcommand
would not be recognized due to strcmp() failing.

Tested with both text and binary protocols.

show more ...

Fix for stats opaque issue pointed out at the hackathon and removed some wasteful function calls (more to come).

The slabber no longer needs a is_binary like flag for stats due to abstraction by the callback.

Changed the argument ordering for stats callback to something more common.

Support of slabber stats via the binary protocol.

Code cleaned by running devtools/clean-whitespace.pl script.

The slabber is now given a callback to perform binary stats.

Initial attempt of supporting stats in binary protocol.

Don't re-calculate the slab class id for slabs_free() either.
This + previous patch slightly reduce user CPU time, especially during heavy evictions.


git-svn-id: http://code.sixapart.com/svn/memcac

Don't re-calculate the slab class id for slabs_free() either.
This + previous patch slightly reduce user CPU time, especially during heavy evictions.


git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@739 b0b603af-a30f-0410-a34e-baf09ae79d0b

show more ...

Don't re-calculate the slab class id.
slabs_alloc() internally calls slabs_clsid(), so an eviction case would crawl the list of slab classes three times.


git-svn-id: http://code.sixapart.com/svn/me

Don't re-calculate the slab class id.
slabs_alloc() internally calls slabs_clsid(), so an eviction case would crawl the list of slab classes three times.


git-svn-id: http://code.sixapart.com/svn/memcached/trunk/server@738 b0b603af-a30f-0410-a34e-baf09ae79d0b

show more ...