fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, n

fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, new item allocation consume memory space
every time an item is set, that leads to slab OOM errors.

show more ...

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat for `stats settings`.

Finally, adds documentation and cleans up status outputs.

Could use some automated tests but not make or break for release.

show more ...

online hang-free "stats sizes" command.

"stats sizes" is one of the lack cache-hanging commands. With millions of
items it can hang for many seconds.

This commit changes the command to be dynamic.

online hang-free "stats sizes" command.

"stats sizes" is one of the lack cache-hanging commands. With millions of
items it can hang for many seconds.

This commit changes the command to be dynamic. A histogram is tracked as items
are linked and unlinked from the cache. The tracking is enabled or disabled at
runtime via "stats sizes_enable" and "stats sizes_disable".

This presently "works" but isn't accurate. Giving it some time to think over
before switching to requiring that CAS be enabled. Otherwise the values could
underflow if items are removed that existed before the sizes tracker is
enabled. This attempts to work around it by using it->time, which gets updated
on fetch, and is thus inaccurate.

show more ...

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as
unsigned up until a change made in 2007. Now treat it fully as unsigned and
print as unsigned.

show more ...

Implement get_expired stats

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destinatio

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destination of "0", which is a global
page pool. Pages can be re-assigned out of that pool during allocation.

Combined with item rescuing from the previous patch, we can safely shuffle
pages back to the reassignment pool as chunks free up naturally. This should
be a safe default going forward. Users should be able to decide to free or
move pages based on eviction pressure as well. This is coming up in another
commit.

This also fixes a calculation of the NOEXP LRU size, and completely removes
the old slab automover thread. Slab automove decisions will now be part of the
lru maintainer thread.

show more ...

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
s

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
slab class, copy the item over and replace it.

It's up to external systems to try to ensure free chunks are available before
moving a slab page. If there is no memory it will simply evict them as normal.

Also adds counters so we can finally tell how often these cases happen.

show more ...

ding-dong the cache_lock is dead.

LRU maintainer thread now fires LRU crawler

... if available. Very simple starter heuristic for how often to run the
crawler.

At this point, this patch series should have a significant impact on hi

LRU maintainer thread now fires LRU crawler

... if available. Very simple starter heuristic for how often to run the
crawler.

At this point, this patch series should have a significant impact on hit
ratio.

show more ...

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also remove

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also removes the condition wakeup for the background thread. Instead runs on a
timer, and meters its aggressiveness by how much shuffling is going on.

Also fixes a segfault in lru_pull_tail(), was unlinking `it` instead of
`search`.

show more ...

first pass at LRU maintainer thread

The basics work, but tests still do not pass.

A background thread wakes up once per second, or when signaled. It is signaled
if a slab class gets an allocation r

first pass at LRU maintainer thread

The basics work, but tests still do not pass.

A background thread wakes up once per second, or when signaled. It is signaled
if a slab class gets an allocation request and has fewer than N chunks free.

The background thread shuffles LRU's: HOT, WARM, COLD. HOT is where new items
exist. HOT and WARM flow into COLD. Active items in COLD flow back to WARM.
Evictions are pulled from COLD.

item_update's no longer do anything (and need to be fixed to tick it->time).
Items are reshuffled within or around LRU's as they reach the bottom.

Ratios of HOT/WARM memory are hardcoded, as are the low/high watermarks.
Thread is not fast enough right now, sets cannot block on it.

show more ...

Beginning work for LRU rework

Primarily splitting cache_lock into a lock-per LRU, and making the
it->slab_clsid lookup indirect. cache_lock is now more or less gone.

Stats are still wrong. they nee

Beginning work for LRU rework

Primarily splitting cache_lock into a lock-per LRU, and making the
it->slab_clsid lookup indirect. cache_lock is now more or less gone.

Stats are still wrong. they need to internally summarize over each
sub-class.

show more ...

flush_all was not thread safe.

Unfortunately if you disable CAS, all items set in the same second as a
flush_all will immediately expire. This is the old (2006ish) behavior.

However, if CAS is enab

flush_all was not thread safe.

Unfortunately if you disable CAS, all items set in the same second as a
flush_all will immediately expire. This is the old (2006ish) behavior.

However, if CAS is enabled (as is the default), it will still be more or less
exact.

The locking issue is that if the LRU lock is held, you may not be able to
modify an item if the item lock is also held. This means that some items may
not be flushed if locking is done correctly.

In the current code, it could lead to corruption as an item could be locked
and in use while the expunging is happening.

show more ...

Pause all threads while swapping hash table.

We used to hold a global lock around all modifications to the hash table.

Then it was switched to wrapping hash table accesses in a global lock during
h

Pause all threads while swapping hash table.

We used to hold a global lock around all modifications to the hash table.

Then it was switched to wrapping hash table accesses in a global lock during
hash table expansion, set by notifying each worker thread to change lock
styles. There was a bug here which causes trylocks to clobber, due to the
specific item locks not being held during the global lock:
https://code.google.com/p/memcached/issues/detail?id=370

The patch previous to this one uses item locks during hash table expansion.
Since the item lock table is always smaller than the hash table, an item lock
will always cover both its new and old buckets.

However, we still need to pause all threads during the pointer swap and setup.
This patch pauses all background threads and worker threads, swaps the hash
table, then unpauses them.

This trades the (possibly significant) slowdown during the hash table copy,
with a short total hang at the beginning of each expansion. As previously;
those worried about consistent performance can presize the hash table with
`-o hashpower=n`

show more ...

Avoid OOM errors when locked items stuck in tail

If a client fetches a few thousand keys, then does not ever read the socket,
those keys will stay reflocked until the client disconnects or resumes.

Avoid OOM errors when locked items stuck in tail

If a client fetches a few thousand keys, then does not ever read the socket,
those keys will stay reflocked until the client disconnects or resumes. If
some of those items are unpopular they can drop to the tail, causing all
writes in the slab class to OOM.

This creates some relief by chucking the items back to the head.

Big thanks to Jay Grizzard and other folks at Box for helping narrow this
down.

show more ...

optionally take a list of slabs to run against.

lru_crawler crawl 1,2,3,10,20 will kick crawlers off for all of those slabs in
parallel.

control system

nothing internally magically fires it off yet, but now there is an external
command:

lru_crawler crawl [classid]
... will signal the thread to wake up and immediately reap through a

control system

nothing internally magically fires it off yet, but now there is an external
command:

lru_crawler crawl [classid]
... will signal the thread to wake up and immediately reap through a
particular class.

need some thought/feedback for internal kickoffs (plugins?)

show more ...

barebones LRU crawler proof of concept

so many things undone... TODO is inline in items.c.

this seems to work, and the locking should be correct. it is a background
thread so shouldn't cause signif

barebones LRU crawler proof of concept

so many things undone... TODO is inline in items.c.

this seems to work, and the locking should be correct. it is a background
thread so shouldn't cause significant latency. However it does quickly roll
through the entire LRU (and as of this PoC it just constantly runs), so there
will be cpu impact.

show more ...

remove global stats lock from item allocation

This doesn't reduce mutex contention much, if at all, for the global stats
lock, but it does remove a handful of instructions from the alloc hot path,
w

remove global stats lock from item allocation

This doesn't reduce mutex contention much, if at all, for the global stats
lock, but it does remove a handful of instructions from the alloc hot path,
which is always worth doing.

Previous commits possibly added a handful of instructions for the loop and for
the bucket readlock trylock, but this is still faster than .14 for writes
overall.

show more ...

alloc loop now attempts an item_lock

Fixes a few issues with a restructuring... I think -M was broken before,
should be fixed now. It had a refcount leak.

Now walks up to five items from the bottom

alloc loop now attempts an item_lock

Fixes a few issues with a restructuring... I think -M was broken before,
should be fixed now. It had a refcount leak.

Now walks up to five items from the bottom in case of the bottomost items
being item_locked, or refcount locked. Helps avoid excessive OOM errors for
some oddball cases. Those happen more often if you're hammering on a handful
of pages in a very large class size (100k+)

The hash item lock ensures that if we're holding that lock, no other thread
can be incrementing the refcount lock at that time. It will mean more in
future patches.

slab rebalancer gets a similar update.

show more ...

initial slab automover

Enable at startup with -o slab_reassign,slab_automove

Enable or disable at runtime with "slabs automove 1\r\n"

Has many weaknesses. Only pulls from slabs which have had zero

initial slab automover

Enable at startup with -o slab_reassign,slab_automove

Enable or disable at runtime with "slabs automove 1\r\n"

Has many weaknesses. Only pulls from slabs which have had zero recent
evictions. Is slow, not tunable, etc. Use the scripts/mc_slab_mover example to
write your own external automover if this doesn't satisfy.

show more ...

use item partitioned lock for as much as possible

push cache_lock deeper into the abyss

move hash calls outside of cache_lock

been hard to measure while using the intel hash (since it's very fast), but
should help with the software hash.

Backport binary TOUCH/GAT/GATQ commands

Taken from the 1.6 branch, partly written by Trond. I hope the CAS handling is
correct.

Kill off redundant item_init.

These are automatically initialized to 0 (both Trond and the spec says
so, and I asserted it on all current builders at least once before
killing it off).