slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-ch

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-chunk
items.

also fixes a bug where the lru limit wasn't being utilized for HOT_LRU

also some cleanup from previous commits.

show more ...

CLRF in binprot was inverted.

now extends data in the chunk rather than overwriting the end.

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values.
should no longer do that.

also removes get_expired from the global stats list. I missed that on review.

show more ...

bump binary.t

memcached idle connection killer

misc cleanups for logger code.

tests pass now.

update binary test count for get_expired stat

minor fixes and docs for listen_disabled timer

add a handful of missing stats docs (doh) and shorted the STATS_LOCK a little
bit. the rest seems fine.

Record and report on time spent in listen_disabled

Make LRU crawler work from maint thread.

Wasn't sending the condition signal after a refactor :(

Also adds some stats to inspect how much work the LRU crawler is doing, and
removes some printf nois

Make LRU crawler work from maint thread.

Wasn't sending the condition signal after a refactor :(

Also adds some stats to inspect how much work the LRU crawler is doing, and
removes some printf noise for the LRU maintainer.

show more ...

add `-o expirezero_does_not_evict` feature

When enabled, items with an expiration time of 0 are placed into a separate
LRU and are not subject to evictions. This allows a mixed-mode instance where
y

add `-o expirezero_does_not_evict` feature

When enabled, items with an expiration time of 0 are placed into a separate
LRU and are not subject to evictions. This allows a mixed-mode instance where
you can have a stronger "guarantee" (not a real guarantee) that items aren't
removed from the cache due to low memory.

This is a dangerous option, as mixing unevictable items has obvious
repercussions.

show more ...

make HOT/WARM ratios starttime tunable.

runtime tunable is difficult and may require either atomics, or adding an
extra items.c array. Adjusting the value would roll through and lock each
LRU before

make HOT/WARM ratios starttime tunable.

runtime tunable is difficult and may require either atomics, or adding an
extra items.c array. Adjusting the value would roll through and lock each
LRU before changing the value.

show more ...

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also remove

direct reclaim mode for evictions

Only way to do eviction case fast enough is to inline it, sadly.
This finally deletes the old item_alloc code now that I'm not intending on
reusing it.

Also removes the condition wakeup for the background thread. Instead runs on a
timer, and meters its aggressiveness by how much shuffling is going on.

Also fixes a segfault in lru_pull_tail(), was unlinking `it` instead of
`search`.

show more ...

rename reflocked + add protocol.txt line

now lrutail_reflocked.. gross but more descriptive. will still make people
thing something went wrong, oh well.

lru_crawler_tocrawl to limit inspected per run

Since we start from the tail, and some slabs could be many millions of items,
it doesn't always make sense to walk the entire thing looking for data to

lru_crawler_tocrawl to limit inspected per run

Since we start from the tail, and some slabs could be many millions of items,
it doesn't always make sense to walk the entire thing looking for data to use.

This could be a lot smarter: say specify a percentage, and it'll only crawl a
percentage of used_chunks for that slab for instance.

show more ...

lru_crawler fixups for latest master branch

Make hash table algorithm selectable

jenkins hash is old. Lets try murmur3 to start! Default is the old one, so
people aren't surprised.

Fix for incorrect length of initial value set via binary increment protocol.

Added parameter to disable flush_all (nice to have on production)

Fix tests again..

Some tail_repair_time adjustments

fix t/binary.t test.
Default of 3 hours is super overkill. 1 hour is still way too much but within
realm of reason.
Bail if someone sets it really low (10s or negat

Some tail_repair_time adjustments

fix t/binary.t test.
Default of 3 hours is super overkill. 1 hour is still way too much but within
realm of reason.
Bail if someone sets it really low (10s or negative)

show more ...

fix some binary.t stuff

I broke 'em earlier

add a binary touch test that actaully fails

tests: loop on short binary packet reads

Awesome bug goes like this:

let "c1" be the commit of the "good state" and "c2" be the commit
immediately after (in a bad state). "t1" is the state of the

tests: loop on short binary packet reads

Awesome bug goes like this:

let "c1" be the commit of the "good state" and "c2" be the commit
immediately after (in a bad state). "t1" is the state of the tree in "c1"
and "t2" is the state of the tree in "c2"

In their natural states, we have this:

c1 -> t1 -> success
c1 -> t2 -> fail

However, if you take

c1 -> t1 -> patch to t2 -> success
c2 -> t2 -> patch to t1 -> fail

So t1 *and* t2 both succeed if the committed tree is c1, but both fail of
the committed tree is c2.

The difference? c1 has a tag that points to it so the version number is
"1.2.10" whereas the version number for the unreleased c2 is
"1.4.10-1-gee486ab" -- a bit longer, breaks stuff in tests that try to
print stats.

show more ...

slab reassignment

Adds a "slabs reassign src dst" manual command, and a thread to safely process
slab moves in the background.

- slab freelist is now a linked list, reusing the item structure
- is

slab reassignment

Adds a "slabs reassign src dst" manual command, and a thread to safely process
slab moves in the background.

- slab freelist is now a linked list, reusing the item structure
- is -o slab_reassign is enabled, an extra background thread is started
- thread attempts to safely free up items when it's been told to move a page
from one slab to another.

-o slab_automove is stubbed.

There are some limitations. Most notable is that you cannot repeatedly move
pages around without first having items use up the memory. Slabs with newly
assigned memory work off of a pointer, handing out chunks individually. We
would need to change that to quickly split chunks for all newly assigned pages
into that slabs freelist.

Further testing is required to ensure such is possible without impacting
performance.

show more ...