slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-ch

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-chunk
items.

also fixes a bug where the lru limit wasn't being utilized for HOT_LRU

also some cleanup from previous commits.

show more ...

chunked item second checkpoint

can actually fetch items now, and fixed a few bugs with storage/freeing.

added fetching for binprot.
added some basic tests.

many tests still fail for various reason

chunked item second checkpoint

can actually fetch items now, and fixed a few bugs with storage/freeing.

added fetching for binprot.
added some basic tests.

many tests still fail for various reasons, and append/prepend isn't fixed yet.

show more ...

chunked items checkpoint commit

can set and store large items via asciiprot. gets/append/prepend/binprot not
implemented yet.

use X macros to remove several struct iterations

Adding new stats would require updating in too many locations. No longer have
to. Could extend this a bit to do the actual stats printing, but for cl

use X macros to remove several struct iterations

Adding new stats would require updating in too many locations. No longer have
to. Could extend this a bit to do the actual stats printing, but for clarity's
sake I'll decide on that later.

show more ...

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an un

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an unused mutex.

Split global stats into `stats` and `stats_state`. initialize via memset,
reset only `stats` via memset, removing several places where stats values get
repeated. Looks much cleaner and should be less error prone.

show more ...

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values.
should no longer do that.

also removes get_expired from the global stats list. I missed that on review.

show more ...

add -o watcher_logbuf_size=N worker_logbuf_size=N

the defaults should just about always be used, but this is now tunable for
completeness. Also handy for writing tests.

This also hopefully makes th

add -o watcher_logbuf_size=N worker_logbuf_size=N

the defaults should just about always be used, but this is now tunable for
completeness. Also handy for writing tests.

This also hopefully makes the watcher.t test not flaky anymore.

show more ...

memcached idle connection killer

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so tw

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so two log lines are generated. An ascii multiget one make one log line
per key fetched.

It's a good place to start. Need to come back and refactor parts of logger.c
again, then convert all prints in the codebase to log entries.

show more ...

some global stats for bg logger

four obvious ones. have a handy place to do the summarization when the logger
wakes up for its run, avoiding the locks.

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to them, and all have a GID so they can be put
back into strict order.

This is an early preview. Code needs refactoring and a more complete set of
options. All watchers are also stuck viewing the global feed of logs, even if
they asked for different data.

As of this commit there's no way to toggle the "stderr" watcher.

show more ...

bump some global stats to 64bit uints

total_items is pretty easy to overflow. Upped some of the others just in case.

Implement get_expired stats

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of memory.

show more ...

split rebal_evictions into _nomem and _samepage

gross oversight putting two conditions into the same variable. now can tell if
we're evicting because we're hitting the bottom of the free memory pool

split rebal_evictions into _nomem and _samepage

gross oversight putting two conditions into the same variable. now can tell if
we're evicting because we're hitting the bottom of the free memory pool, or if
we keep trying to rescue items into the same page as the one being cleared.

show more ...

call STATS_LOCK() less in slab mover.

uses the slab_rebal struct to summarize stats, more occasionally grabbing the
global lock to fill them in, instead.

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destinatio

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destination of "0", which is a global
page pool. Pages can be re-assigned out of that pool during allocation.

Combined with item rescuing from the previous patch, we can safely shuffle
pages back to the reassignment pool as chunks free up naturally. This should
be a safe default going forward. Users should be able to decide to free or
move pages based on eviction pressure as well. This is coming up in another
commit.

This also fixes a calculation of the NOEXP LRU size, and completely removes
the old slab automover thread. Slab automove decisions will now be part of the
lru maintainer thread.

show more ...

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
s

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
slab class, copy the item over and replace it.

It's up to external systems to try to ensure free chunks are available before
moving a slab page. If there is no memory it will simply evict them as normal.

Also adds counters so we can finally tell how often these cases happen.

show more ...

Record and report on time spent in listen_disabled

fix major off by one issue

none of my machines could repro a crash, but it's definitely wrong :/ Very
sad.

fix off-by-one with slab management

data sticking into the highest slab class was unallocated. Thanks to pyry for
the repro case:

perl -e 'use Cache::Memcached;$memd = new Cache::Memcached {
server

fix off-by-one with slab management

data sticking into the highest slab class was unallocated. Thanks to pyry for
the repro case:

perl -e 'use Cache::Memcached;$memd = new Cache::Memcached {
servers=>["127.0.0.1:11212"]};for(20..1000){print "$_\n";$memd->set("fo2$_",
"a"x1024)};'
(in a loop)
with:
./memcached -v -m 32 -p 11212 -f 1.012

This serves as a note to turn this into a test.

show more ...

Make LRU crawler work from maint thread.

Wasn't sending the condition signal after a refactor :(

Also adds some stats to inspect how much work the LRU crawler is doing, and
removes some printf nois

Make LRU crawler work from maint thread.

Wasn't sending the condition signal after a refactor :(

Also adds some stats to inspect how much work the LRU crawler is doing, and
removes some printf noise for the LRU maintainer.

show more ...

spinlocks never seem to help in benchmarks

If a thread is allowed to go to sleep, it can be woken up early as soon as the
lock is freed. If we spinlock, the scheduler can't help us and threads will

spinlocks never seem to help in benchmarks

If a thread is allowed to go to sleep, it can be woken up early as soon as the
lock is freed. If we spinlock, the scheduler can't help us and threads will
randomly run out their timeslice until the thread actually holding the lock
finishes its work.

In my benchmarks killing the spinlock only makes things better.

show more ...

small crawler refactor

Separate the start function from what was string parsing and allow passing in
the 'remaining' value as an argument.

Also adds a (non-configurable yet) settings for how many c

small crawler refactor

Separate the start function from what was string parsing and allow passing in
the 'remaining' value as an argument.

Also adds a (non-configurable yet) settings for how many crawls to run per
sleep, to raise the default aggressiveness of the crawler.

show more ...

add `-o expirezero_does_not_evict` feature

When enabled, items with an expiration time of 0 are placed into a separate
LRU and are not subject to evictions. This allows a mixed-mode instance where
y

add `-o expirezero_does_not_evict` feature

When enabled, items with an expiration time of 0 are placed into a separate
LRU and are not subject to evictions. This allows a mixed-mode instance where
you can have a stronger "guarantee" (not a real guarantee) that items aren't
removed from the cache due to low memory.

This is a dangerous option, as mixing unevictable items has obvious
repercussions.

show more ...