fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, n

fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, new item allocation consume memory space
every time an item is set, that leads to slab OOM errors.

show more ...

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-ch

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-chunk
items.

also fixes a bug where the lru limit wasn't being utilized for HOT_LRU

also some cleanup from previous commits.

show more ...

startup options for chunked items.

has spent some time under performance testing. For larger items there's less
than 5% extra CPU usage, however the max usable CPU when using large items is
1/10th o

startup options for chunked items.

has spent some time under performance testing. For larger items there's less
than 5% extra CPU usage, however the max usable CPU when using large items is
1/10th or less before you run out of bandwidth. Mixed small/large items will
still balance out.

comments out debugging (which must be removed for release).

restores defaults and ensures only t/chunked-items.t is affected.

dyn-maxbytes and item_size_max tests still fail.

append/prepend aren't implemented, sasl needs to be guarded.

slab mover needs to be updated.

show more ...

chunked items checkpoint commit

can set and store large items via asciiprot. gets/append/prepend/binprot not
implemented yet.

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an un

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an unused mutex.

Split global stats into `stats` and `stats_state`. initialize via memset,
reset only `stats` via memset, removing several places where stats values get
repeated. Looks much cleaner and should be less error prone.

show more ...

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values.
should no longer do that.

also removes get_expired from the global stats list. I missed that on review.

show more ...

change LRU crawler fprintf to use %u format

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat for `stats settings`.

Finally, adds documentation and cleans up status outputs.

Could use some automated tests but not make or break for release.

show more ...

online hang-free "stats sizes" command.

"stats sizes" is one of the lack cache-hanging commands. With millions of
items it can hang for many seconds.

This commit changes the command to be dynamic.

online hang-free "stats sizes" command.

"stats sizes" is one of the lack cache-hanging commands. With millions of
items it can hang for many seconds.

This commit changes the command to be dynamic. A histogram is tracked as items
are linked and unlinked from the cache. The tracking is enabled or disabled at
runtime via "stats sizes_enable" and "stats sizes_disable".

This presently "works" but isn't accurate. Giving it some time to think over
before switching to requiring that CAS be enabled. Otherwise the values could
underflow if items are removed that existed before the sizes tracker is
enabled. This attempts to work around it by using it->time, which gets updated
on fetch, and is thus inaccurate.

show more ...

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as
unsigned up until a change made in 2007. Now treat it fully as unsigned and
print as unsigned.

show more ...

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so tw

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so two log lines are generated. An ascii multiget one make one log line
per key fetched.

It's a good place to start. Need to come back and refactor parts of logger.c
again, then convert all prints in the codebase to log entries.

show more ...

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker circular buffer. This moves the
expensive snprintf calls to the background thread, and also allows making the
logging format switchable.

also switched the format to key=value. it's still largely readable but much
easier to parse.

show more ...

switch from bit fields to bit manipulation

can't compare bit fields en-masse, which makes things too difficult.

manage logger watcher flags.

very temporary user control. allows to watch either fetchers or evictions, but
not both, and always with timestamps.

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to them, and all have a GID so they can be put
back into strict order.

This is an early preview. Code needs refactoring and a more complete set of
options. All watchers are also stuck viewing the global feed of logs, even if
they asked for different data.

As of this commit there's no way to toggle the "stderr" watcher.

show more ...

Add braces around APPEND_NUM_FMT_STAT conditional

APPEND_NUM_FMT_STAT is a macro that expands to multiple statements, so it must have braces when used in a conditional.

Implement get_expired stats

make cachedump print 0exp for noexpire items

re GH#128

fix build with musl libc

musl libc will warn if you include sys/signal.h instead of signal.h as
specified by posix. Build will fail due to -Werror explicitly beeing
set.

Fix it by use the posix loc

fix build with musl libc

musl libc will warn if you include sys/signal.h instead of signal.h as
specified by posix. Build will fail due to -Werror explicitly beeing
set.

Fix it by use the posix location.

fixes #138

show more ...

'lru_crawler enable' blocks until ready

single CPU VM builders could fail:
- spawn LRU crawler thread.
- signal LRU crawler.
- LRU crawler thread now waits on condition.
- Crawler thread misses

'lru_crawler enable' blocks until ready

single CPU VM builders could fail:
- spawn LRU crawler thread.
- signal LRU crawler.
- LRU crawler thread now waits on condition.
- Crawler thread misses condition, sits forever.

Might also want to move the "stats.lru_crawler_running" bit to be updated when
the crawler thread picks up the work to do, somehow.

show more ...

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of

try harder to save items

previously the slab mover would evict items if the new chunk was within the
slab page being moved. now it will do an inline reclaim of the chunk and try
until it runs out of memory.

show more ...

tune automove to required 2.5 pages of free chunks

if we're deciding to move pages right on the chunk boundary it's too easy to
cause flapping.

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destinatio

first half of new slab automover

If any slab classes have more than two pages worth of free chunks, attempt to
free one page back to a global pool.

Create new concept of a slab page move destination of "0", which is a global
page pool. Pages can be re-assigned out of that pool during allocation.

Combined with item rescuing from the previous patch, we can safely shuffle
pages back to the reassignment pool as chunks free up naturally. This should
be a safe default going forward. Users should be able to decide to free or
move pages based on eviction pressure as well. This is coming up in another
commit.

This also fixes a calculation of the NOEXP LRU size, and completely removes
the old slab automover thread. Slab automove decisions will now be part of the
lru maintainer thread.

show more ...

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
s

slab mover rescues valid items with free chunks

During a slab page move items are typically ejected regardless of their
validity. Now, if an item is valid and free chunks are available in the same
slab class, copy the item over and replace it.

It's up to external systems to try to ensure free chunks are available before
moving a slab page. If there is no memory it will simply evict them as normal.

Also adds counters so we can finally tell how often these cases happen.

show more ...

restore slab_automove=2 support and add test

Test is a port of a golang test submitted by Scott Mansfield.

There used to be an "angry birds mode" to slabs_automove, which attempts to
force a slab m

restore slab_automove=2 support and add test

Test is a port of a golang test submitted by Scott Mansfield.

There used to be an "angry birds mode" to slabs_automove, which attempts to
force a slab move from "any" slab into the one which just had an eviction.
This is an imperfect but fast way of responding to shifts in memory
requirements.

This change adds it back in plus a test which very quickly attempts to set
data in via noreply. This isn't the end of improvements here. This commit is a
starting point.

show more ...