fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, n

fix zero hash items eviction

If all hash values of five tail items are zero on the specified slab
class, expire check is unintentionally skipped and items stay without
being evicted. Consequently, new item allocation consume memory space
every time an item is set, that leads to slab OOM errors.

show more ...

shuffle the EXPERIMENTAL tag.

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-ch

slabs reassigns works with chunks and chunked items.

also fixes the new LRU algorithm to balance by total bytes used rather than
total chunks used, since total chunks used isn't tracked for multi-chunk
items.

also fixes a bug where the lru limit wasn't being utilized for HOT_LRU

also some cleanup from previous commits.

show more ...

CLRF in binprot was inverted.

now extends data in the chunk rather than overwriting the end.

chunked append/prepend now work.

This is also the only actual tests of append and prepend in the codebase :/
t/binary.t had some simple tests but no others were ever written.

protect binary sasl from chunked items.

not entirely sure how to test this or guarantee people don't set the chunk
low enough to cause problems. might circle back to add better tests.

startup options for chunked items.

has spent some time under performance testing. For larger items there's less
than 5% extra CPU usage, however the max usable CPU when using large items is
1/10th o

startup options for chunked items.

has spent some time under performance testing. For larger items there's less
than 5% extra CPU usage, however the max usable CPU when using large items is
1/10th or less before you run out of bandwidth. Mixed small/large items will
still balance out.

comments out debugging (which must be removed for release).

restores defaults and ensures only t/chunked-items.t is affected.

dyn-maxbytes and item_size_max tests still fail.

append/prepend aren't implemented, sasl needs to be guarded.

slab mover needs to be updated.

show more ...

chunked item second checkpoint

can actually fetch items now, and fixed a few bugs with storage/freeing.

added fetching for binprot.
added some basic tests.

many tests still fail for various reason

chunked item second checkpoint

can actually fetch items now, and fixed a few bugs with storage/freeing.

added fetching for binprot.
added some basic tests.

many tests still fail for various reasons, and append/prepend isn't fixed yet.

show more ...

chunked items checkpoint commit

can set and store large items via asciiprot. gets/append/prepend/binprot not
implemented yet.

Update memcached.c

It's returning an error when we try to cachedump the slab 63

Fix ancient binprot bug with OOM errors desycning

sets getting OOM's for item alloc would swallow the upload bytes in ascii
protocol but missed in binprot. Swallow is usually passed through
write_bi

Fix ancient binprot bug with OOM errors desycning

sets getting OOM's for item alloc would swallow the upload bytes in ascii
protocol but missed in binprot. Swallow is usually passed through
write_bin_error, but out_of_memory() calls write_bin_error indirectly with a
zero for swallow.

show more ...

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an un

clean up global stats code a little.

tons of stats were left in the global stats structure that're no longer used,
and it looks like we kept accidentally adding new ones in there. There's also
an unused mutex.

Split global stats into `stats` and `stats_state`. initialize via memset,
reset only `stats` via memset, removing several places where stats values get
repeated. Looks much cleaner and should be less error prone.

show more ...

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values

add get_flushed counter, fix expired_unfetched

get_flushed ticks when directly fetching something that classified as expired.
LRU crawler was ticking expired_unfetched for unfetched _flushed_ values.
should no longer do that.

also removes get_expired from the global stats list. I missed that on review.

show more ...

Fix undefined behavior in vlen check

The size check for vlen in process_get_command was relying on
undefined behavior (signed integer overflow) to check to see
if vlen was greater than 2^31-2. This

Fix undefined behavior in vlen check

The size check for vlen in process_get_command was relying on
undefined behavior (signed integer overflow) to check to see
if vlen was greater than 2^31-2. This made certain compilers
(such as clang 7.3.x) optimize out part of the conditional and
prevent memcache from erroring when the specified size was
exactly 2^31-1, leading to an incorrect error message and a core
dump. This patch does that overflow test in a not-undefined-behavior
way.

Fixes failure in issue_70.t test #4 on those platforms.

show more ...

cache_memlimit command for tuning runtime maxbytes

Allows dynamically increasing the memory limit of a running system, if memory
isn't being preallocated.

If `-o modern` is in use, can also dynamic

cache_memlimit command for tuning runtime maxbytes

Allows dynamically increasing the memory limit of a running system, if memory
isn't being preallocated.

If `-o modern` is in use, can also dynamically lower memory usage. pages are
free()'ed back to the OS via the slab rebalancer as memory is freed up. Does
not guarantee the OS will actually give the memory back for other applications
to use, that depends on how the OS handles memory.

show more ...

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat

finish stats_sizes rewrite

Now relies on CAS feature for runtime enable/disable tracking. Still usable if
enabled at starttime with CAS disabled. Also adds start option `-o
track_sizes`, and a stat for `stats settings`.

Finally, adds documentation and cleans up status outputs.

Could use some automated tests but not make or break for release.

show more ...

allow manually specifying slab class sizes

"-o slab_sizes=100-200-300-400-500" will create 5 slab classes of those
specified sizes, with the final class being item_max_size.

Using the new online st

allow manually specifying slab class sizes

"-o slab_sizes=100-200-300-400-500" will create 5 slab classes of those
specified sizes, with the final class being item_max_size.

Using the new online stats sizes command, it's possible to determine if the
typical factoral slab class growth rate doesn't align well with how items are
stored.

This is dangerous unless you really know what you're doing. If your items have
an exact or very predictable size this makes a lot of sense. If they do not,
the defaults are safer.

show more ...

add -o watcher_logbuf_size=N worker_logbuf_size=N

the defaults should just about always be used, but this is now tunable for
completeness. Also handy for writing tests.

This also hopefully makes th

add -o watcher_logbuf_size=N worker_logbuf_size=N

the defaults should just about always be used, but this is now tunable for
completeness. Also handy for writing tests.

This also hopefully makes the watcher.t test not flaky anymore.

show more ...

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as

treat and print item flags as unsigned int

most of the code would parse and handle flags as unsigned int, but passed into
alloc functions as a signed int... which would then continue to print it as
unsigned up until a change made in 2007. Now treat it fully as unsigned and
print as unsigned.

show more ...

memcached idle connection killer

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so tw

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so two log lines are generated. An ascii multiget one make one log line
per key fetched.

It's a good place to start. Need to come back and refactor parts of logger.c
again, then convert all prints in the codebase to log entries.

show more ...

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker circular buffer. This moves the
expensive snprintf calls to the background thread, and also allows making the
logging format switchable.

also switched the format to key=value. it's still largely readable but much
easier to parse.

show more ...

some global stats for bg logger

four obvious ones. have a handy place to do the summarization when the logger
wakes up for its run, avoiding the locks.

improve 'watch' command processing a little

moves to its own function, respods "OK\r\n" to client, allows multiple
arguments for multiple flags. Needs more thought before adding sampling.

remove "logger chunks", add individualized streams

Stop doing a multi-reader circular buffer structure on top of a circular
buffer. Also adds individualized streams based off of the central buffer.

remove "logger chunks", add individualized streams

Stop doing a multi-reader circular buffer structure on top of a circular
buffer. Also adds individualized streams based off of the central buffer.

Sadly this requires managing iovecs and dealing with partial writes into said
iovecs. That makes things very complicated. Since it's not clear to me how to
simplify it too much (as of this writing), one of the next commits should
remove iovecs and instead give each watcher its own circular buffer. The
parser thread will copy watched events into each buffer.

The above would only be slower for the case of many watchers watching the same
event streams. Given all of the extra loops required for managing the iovecs,
and the more complicated syscall handling, it might even be the same speed to
manage multiple buffers anyway.

I completed this intermediary change since it simplifies the surrounding code
and was educational to fiddle with iovecs again.

show more ...