logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so tw

logger endpoints for first release

swapping "RAWCMDS" for internal hooks on when items are fetched or stored.

This doesn't map 1:1 with commands, ie: a store is internally a fetch then
store, so two log lines are generated. An ascii multiget one make one log line
per key fetched.

It's a good place to start. Need to come back and refactor parts of logger.c
again, then convert all prints in the codebase to log entries.

show more ...

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker

convert eviction log to new process

add LOGGER_LOG() macro wrapper for callers.

eviction logging is now using the "more final" method, where data is copied
into structs embedded into the per-worker circular buffer. This moves the
expensive snprintf calls to the background thread, and also allows making the
logging format switchable.

also switched the format to key=value. it's still largely readable but much
easier to parse.

show more ...

some global stats for bg logger

four obvious ones. have a handy place to do the summarization when the logger
wakes up for its run, avoiding the locks.

improve 'watch' command processing a little

moves to its own function, respods "OK\r\n" to client, allows multiple
arguments for multiple flags. Needs more thought before adding sampling.

poll() smarter when full, print skipped count

When lines are skipped to a watcher, a [skipped: n] is printed before
resuming. Also polls full watcher directly once, and will wait until the
outer loo

poll() smarter when full, print skipped count

When lines are skipped to a watcher, a [skipped: n] is printed before
resuming. Also polls full watcher directly once, and will wait until the
outer loop to re-poll.

Think the routine can be simplified... it works so will leave it and revisit
later.

show more ...

use one bipbuffer per watcher. remove iovec code.

after the previous commit's exercise, this greatly simplifies the writing
process. The buffers are large but can be tuned with performance testing.

use one bipbuffer per watcher. remove iovec code.

after the previous commit's exercise, this greatly simplifies the writing
process. The buffers are large but can be tuned with performance testing.

Lines are now rendered into scratch space, and if event flags (eflags) match
for a watcher, copy the bytes into that watcher's specific buffer.

In most realistic cases, watchers will be watching different streams of
information. While this does require a minimum of one memcpy instead of
directly writing into the bipbuf as before, it removes all of the loops and
management of iovecs required for filtering events on the write-to-socket end.

show more ...

remove "logger chunks", add individualized streams

Stop doing a multi-reader circular buffer structure on top of a circular
buffer. Also adds individualized streams based off of the central buffer.

remove "logger chunks", add individualized streams

Stop doing a multi-reader circular buffer structure on top of a circular
buffer. Also adds individualized streams based off of the central buffer.

Sadly this requires managing iovecs and dealing with partial writes into said
iovecs. That makes things very complicated. Since it's not clear to me how to
simplify it too much (as of this writing), one of the next commits should
remove iovecs and instead give each watcher its own circular buffer. The
parser thread will copy watched events into each buffer.

The above would only be slower for the case of many watchers watching the same
event streams. Given all of the extra loops required for managing the iovecs,
and the more complicated syscall handling, it might even be the same speed to
manage multiple buffers anyway.

I completed this intermediary change since it simplifies the surrounding code
and was educational to fiddle with iovecs again.

show more ...

switch from bit fields to bit manipulation

can't compare bit fields en-masse, which makes things too difficult.

manage logger watcher flags.

very temporary user control. allows to watch either fetchers or evictions, but
not both, and always with timestamps.

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to

initial logger code.

Logs are written to per-thread buffers. A new background thread aggregates the
logs, further processes them, then writes them to any "watchers".

Logs can have the time added to them, and all have a GID so they can be put
back into strict order.

This is an early preview. Code needs refactoring and a more complete set of
options. All watchers are also stuck viewing the global feed of logs, even if
they asked for different data.

As of this commit there's no way to toggle the "stderr" watcher.

show more ...