[core] chunk_file_pread() to wrap pread()

[core] use pread() to skip lseek()

[multiple] internal control for backend read bytes

separate internal control for backend max_per_read

When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a sm

[multiple] internal control for backend read bytes

separate internal control for backend max_per_read

When not streaming, large reads will be flushed to temp files on disk.
When streaming, use a smaller buffer to help reduce memory usage.

When not streaming, attempt to read and empty kernel socket bufs.
(e.g. MAX_READ_LIMIT 256k)

When writing to sockets (or pipes) attempt to fill kernel socket bufs.
(e.g. MAX_WRITE_LIMIT 256k)

show more ...

[core] clear buffer after backend dechunk

(thx flynn)

clear buffer after backend dechunk if not sending chunked to client

x-ref:
"Memory fragmentation with HTTP/2 enabled"
https://redmine.ligh

[core] clear buffer after backend dechunk

(thx flynn)

clear buffer after backend dechunk if not sending chunked to client

x-ref:
"Memory fragmentation with HTTP/2 enabled"
https://redmine.lighttpd.net/issues/3084

show more ...

[multiple] remove base.h include where not used

(substitute request.h if file only accesses request_st,
and not connection or server structs)

[core] http_response_append_{buffer,mem}()

manage r->resp_body_scratchpad in new funcs
http_response_append_buffer()
http_response_append_mem()
rather than
http_chunk_decode_append_buffer()

[core] http_response_append_{buffer,mem}()

manage r->resp_body_scratchpad in new funcs
http_response_append_buffer()
http_response_append_mem()
rather than
http_chunk_decode_append_buffer()
http_chunk_decode_append_mem()
which now only decode chunked encoding, more apropos for the func names

show more ...

[core] reduce excess cc inlining in http_chunk.c

[multiple] reduce redundant NULL buffer checks

This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot co

[multiple] reduce redundant NULL buffer checks

This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.

Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.

In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.

- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend

Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)

internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg

show more ...

[core] remove some (now) unused http_chunk APIs

remove http_chunk_append_file() and http_chunk_append_file_range()

callers should choose to use stat_cache_entry_open() for caching
or should open fi

[core] remove some (now) unused http_chunk APIs

remove http_chunk_append_file() and http_chunk_append_file_range()

callers should choose to use stat_cache_entry_open() for caching
or should open file and check sizes and ranges

show more ...

[core] range chk http_chunk_append_file_ref_range

add range sanity check in http_chunk_append_file_ref_range()
(before potentially sending HTTP/1.1 chunked header)

[core] fix chunkqueue_small_resp_optim partial rd

[core] fix decoding chunked from backend (fixes #3049)

(thx flynn)

fix decoding chunked from backend

truncate response and error out if backend sends excess data
after chunked encoding

x-ref:
"

[core] fix decoding chunked from backend (fixes #3049)

(thx flynn)

fix decoding chunked from backend

truncate response and error out if backend sends excess data
after chunked encoding

x-ref:
"Too much content with HTTP/2.0"
https://redmine.lighttpd.net/issues/3049

show more ...

[core] track Content-Length from backend (fixes #3046)

track Content-Length from backend in r->resp_body_scratchpad

x-ref:
"Failure on second request in http proxy backend"
https://redmine.ligh

[core] track Content-Length from backend (fixes #3046)

track Content-Length from backend in r->resp_body_scratchpad

x-ref:
"Failure on second request in http proxy backend"
https://redmine.lighttpd.net/issues/3046

show more ...

[core] track chunked encoding state from backend (fixes #3046)

(thx flynn)

track chunked encoding state when parsing backend response

x-ref:
"Failure on second request in http proxy backend"
h

[core] track chunked encoding state from backend (fixes #3046)

(thx flynn)

track chunked encoding state when parsing backend response

x-ref:
"Failure on second request in http proxy backend"
https://redmine.lighttpd.net/issues/3046

show more ...

[core] reject excess data after chunked encoding (#3046)

reject excess data after chunked encoding when parsing backend response

x-ref:
"Failure on second request in http proxy backend"
https:/

[core] reject excess data after chunked encoding (#3046)

reject excess data after chunked encoding when parsing backend response

x-ref:
"Failure on second request in http proxy backend"
https://redmine.lighttpd.net/issues/3046

show more ...

[core] fix chunked decoding from backend (fixes #3044)

(thx flynn)

x-ref:
"Socket errors after update to version 1.4.56"
https://redmine.lighttpd.net/issues/3044

[core] consolidate chunk size checks

[core] always lseek() with shared fd

always lseek() with shared fd; remove optim to skip with offset = 0

[core] http_chunk_append_file_ref_range()

http_chunk_append_file_ref() and http_chunk_append_file_ref_range()
to take stat_cache_entry ref and append FILE_CHUNK

[core] alloc optim reading file, sending chunked

avoid potential double-copy due to not enough space for final '\0'
in http_chunk_append_read_fd_range() if read size is exactly multiple
of 8k and se

[core] alloc optim reading file, sending chunked

avoid potential double-copy due to not enough space for final '\0'
in http_chunk_append_read_fd_range() if read size is exactly multiple
of 8k and sending chunked response

show more ...

[core] pass open fd to http_response_parse_range

[multiple] inline chunkqueue where always alloc'd

inline struct chunkqueue where always allocated in other structs

(memory locality)

[multiple] inline chunkqueue_length()

[core] defer optimization to read small files

defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether o

[core] defer optimization to read small files

defer optimization to read small files into memory until after
response_start hooks have a chance to run, e.g. until after
mod_deflate chooses whether or not to serve file from compressed
cache, if deflate.cache-dir is configured

show more ...

[core] decode Transfer-Encoding: chunked from gw

decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a

[core] decode Transfer-Encoding: chunked from gw

decode Transfer-Encoding: chunked from gw (gateway backends)

Transfer-Encoding: chunked is a hop-by-hop header.

Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1
requests to backends and be able to handle HTTP/1.1 responses.

Other backends ought not to send Transfer-Encoding: chunked, but in
practice, some implementations do.

show more ...