[core] cast to fix compiler error in prior commit

[mod_accesslog] %{mask}a to mask/anonymize IP

(thx pmconrad)

IPv4: mask final octet (8 bits) of address
IPv6: mask final 10 octets (80 bits) of address

x-ref:
Enable partial masking of IP addres

[mod_accesslog] %{mask}a to mask/anonymize IP

(thx pmconrad)

IPv4: mask final octet (8 bits) of address
IPv6: mask final 10 octets (80 bits) of address

x-ref:
Enable partial masking of IP addresses in access logs
https://github.com/lighttpd/lighttpd1.4/pull/124
IP masking in Universal Analytics
https://support.google.com/analytics/answer/2763052

github: closes #124

show more ...

[multiple] store ptrs to remote addr in request_st (#3192)

adds two pointers to (request_st *) (cost: 16 bytes in 64-bit builds)

prepares for upcoming changes to mod_extforward to manage remote add

[multiple] store ptrs to remote addr in request_st (#3192)

adds two pointers to (request_st *) (cost: 16 bytes in 64-bit builds)

prepares for upcoming changes to mod_extforward to manage remote addr
per request for HTTP/2 requests, rather than remote addr per connection.

Modern load balancers often provide options to reuse connections for
*different* clients, and therefore mod_extforward might change the
remote addr per request.

x-ref:
"RFE: mod_extforward and multiplexed requests via HTTP/2"
https://redmine.lighttpd.net/issues/3192
"Evaluation of remote_addr for mod_maxminddb for multiplexed connections"
https://redmine.lighttpd.net/issues/3191

show more ...

[multiple] employ ck_calloc, ck_malloc shared code

employ ck_calloc(), ck_malloc() shared code to slightly reduce code size
(centralize the ck_assert() to check that memory allocation succeeded)

[multiple] mark mod_*_plugin_init() funcs cold

[core] buffer_append_bs_escaped_json()

separate func from buffer_append_bs_escaped() so that both
buffer_append_bs_escaped() and buffer_append_bs_escaped_json()
can be slightly more specialized and

[core] buffer_append_bs_escaped_json()

separate func from buffer_append_bs_escaped() so that both
buffer_append_bs_escaped() and buffer_append_bs_escaped_json()
can be slightly more specialized and optimized

show more ...

[core] buffer_append_bs_escaped()

move accesslog_append_escaped() to buffer_append_bs_escaped()
replace buffer_append_string_encoded_json()

[mod_accesslog] reorder fields in switch()

sort enum and switch to popular use, and have switch order match enum

split less popular format options to separate cold func

[multiple] simplify bytes_in/bytes_out accounting

encapsulate accounting calculations in
http_request_stats_bytes_in()
http_request_stats_bytes_out()

more accurate accounting for HTTP/1.1 bytes_i

[multiple] simplify bytes_in/bytes_out accounting

encapsulate accounting calculations in
http_request_stats_bytes_in()
http_request_stats_bytes_out()

more accurate accounting for HTTP/1.1 bytes_in on keep-alive requests
(affects case where client pipelines HTTP/1.1 requests)

remove con->bytes_read and con->bytes_written
(no longer needed since request_st was split from connection struct
and request bytes_read_ckpt and bytes_written_ckpt are maintained
for HTTP/1.x bytes_in and bytes_out accounting. Also, further back,
chunkqueue internal accounting was simplified to maintain bytes_in
and bytes_out to always match chunkqueue length)

show more ...

[mod_accesslog] remove begin/end tags from %{}t

remove begin/end tags from %{begin:...}t and %{end:...}t
as those should not be part of the strftime format string

[mod_accesslog] accesslog.escaping = "json"

option to escape strings using json-compliant \uXXXX escape sequences
accesslog.escaping = "json"
(default is accesslog.escaping = "default" or anything

[mod_accesslog] accesslog.escaping = "json"

option to escape strings using json-compliant \uXXXX escape sequences
accesslog.escaping = "json"
(default is accesslog.escaping = "default" or anything else for default)

additional mod_accesslog optimizations, code reuse, smaller funcs

show more ...

[mod_accesslog] do not double-count hdr len in %I

do not double-count HTTP/2 stream request header len in %I

(overlooked and should have been adjusted in 8fc8ab89)

[core] better asm for binary num to ascii string

compiler optimizers generally convert div to an equivalent mul,
though not always optimally for modulus (%). In places where
lighttpd is using both

[core] better asm for binary num to ascii string

compiler optimizers generally convert div to an equivalent mul,
though not always optimally for modulus (%). In places where
lighttpd is using both quotient and remainder, calculate the
remainder from the quotient.

x-ref: inspiration:
https://lemire.me/blog/2019/02/08/faster-remainders-when-the-divisor-is-a-constant-beating-compilers-and-libdivide/
https://lemire.me/blog/2019/02/20/more-fun-with-fast-remainders-when-the-divisor-is-a-constant/

show more ...

[multiple] de-dup file and piped loggers (fixes #3101)

de-dup file and piped loggers for error logs and access logs

x-ref:
"RFE: de-dup file and piped loggers"
https://redmine.lighttpd.net/issu

[multiple] de-dup file and piped loggers (fixes #3101)

de-dup file and piped loggers for error logs and access logs

x-ref:
"RFE: de-dup file and piped loggers"
https://redmine.lighttpd.net/issues/3101

show more ...

[core] fdlog.[ch]; fdevent_*_logger_* -> fdlog_*

code move and rename fdevent_*_logger_*() to fdlog_*()

[multiple] Y2038 32-bit signed time_t mitigations

Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and Fr

[multiple] Y2038 32-bit signed time_t mitigations

Most OS platforms have already provided solutions to
Y2038 32-bit signed time_t 5 - 10 years ago (or more!)
Notable exceptions are Linux i686 and FreeBSD i386.

Since 32-bit systems tend to be embedded systems,
and since many distros take years to pick up new software,
this commit aims to provide Y2038 mitigations for lighttpd
running on 32-bit systems with Y2038-unsafe 32-bit signed time_t

* Y2038: lighttpd 1.4.60 and later report Y2038 safety
$ lighttpd -V
+ Y2038 support # Y2038-SAFE
$ lighttpd -V
- Y2038 support (unsafe 32-bit signed time_t) # Y2038-UNSAFE

* Y2038: general platform info
* Y2038-SAFE: lighttpd 64-bit builds on platforms using 64-bit time_t
- all major 64-bit platforms (known to this author) use 64-bit time_t
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 64-bit time_t
- Linux x32 ABI (different from i686)
- FreeBSD all 32-bit and 64-bit architectures *except* 32-bit i386
- NetBSD 6.0 (released Oct 2012) all 32-bit and 64-bit architectures
- OpenBSD 5.5 (released May 2014) all 32-bit and 64-bit architectures
- Microsoft Windows XP and Visual Studio 2005 (? unsure ?)
Another reference suggests Visual Studio 2015 defaults to 64-bit time_t
- MacOS 10.15 Catalina (released 2019) drops support for 32-bit apps
* Y2038-SAFE: lighttpd 32-bit builds on platforms using 32-bit unsigned time_t
- e.g. OpenVMS (unknown if lighttpd builds on this platform)
* Y2038-UNSAFE: lighttpd 32-bit builds on platforms using 32-bit signed time_t
- Linux 32-bit (including i686)
- glibc 32-bit library support not yet available for 64-bit time_t
- https://sourceware.org/glibc/wiki/Y2038ProofnessDesign
- Linux kernel 5.6 on 32-bit platforms does support 64-bit time_t
https://itsubuntu.com/linux-kernel-5-6-to-fix-the-year-2038-issue-unix-y2k/
- https://www.gnu.org/software/libc/manual/html_node/64_002dbit-time-symbol-handling.html
"Note: at this point, 64-bit time support in dual-time
configurations is work-in-progress, so for these
configurations, the public API only makes the 32-bit time
support available. In a later change, the public API will
allow user code to choose the time size for a given
compilation unit."
- compiling with -D_TIME_BITS=64 currently has no effect
- glibc recent (Jul 2021) mailing list discussion
- https://public-inbox.org/bug-gnulib/[email protected]/T/
- FreeBSD i386
- DragonFlyBSD 32-bit

* Y2038 mitigations attempted on Y2038-UNSAFE platforms (32-bit signed time_t)
* lighttpd prefers system monotonic clock instead of realtime clock
in places where realtime clock is not required
* lighttpd treats negative time_t values as after 19 Jan 2038 03:14:07 GMT
* (lighttpd presumes that lighttpd will not encounter dates before 1970
during normal operation.)
* lighttpd casts struct stat st.st_mtime (and st.st_*time) through uint64_t
to convert negative timestamps for comparisions with 64-bit timestamps
(treating negative timestamp values as after 19 Jan 2038 03:14:07 GMT)
* lighttpd provides unix_time64_t (int64_t) and
* lighttpd provides struct unix_timespec64 (unix_timespec64_t)
(struct timespec equivalent using unix_time64_t tv_sec member)
* lighttpd provides gmtime64_r() and localtime64_r() wrappers
for platforms 32-bit platforms using 32-bit time_t and
lighttpd temporarily shifts the year in order to use
gmtime_r() and localtime_r() (or gmtime() and localtime())
from standard libraries, before readjusting year and passing
struct tm to formatting functions such as strftime()
* lighttpd provides TIME64_CAST() macro to cast signed 32-bit time_t to
unsigned 32-bit and then to unix_time64_t

* Note: while lighttpd tries handle times past 19 Jan 2038 03:14:07 GMT
on 32-bit platforms using 32-bit signed time_t, underlying libraries and
underlying filesystems might not behave properly after 32-bit signed time_t
overflows (19 Jan 2038 03:14:08 GMT). If a given 32-bit OS does not work
properly using negative time_t values, then lighttpd likely will not work
properly on that system.

* Other references and blogs
- https://en.wikipedia.org/wiki/Year_2038_problem
- https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs
- http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/

show more ...

[multiple] inline struct in con->dst_addr_buf

(mod_extforward recently changed to use buffer_move() to save addr
instead of swapping pointers)

[multiple] reduce redundant NULL buffer checks

This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot co

[multiple] reduce redundant NULL buffer checks

This commit is a large set of code changes and results in removal of
hundreds, perhaps thousands, of CPU instructions, a portion of which
are on hot code paths.

Most (buffer *) used by lighttpd are not NULL, especially since buffers
were inlined into numerous larger structs such as request_st and chunk.

In the small number of instances where that is not the case, a NULL
check is often performed earlier in a function where that buffer is
later used with a buffer_* func. In the handful of cases that remained,
a NULL check was added, e.g. with r->http_host and r->conf.server_tag.

- check for empty strings at config time and set value to NULL if blank
string will be ignored at runtime; at runtime, simple pointer check
for NULL can be used to check for a value that has been set and is not
blank ("")
- use buffer_is_blank() instead of buffer_string_is_empty(),
and use buffer_is_unset() instead of buffer_is_empty(),
where buffer is known not to be NULL so that NULL check can be skipped
- use buffer_clen() instead of buffer_string_length() when buffer is
known not to be NULL (to avoid NULL check at runtime)
- use buffer_truncate() instead of buffer_string_set_length() to
truncate string, and use buffer_extend() to extend

Examples where buffer known not to be NULL:
- cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL
(though we might set it to NULL if buffer_is_blank(cpv->v.b))
- address of buffer is arg (&foo)
(compiler optimizer detects this in most, but not all, cases)
- buffer is checked for NULL earlier in func
- buffer is accessed in same scope without a NULL check (e.g. b->ptr)

internal behavior change:
callers must not pass a NULL buffer to some funcs.
- buffer_init_buffer() requires non-null args
- buffer_copy_buffer() requires non-null args
- buffer_append_string_buffer() requires non-null args
- buffer_string_space() requires non-null arg

show more ...

[multiple] strftime %F and %T

strftime %F for %Y-%m-%d, and %T for %H:%M:%S

[mod_accesslog] reformat numeric timestamp code

[mod_accesslog] strftime %z for numeric timestamp

[mod_accesslog] reformat numeric timestamp

[multiple] prefer monotonic time for internal use

Note: monotonic time does not change while VM is suspended

Continue to use real time where required by HTTP protocol, for logging
and for other use

[multiple] prefer monotonic time for internal use

Note: monotonic time does not change while VM is suspended

Continue to use real time where required by HTTP protocol, for logging
and for other user-visible instances, such as mod_status, as well as for
external databases and caches.

show more ...

[core] save parsed listen addrs at startup

save parsed listen addrs at startup for reuse at runtime

srv_socket->srv_token is normalized at startup and contains IP and port.
save offset to colon, if

[core] save parsed listen addrs at startup

save parsed listen addrs at startup for reuse at runtime

srv_socket->srv_token is normalized at startup and contains IP and port.
save offset to colon, if present, or else length of string (unix socket)

At runtime, srv_token_colon can be quickly used as length of IP string
(without port) or, if not length of string, offset of stringified port
following the colon.

show more ...

[mod_auth] close HTTP/2 connection after bad pass

mitigation slows down brute force password attacks

x-ref:
"Possible feature: authentication brute force hardening"
https://redmine.lighttpd.net

[mod_auth] close HTTP/2 connection after bad pass

mitigation slows down brute force password attacks

x-ref:
"Possible feature: authentication brute force hardening"
https://redmine.lighttpd.net/boards/3/topics/8885

show more ...