Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

fetch(3): Add SOCKS5 support

This change adds SOCKS5 support to the library fetch(3) and updates the man
page.

Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY envi

fetch(3): Add SOCKS5 support

This change adds SOCKS5 support to the library fetch(3) and updates the man
page.

Details: Within the fetch_connect() function, fetch(3) checks if the
SOCKS5_PROXY environment variable is set. If so, it connects to this host
rather than the end-host. It then initializes the SOCKS5 connection in
accordance with RFC 1928 and returns the resulting conn_t (file descriptor)
for usage by the regular FTP/HTTP handlers.

Design Decision: This change defaults all DNS resolutions through the proxy
by sending all IPs as hostnames. Going forward, another feature might be to
create another environmental variable to toggle resolutions through the
proxy or not..

One may set the SOCKS5_PROXY environment variable in any of the formats:

SOCKS5_PROXY=proxy.example.com
SOCKS5_PROXY=proxy.example.com:1080
SOCKS5_PROXY=192.0.2.0
SOCKS5_PROXY=198.51.100.0:1080
SOCKS5_PROXY=[2001:db8::1]
SOCKS5_PROXY=[2001:db8::2]:1080

Then perform a request with fetch(1).

(note by kevans)
I've since been informed that Void Linux/xbps has a fork of libfetch that
also implements SOCKS5. I may compare/contrast the two in the mid-to-near
future.

Submitted by: Farhan Khan <farhan farhan codes>
Differential Revision: https://reviews.freebsd.org/D18908

show more ...

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
pr

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

Add a __printflike() that would have caught the bug fixed in r314396.

Refactor fetch_connect() and fetch_bind() to improve readability and avoid
repeating the same DNS lookups.

MFC after: 3 weeks

Add support for arbitrary http requests

Submitted by: Alex Hornung <[email protected]>
Reviewed by: des
Obtained from: Dragonfly
MFC after: 3 week

Bump copyright dates

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of w

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after: 3 days

show more ...

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <[email protected]>
MFC after: 1 week

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR: bin/153240
Submitted by: Mark <[email protected]>
MFC after: 3 weeks

show more ...

latin1 -> utf8

Update copyright dates and strip my middle name.

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting t

Add support for the NO_PROXY / no_proxy environment variable as used by
lynx, curl etc. Note that this patch differs significantly from that
in the PR, as the submitter refined it after submitting the PR.

PR: 110388
Submitted by: Alexander Pohoyda <[email protected]>
MFC after: 3 weeks

show more ...

Clean up namespace violations.

MFC after: 1 week

Update copyright years.

Add and document support for a FETCH_BIND_ADDRESS environment variable
specifying a local address to bind sockets to. Caveat: lightly tested.

PR: bin/37572

Experimental support for .netrc.

Introduce _fetch_writev(), which is the conn_t version of writev(2). In
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vec

Introduce _fetch_writev(), which is the conn_t version of writev(2). In
the SSL case, it is no different from the old _fetch_write(), but in the
non-SSL case it uses writev(2) to send the entire vector as a single
packet (provided it can fit in one packet). Implement _fetch_write()
and _fetch_putln() in terms of _fetch_writev().

This should improve performance in the non-SSL case (by reducing protocol
overhead) and solve the problem where too-smart-for-their-own-good
firewalls reject FTP packets that do not end in CRLF.

PR: bin/44123
Submitted by: fenner

show more ...

Add a reference count to struct fetchconn so we don't prematurely close and
free a cached FTP connection.

Make SSL support conditional on NOCRYPT.

Add SSL support + slight cleanup.

Submitted by: Henry Whincup <[email protected]> (in principle)

Wrap everything in struct connection, and enforce timeouts everywhere
(except for DNS operations). Always use funopen() for HTTP, to support
both timeouts and SSL.

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect o

First step towards SSL support: wrap connections in a 'struct connection'
which contains the socket descriptor, the input buffer and (yet unused)
SSL state variables. This has the neat side effect of greatly improving
reentrance (though we're not *quite* there yet) and opening the door to
HTTP connection caching.

This commit is inspired by email conversations with and patches from
Henry Whincup <[email protected]> last fall.

show more ...

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

show more ...