Make libfetch buildable.

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (nu

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after: 3 weeks

show more ...

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
pr

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

In fetch_resolve(), if the port number or service name is included in
the host argument (e.g. "www.freebsd.org:443"), the service pointer,
which is supposed to point to the port or service part, inst

In fetch_resolve(), if the port number or service name is included in
the host argument (e.g. "www.freebsd.org:443"), the service pointer,
which is supposed to point to the port or service part, instead points
to the separator, causing getaddrinfo() to fail.

Note that I have not been able to trigger this bug with fetch(1), nor
do I believe it is possible, as libfetch always parses the host:port
specification itself. I discovered it when I copied fetch_resolve()
into an unrelated project.

MFC after: 3 days

show more ...

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie

show more ...

libfetch: extra bounds checking through reallocarray(3).

Reviewed by: des
MFC after: 1 week

Properly indent a default: label and avoid crashing when running
under -v but cannot connect due to trying to print an int as %s [1].

Reported by: andrew [1]
MFC after: 3 days

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
$

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by: allanjude, des
Approved by: des
Differential Revision: https://reviews.freebsd.org/D9678

show more ...

More debugging code I missed in r309051.

Reported by: jbeich, jkim

Remove debugging code.

Refactor fetch_connect() and fetch_bind() to improve readability and avoid
repeating the same DNS lookups.

MFC after: 3 weeks

Don't leak addrinfo in fetch_bind()

Reported by: Coverity
CID: 1225038
MFC after: 1 week

lib: minor spelling fixes in comments.

No functional change.

Test for /etc/ssl/cert.pem existence to avoid masking SSL_CA_CERT_PATH

Prior to this patch, unless SSL_CA_CERT_FILE is set in the environment,
libfetch will set the CA file to "/usr/local/etc/cert.p

Test for /etc/ssl/cert.pem existence to avoid masking SSL_CA_CERT_PATH

Prior to this patch, unless SSL_CA_CERT_FILE is set in the environment,
libfetch will set the CA file to "/usr/local/etc/cert.pem" if it exists,
and to "/etc/ssl/cert.pem" otherwise. This has the consequence of
masking SSL_CA_CERT_PATH, because OpenSSL will ignore the CA path if a CA
file is set but fails to load (see X509_STORE_load_locations()).

While here, fall back to OpenSSL defaults if neither SSL_CA_CERT_FILE nor
SSL_CA_CERT_PATH are set in the environment, and if neither of the
libfetch default CA files exists.

PR: 193871
Submitted by: John W. O'Brien <[email protected]>
Approved by: des
MFC after: 1 week

show more ...

Fix non-POSIX-compliant use of getaddrinfo in libfetch

Submitted by: Boris Kolpackov <[email protected]>
Reviewed by: bapt
Approved by: bapt
MFC after: 1 week
Differential Revision: https://re

Fix non-POSIX-compliant use of getaddrinfo in libfetch

Submitted by: Boris Kolpackov <[email protected]>
Reviewed by: bapt
Approved by: bapt
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D3724

show more ...

Remove defunct SSLv2 support from fetch(1) and fetch(3).

As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0. Replace it with a more carefully
wrought patch:

- Switch the default for SSLv3 from

As pointed out by several people, r273114 was incorrect: it unconditionally
disabled everything except TLS 1.0. Replace it with a more carefully
wrought patch:

- Switch the default for SSLv3 from on to off
- Add environment variables to control TLS 1.1 and 1.2
- In verbose mode, report which version is used
- Update the man page to reflect these changes.

MFC after: 1 week

show more ...

Drop support for SSLv3.

MFC after: 1 week

Add a comment to explain the EAGAIN is only there for POSIX compliance

Resquested by: kib
Reviewed by: des

Support EAGAIN in fetch_writev

Reviewed by: des
Approved by: des

Look for root certificates in /usr/local/etc/ssl before /etc/ssl.

MFH: 1 week

Bump copyright dates

r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blo

r261230 broke the cases where the amount of data to be read is not
known in advance, or where the caller doesn't care and just keeps
reading until it hits EOF.

In fetch_read(): the socket is non-blocking, so read() will return 0
on EOF, and -1 (errno == EAGAIN) when the connection is still open but
there is no data waiting. In the first case, we should immediately
return 0. The EINTR case was also broken, although not in a way that
matters.

In fetch_writev(): use timersub() and timercmp() as in fetch_read().

In http_fillbuf(): set errno to a sensible value when an invalid chunk
header is encountered.

In http_readfn(): as in fetch_read(), a zero return from down the
stack indicates EOF, not an error. Furthermore, when io->error is
EINTR, clear it (but no errno) before returning so the caller can
retry after dealing with the interrupt.

MFC after: 3 days

show more ...

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of w

Solve http buffering issues and hangs once and for all (hopefully!) by
simply not trying to return exactly what the caller asked for - just
return whatever we got and let the caller be the judge of whether it
was enough. If an error occurs or the connection times out after we
already received some data, return a short read, under the assumption
that the next call will fail or time out before we read anything.

As it turns out, none of the code that calls fetch_read() assumes an
all-or-nothing result anyway, except for a couple of lines where we
read the CR LF at the end of a hunk in HTTP hunked encoding, so the
changes outside of fetch_read() and http_readfn() are minimal.

While there, replace select(2) with poll(2).

MFC after: 3 days

show more ...