libfetch: don't include fragments in HTTP requests

Fragments are reserved for client-side processing, see
https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1

Also, some servers don't like to re

libfetch: don't include fragments in HTTP requests

Fragments are reserved for client-side processing, see
https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1

Also, some servers don't like to receive HTTP requests with fragments.

```
$ fetch 'https://dropbox.com/a/b'
fetch: https://dropbox.com/a/b: Not Found

$ fetch 'https://dropbox.com/a/b#'
fetch: https://dropbox.com/a/b#: Bad Request
```

This is a real-world scenario, where some download link from dropbox
(eventually) redirects to an URL with a fragment:

```
$ fetch -v 'https://www.dropbox.com/sh/<some>/<thing>?dl=1' 2>&1 | grep requesting
requesting https://www.dropbox.com/sh/<some>/<thing>?dl=1
requesting https://www.dropbox.com/scl/fo/<foo>/<bar>?rlkey=<baz>&dl=1
requesting https://<boo>.dl.dropboxusercontent.com/zip_download_get/<some-long-strig>#
```

See how the last redirect ends with a `#`.

Currently, libfetch includes the ending fragment and makes it impossible
to download the file.

Differential Revision: https://reviews.freebsd.org/D46318
MFC after: 2 weeks

(cherry picked from commit 1af7d5f389536a2f391153513d95d92ffdf360e4)

show more ...

libfetch: parse scheme://domain:/ correctly

This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a24.

Sponsored by: Juniper Networks,

libfetch: parse scheme://domain:/ correctly

This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a24.

Sponsored by: Juniper Networks, Inc.
Reviewed by: des
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D44493

(cherry picked from commit fb860ed0c52c2c1e7792ef86718620a439663c7f)

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

libfetch: disallow invalid escape sequences

Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL.

Suggested by: Matthew Dillon
Reviewed by: Matthew Dillon
MFC after: 1 week

Fix urldecode buffer overrun.

Reported by: Duncan Overbruck
Security: CVE-2020-7450

Improve URL parsing. In particular, convert scheme and host to lowercase.

MFC after: 1 week

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (nu

Fix an inverted conditional in the netrc code, which would ignore the
value of $HOME and always use the home directory from the passwd
database, unless $HOME was unset, in which case it would use (null).

While there, clean up handling of netrcfd and add debugging aids.

MFC after: 3 weeks

show more ...

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
pr

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie

show more ...

Properly initialize netrcfd in fetchParseURL

This fixes ftp with fetch(1) which was broken after r313974

Submitted by: dim
Reported by: olivier
Pointyhat to: bapt

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
$

Add a file descriptor in struct url for netrc

When using libfetch in an application that drops privileges when fetching
like pkg(8) then user complain because the application does not read anymore
${HOME}/.netrc. Now a caller can prepare a fd to the said file and manually
assign it to the structure.

It is also a first step to allow to capsicumize libfetch applications

Reviewed by: allanjude, des
Approved by: des
Differential Revision: https://reviews.freebsd.org/D9678

show more ...

Fix -Wunsequenced warning.

Submitted by: [email protected]

Support percent-encoded user and password

RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters

Support percent-encoded user and password

RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.

Reviewed by: rstone@
MFC after: 1 month

show more ...

latin1 -> utf8

Update copyright dates and strip my middle name.

1. Update fetch to consistently return 1 on error, as the man page states,
rather than usually returning 1 but in a few instances using a sysexits(3)
return value.

2. Remove a few unused varia

1. Update fetch to consistently return 1 on error, as the man page states,
rather than usually returning 1 but in a few instances using a sysexits(3)
return value.

2. Remove a few unused variables from libfetch.

PR: docs/122470 (1, only)
Reviewed by: des

show more ...

As several people pointed out, I did all the ctype casts the wrong
way (not for the first time...)

Noticed by: bde, ru ++
MFC after: 1 week

Old patch I had lying around: correctly cast the argument to is*().
IWBNI gcc could warn about this the way it warns about printf() abuse.

MFC after: 1 week

Clean up namespace violations.

MFC after: 1 week

Update copyright years.

style(9): add parentheses to sizeof even when not strictly required.

MFC after: 3 days

Initialization paranoia.

Add SSL support + slight cleanup.

Submitted by: Henry Whincup <[email protected]> (in principle)

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later

Reindent, and add parentheses to return statements. Some functions in
ftp.c and http.c now have exceedingly long lines due to deep nesting;
this will be corrected by reorganizing the code in a later revision.

show more ...