MFC r351573:
Document fetchReqHTTP().

Fix a few (but far from all) style issues.

MFC after: 3 weeks

Use __VA_ARGS__ to simplify the DEBUG macro.

MFC after: 3 weeks

Preserve if-modified-since timestamps across redirects.

PR: 224426
MFC after: 1 week

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
pr

lib: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

show more ...

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals

r308996 broke IP literals by assuming that a colon could only occur as
a separator between host and port, and using strchr() to search for it.
Rewrite fetch_resolve() so it handles bracketed literals correctly, and
remove similar code elsewhere to avoid passing unbracketed literals to
fetch_resolve(). Remove #ifdef INET6 so we still parse IP literals
correctly even if we do not have the ability to connect to them.

While there, fix an off-by-one error which caused HTTP 400 errors to be
misinterpreted as redirects.

PR: 217723
MFC after: 1 week
Reported by: bapt, bz, cem, ngie

show more ...

Fix partial requests (used by fetch -r) when the requested file is
already complete.

Since 416 is an error code, any Content-Range header in the response
would refer to the error message, not the re

Fix partial requests (used by fetch -r) when the requested file is
already complete.

Since 416 is an error code, any Content-Range header in the response
would refer to the error message, not the requested document, so
relying on the value of size when we know we got a 416 is wrong.
Instead, just verify that offset == 0 and assume that we've reached
the end of the document (if offset > 0, we did not request a range,
and the server is screwing with us). Note that we cannot distinguish
between reaching the end and going past it, but that is a flaw in the
protocol, not in the code, so we just have to assume that the caller
knows what it's doing. A smart caller would request an offset
slightly before what it believes is the end and compare the result to
what is already in the file.

PR: 212065
Reported by: mandree
MFC after: 3 weeks

show more ...

Fix inverted loop condition which broke multi-line responses to CONNECT.

PR: 194483
Submitted by: Miłosz Kaniewski <[email protected]>
MFC after: 1 week

r169386 (PR 112515) was incomplete: it treated 307 as an error except
in verbose mode, and did not handle 308 at all.

r241840 (PR 172451) added support for 308, but with the same bug.

Correctly han

r169386 (PR 112515) was incomplete: it treated 307 as an error except
in verbose mode, and did not handle 308 at all.

r241840 (PR 172451) added support for 308, but with the same bug.

Correctly handle both by recognizing them as redirects in all places
where we check the HTTP result code.

PR: 112515 173451 209546
Submitted by: novel@
MFC after: 1 week

show more ...

Use strlcpy() instead of strncpy() to copy the string returned by
setlocale() so that static analyzers know that the string is NUL
terminated. This was causing a false positive in Coverity even
thou

Use strlcpy() instead of strncpy() to copy the string returned by
setlocale() so that static analyzers know that the string is NUL
terminated. This was causing a false positive in Coverity even
though the longest string returned by setlocale() is ENCODING_LEN
(31) and we are copying into a 64 byte buffer. This change is also
a bit of an optimization since we don't need the strncpy() feature
of padding the rest of the destination buffer with NUL characters.

Reported by: Coverity
CID: 974654

show more ...

Fix double-free error: r289419 moved all error handling in http_connect()
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-h

Fix double-free error: r289419 moved all error handling in http_connect()
to the end of the function, but did not remove a fetch_close() call which
was made redundant by the one in the shared error-handling code.

PR: 206774
Submitted by: Christian Heckendorf <[email protected]>
MFC after: 3 days

show more ...

As a followup to r292330, standardize on size_t and add a few comments.

Reset bufpos to 0 immediately after refilling the buffer. Otherwise, we
risk leaving the connection in an indeterminate state if the server fails
to send a chunk delimiter. Depending on the applica

Reset bufpos to 0 immediately after refilling the buffer. Otherwise, we
risk leaving the connection in an indeterminate state if the server fails
to send a chunk delimiter. Depending on the application and on the sizes
of the preceding chunks, the result can be anything from missing data to a
segfault. With this patch, it will be reported as a protocol error.

PR: 204771
MFC after: 1 week

show more ...

Fix buildworld after r291453, similar to r284346: url->user and url->pwd
are arrays, so they can never be NULL.

Reported by: many
Pointy hat to: des

Use .netrc for HTTP sites and proxies, not just FTP.

PR: 193740
Submitted by: TEUBEL György <[email protected]>
MFC after: 1 week

Fix two bugs in HTTPS tunnelling:
- If the proxy returns a non-200 result, set the error code accordingly
so the caller / user gets a somewhat meaningful error message.
- Consume and discard any

Fix two bugs in HTTPS tunnelling:
- If the proxy returns a non-200 result, set the error code accordingly
so the caller / user gets a somewhat meaningful error message.
- Consume and discard any HTTP response header following the result line.

PR: 194483
Tested by: Fabian Keil <[email protected]>
MFC after: 1 week

show more ...

Remove unused variable to silence clang warning.

Differential Revision: D2683
Reviewed by: rodrigc, bapt

Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:

lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conver

Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:

lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = purl->user ?
~~~~~~^~~~ ~
lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = purl->pwd?
~~~~~~^~~~
lib/libfetch/http.c:1657:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1659:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~
lib/libfetch/http.c:1669:25: error: address of array 'url->user'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.user = url->user ?
~~~~~^~~~ ~
lib/libfetch/http.c:1671:29: error: address of array 'url->pwd'
will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
aparams.password = url->pwd ?
~~~~~^~~ ~

Since url->user and url->pwd are arrays, they can never be NULL, so the
checks can be removed.

Reviewed by: bapt
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D2673

show more ...

Tentatively fix a few libfetch warnings, see this review for the
details: https://reviews.freebsd.org/D2673

Add support for arbitrary http requests

Submitted by: Alex Hornung <[email protected]>
Reviewed by: des
Obtained from: Dragonfly
MFC after: 3 week

Remove unnecessary semicolons

Patch by Sascha Wildner <[email protected]> for Dragonfly

Reviewed by: des
Obtained from: Dragonfly
MFC after: 1 week

Use NULL instead of 0

Patch by Sascha Wildner <[email protected]> for Dragonfly

Reviewed by: des
Obtained from: Dragonfly
MFC after: 1 week

If HTTP_USER_AGENT is defined but empty, don't send User-Agent at all.

PR: 184507
Submitted by: [email protected] (with modifications)
MFC after: 1 week

Support Last-Modified behind proxies which return UTC instead of GMT.

The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg

Support Last-Modified behind proxies which return UTC instead of GMT.

The standard states that GMT must be used, but that UTC is equivalent. Still
parse UTC as otherwise this causes problems for pkg(8). It will refetch
the repository every time 'pkg update' or other remote operations
are used behind these proxies.

RFC2616: "All HTTP date/time stamps MUST be represented in Greenwich Mean
Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal
to UTC (Coordinated Universal Time).""

Approved by: bapt (mentor)
Reviewed by: des, peter
Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

show more ...

Bump copyright dates