fetch: correct env var name for --crl option to work

Reviewed by: grembo
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D46973

(cherry picked from commit 1d7a33888e09f86feb4c2

fetch: correct env var name for --crl option to work

Reviewed by: grembo
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D46973

(cherry picked from commit 1d7a33888e09f86feb4c2c9f9640d936be07ce9d)

show more ...

MFC: fetch(1): suppress "Not Modified" in quiet mode

"fetch -qi" should skip printing "Not Modified" for successful
http(s) request. Still print it by default (v_level == 1).

(cherry picked from co

MFC: fetch(1): suppress "Not Modified" in quiet mode

"fetch -qi" should skip printing "Not Modified" for successful
http(s) request. Still print it by default (v_level == 1).

(cherry picked from commit 7a6309fdc79427b0a7a5c3876daba150d946ae22)

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is a 64 bit integer type for that reason.

While strtol() returns a 64 bit value on 64 LP64 architectures, it
is limit to 32 bit on e.g. i386. The strtoll() function returns a 64
but result on all supported architectures and therefore supports the
possible file lengths and file offsets on 32 bit archtectures.

Reported by: antoine
MFC after: 3 days

show more ...

fetch(1): more fixes for soft failure handling

Fix logic error introduced in my commit
bf599c03f09dea0f7e188e002b42d782af6841c3

Also, authorization errors should not be considered as soft failures.

fetch(1): fix error in previous commit

strncmp() compares terminating zero and sizeof() includes it.
Un-obsfuscate the code and show what it is indended to do.

fetch(1): do not consider HTTP 5XX errors as soft failures

This change fixes "fetch -a" looping forever on "502 Bad gateway"
error and similar.

MFC after: 1 month

fetch(1): correct progress accounting after previous commit

MFC after: 1 month

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

Thi

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

This makes it more robust against frequent but transient network failures.
For example:

=> sqlite-src-3370200.zip doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://www.sqlite.org/2022/sqlite-src-3370200.zip
sqlite-src-3370200.zip 3% of 12 MB 45 kBps 04m24s
fetch: sqlite-src-3370200.zip appears to be truncated: 524288/13145234 bytes
sqlite-src-3370200.zip 10% of 12 MB 67 kBps 02m56s
fetch: sqlite-src-3370200.zip appears to be truncated: 1327104/13145234 bytes
sqlite-src-3370200.zip 28% of 12 MB 123 kBps 01m14s
fetch: sqlite-src-3370200.zip appears to be truncated: 3735552/13145234 bytes
sqlite-src-3370200.zip 54% of 12 MB 253 kBps 24s
fetch: sqlite-src-3370200.zip appears to be truncated: 7176192/13145234 bytes
sqlite-src-3370200.zip 62% of 12 MB 90 kBps 55s
fetch: sqlite-src-3370200.zip appears to be truncated: 8241152/13145234 bytes
sqlite-src-3370200.zip 82% of 12 MB 113 kBps 20s
fetch: sqlite-src-3370200.zip appears to be truncated: 10862592/13145234 bytes
sqlite-src-3370200.zip 12 MB 185 kBps 12s
===> Fetching all distfiles required by sqlite3-3.37.2,1 for building

MFC after: 1 month

show more ...

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.
Reorganize the code to use caller-provided buffers, tweak the ETA
display a bit, and reduce the visual differences between the tty and
non-tty end-of-transfer displays.

PR: 202424
Approved by: re (gjb@)

show more ...

Support If-Modified-Since for https as well as http.

PR: 224426
Submitted by: [email protected]
MFC after: 1 week

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Remove defunct SSLv2 support from fetch(1) and fetch(3).

Increase the default (and minimum) buffer size from 4 kB to 16 kB. Also,
propagate the buffer size to libc, which uses a 1 kB buffer by default,
negating any hypothetical benefit of increasing fetch

Increase the default (and minimum) buffer size from 4 kB to 16 kB. Also,
propagate the buffer size to libc, which uses a 1 kB buffer by default,
negating any hypothetical benefit of increasing fetch(1)'s buffer size.

MFC after: 3 days

show more ...

whitespace and bump copyright

Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR: kern/175514
Submitted by: Michael Gmelin <[email protected]>
MFC after: 1 week

Correct check to see if file changes.

Reported by: [email protected]
Reviewed by: des

Fix bandwidth reporting when doing a restarted download with "-r".
The offset is already accounted for in xs->lastrcvd and doesn't
have to be subtracted again.

Reported by: Florian Smeets <flo@smeet

Fix bandwidth reporting when doing a restarted download with "-r".
The offset is already accounted for in xs->lastrcvd and doesn't
have to be subtracted again.

Reported by: Florian Smeets <[email protected]>
Submitted by: Mateusz Guzik <[email protected]>
Tested by: Florian Smeets <[email protected]>
MFC after: 1 week

show more ...

Add check for failure of mkstemp and setenv.

Reviewed by: des
Approved by: cperciva (implicit)
Obtained from: DragonFlyBSD
MFC after: 1 week

Change fetch(1) to:

o Report the instantaneous bandwidth instead of an average since the
beginning of the download.

o At the finish of the download report the average bandwidth and also
the tot

Change fetch(1) to:

o Report the instantaneous bandwidth instead of an average since the
beginning of the download.

o At the finish of the download report the average bandwidth and also
the total time it took instead of 00m00s.

Reviewed by: des
MFC after: 1 week

show more ...

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variabl

More -Wmissing-variable-declarations fixes.

In addition to adding `static' where possible:

- bin/date: Move `retval' into extern.h to make it visible to date.c.
- bin/ed: Move globally used variables into ed.h.
- sbin/camcontrol: Move `verbose' into camcontrol.h and fix shadow warnings.
- usr.bin/calendar: Remove unneeded variables.
- usr.bin/chat: Make `line' local instead of global.
- usr.bin/elfdump: Comment out unneeded function.
- usr.bin/rlogin: Use _Noreturn instead of __dead2.
- usr.bin/tset: Pull `Ospeed' into extern.h.
- usr.sbin/mfiutil: Put global variables in mfiutil.h.
- usr.sbin/pkg: Remove unused `os_corres'.
- usr.sbin/quotaon, usr.sbin/repquota: Remove unused `qfname'.

show more ...

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it

Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information. The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal. The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR: bin/153240
Submitted by: Mark <[email protected]>
MFC after: 3 weeks

show more ...

Document the fact that passive mode is now the default.
Update copyright dates and strip my middle name.

Use fseeko() instead of fseek(). The rest of the code is off_t-aware,
but the use of fseek() means fetch(1) can't correctly resume a transfer
that was interrupted past the 2 GB mark.

Pointed out by

Use fseeko() instead of fseek(). The rest of the code is off_t-aware,
but the use of fseek() means fetch(1) can't correctly resume a transfer
that was interrupted past the 2 GB mark.

Pointed out by: ache@
MFC after: 3 weeks

show more ...

Followup to r225599: the fseek() was a no-op since the file was opened
in append mode. Open it in read-write mode instead. Also move the
fseek up one level to cover the (unlikely but not impossible

Followup to r225599: the fseek() was a no-op since the file was opened
in append mode. Open it in read-write mode instead. Also move the
fseek up one level to cover the (unlikely but not impossible) case where
the server accepts ranges but does not send a Content-Size header.

PR: bin/117277
MFC after: 3 weeks

show more ...