pkgbase: don't remove the 'latest' symlink until the repo is final

We can encounter many issues in the pkg-repo(8) process which would
currently leave us with no more 'latest' symlink in the reposit

pkgbase: don't remove the 'latest' symlink until the repo is final

We can encounter many issues in the pkg-repo(8) process which would
currently leave us with no more 'latest' symlink in the repository. The
main problems with this are that we've now broken a subsequent
`update-packages` because we can't determine a PKG_VERSION_FROM, but
also we break configured clients that are still expecting to see *some*
repository.

Switch to just replacing the `latest` symlink entirely after we have
made it past the pkg-repo(8) step so that we only swap over when we have
a finished repository.

Reviewed by: bapt, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D47303

(cherry picked from commit 8549e3ce0cd6d4a6187b3977d53ff56ce0ac9c5b)

show more ...

pkgbase: Make src package creation recipes more precise

Just remove the plist created by the respective rule. Otherwise the two
receipes can race with each other.

Fixes: d7d5c9efef03 ("pkgbase: Le

pkgbase: Make src package creation recipes more precise

Just remove the plist created by the respective rule. Otherwise the two
receipes can race with each other.

Fixes: d7d5c9efef03 ("pkgbase: Let source packages be built in parallel")
Reviewed by: bapt, emaste
Reported by: Mark Millard <[email protected]>
Differential Revision: https://reviews.freebsd.org/D46320

(cherry picked from commit d02dcf21eea3973a714294b011537c2af6c747fa)

show more ...

pkgbase: Let source packages be built in parallel

To build the packages target, we build src and src-sys packages
containing the source code from which the repo was built. These
packages take signi

pkgbase: Let source packages be built in parallel

To build the packages target, we build src and src-sys packages
containing the source code from which the repo was built. These
packages take significantly longer than the others, presumably because
they contain many more files. Because both source packages are built
to satisfy the same target, they end up being built serially. Split
them into separate subtargets so that they can run in parallel. This
saves a couple of minutes on my build machine.

Reviewed by: manu, emaste
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D46288

(cherry picked from commit d7d5c9efef031aa11a841d1836f7c937dcbe4ec8)

show more ...

pkgbase: Unify pkg ABI handling for pkgbase targets

Right now, to get the pkg ABI we either use PKG_ABI, derived from
newvers.sh, or use an ABI file from the staged world. This
inconsistency is con

pkgbase: Unify pkg ABI handling for pkgbase targets

Right now, to get the pkg ABI we either use PKG_ABI, derived from
newvers.sh, or use an ABI file from the staged world. This
inconsistency is confusing and can cause problems.

Switch to a single source of truth: use an ABI file from the worldstage
dir to get the ABI of pkgbase packages. In particular, we do not need
to know the ABI until staging is done. More specifically:
- use a shell command to define PKG_ABI,
- replace inline uses of ABI_FILE,
- run sign-packages in a subshell (this was already done for the
update-packages target) so that the staging targets are done before we
try to evaluate the ABI.

Reviewed by: manu
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D46287

(cherry picked from commit b118b6eb4cb7520eb348a6ac965b077fc5179fde)

show more ...

Makefile.inc1: Fix typo affecting incremental pkgbase builds.

Fixes: 4231a5e50404 release: don't keep old packages if the BRANCH changes
Sponsored by: Chelsio Communications

(cherry picked from com

Makefile.inc1: Fix typo affecting incremental pkgbase builds.

Fixes: 4231a5e50404 release: don't keep old packages if the BRANCH changes
Sponsored by: Chelsio Communications

(cherry picked from commit aee4c9c5b88adc899b2e155a1c144acf96719f1e)

show more ...

release: don't keep old packages if the BRANCH changes

(cherry picked from commit 4231a5e5040403f4ae46a63bf6e1209d5eb7adb8)

pkgbase: Create two new rules for creating repo

This adds two new rules named create-packages-kernel-repo and
create-packages-world-repo.
The goal of those rules is to create the {kernel,world} pack

pkgbase: Create two new rules for creating repo

This adds two new rules named create-packages-kernel-repo and
create-packages-world-repo.
The goal of those rules is to create the {kernel,world} packages and
after that the repository.
It helps a lot for developing with pkgbase by adding the dev machine
repository created by those rules on top of the official pkgbase one.

Differential Revision: https://reviews.freebsd.org/D43623
Reviewed by: bapt, emaste
Sponsored by: Beckhoff Automation GmbH & Co. KG

(cherry picked from commit a5ff900535e24e3a3cd02287d733e1d1e5e9583a)

show more ...

dtrace: Add the 'oformat' libdtrace option

This option can be used to specify a format to use in DTrace output.
The following formats are supported:
- json
- xml
- html
- none (default DTrace ou

dtrace: Add the 'oformat' libdtrace option

This option can be used to specify a format to use in DTrace output.
The following formats are supported:
- json
- xml
- html
- none (default DTrace output)

This is implemented using libxo and integrated into libdtrace. Client
code only works with the following API:

- dtrace_oformat_setup(dtrace_hdl_t *) -- to be called when output is starting.
- dtrace_oformat_teardown(dtrace_hdl_t *) -- to be called when output is finished
- dtrace_oformat(dtrace_hdl_t *) -- check if oformat is enabled.
- dtrace_set_outfp(FILE *) -- sets the output file for oformat.
- Ensure that oformat is correctly checked in the drop handler and record
processing callbacks.

This commit also adds tests which check if the generated output is
valid (JSON, XML) and extends the dtrace(1) describing the structured output.

Reviewed by: markj
Discussed with: phil
MFC after: 2 months
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41745

(cherry picked from commit 93f27766a7e1af009c5b1e4ca538632857c91aa1)

show more ...

pkgbase: add a mechanism to be able to force a give ucl include

This is made in order to be able to find add the post-install scripts
for the kernel, where PKGNAME varies for each KERNCONF but we do

pkgbase: add a mechanism to be able to force a give ucl include

This is made in order to be able to find add the post-install scripts
for the kernel, where PKGNAME varies for each KERNCONF but we don't want
to dynamically duplicated the kernel.ucl file.

At the same time we don't want the *-dbg* packages to actually include
those post-install scripts

(cherry picked from commit 45d83c39262d9e2727c6f409e19c653f19a5d7cc)

show more ...

pkgbase: fix typo preventing packaging kernels

(cherry picked from commit 2addba5caf92b78d5d32835e60ec28df7eb05aef)

pkgbase: fix kernel packing after switch ucl rework

(cherry picked from commit 76229061601f495d3c80fdca5914e394e71ec07a)

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which u

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which uses libucl to
ingest the template and use variables as defined in its command line.

the template will include only if it exist a ucl file named after the
package name which will complement the template or overwrite what was
defined in the template if defined in this specific ucl file

this allows to overwrite license, but add script only to the packages
who actually needs them.

As a results the post install scripts are now only added to the right
package and not also added to the subpackages like -man or -dev

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44374

(cherry picked from commit f8e46fd61e055fcc767c2038d867f58c38e43ec3)

show more ...

Remove MOVED_LIBS handling from list-old-libs

In 922337e8d398 I added MOVED_LIBS into list-old-files, so that
delete-old-files would remove the old /usr/lib/libc++.so.1 as soon as
possible (after th

Remove MOVED_LIBS handling from list-old-libs

In 922337e8d398 I added MOVED_LIBS into list-old-files, so that
delete-old-files would remove the old /usr/lib/libc++.so.1 as soon as
possible (after the library moved to /lib).

I left it in list-old-libs in case a user updated their src tree between
delete-old-files and delete-old-libs. Now that some time has passed,
tremove the redundant MOVED_LIBS entry.

PR: 272642
Sponsored by: The FreeBSD Foundation

(cherry picked from commit f38bad0ab0fca9b1f665b2ba79098edf993c0854)

show more ...

pkgbase: pass --recurse-submodules to `git ls-files`

When generating source packages. Although submodules are not used by
FreeBSD itself they may be used by downstream projects. By default
for sub

pkgbase: pass --recurse-submodules to `git ls-files`

When generating source packages. Although submodules are not used by
FreeBSD itself they may be used by downstream projects. By default
for submodules `git ls-files` just emits the submodule directory name,
which resulted in:

pkg: pkg_checksum_hash_sha256_file(read failed): Is a directory

Passing --recurse-submodules lists all of the files in each submodule
(which is desired when submodules are in use), and has no effect when
submodules are not present.

Reviewed by: bapt, manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42983

(cherry picked from commit 3b1904d9eb0468a49be3cd1d97de6d7ecaa66a43)

show more ...

Make kldxref a bootstrap tool and use unconditionally

Now that kldxref is a generic cross tool and can be built on non-FreeBSD
we can bootstrap it during the build and thus remove the condition for

Make kldxref a bootstrap tool and use unconditionally

Now that kldxref is a generic cross tool and can be built on non-FreeBSD
we can bootstrap it during the build and thus remove the condition for
whether it exists. We also need to make sure to add it to the METALOG
for -DNO_ROOT builds.

Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D43051

(cherry picked from commit ff7c12c1f17e52337169d8ce48433fae96a9c9db)

show more ...

Makefile.inc1: Forward on METALOG and DISTBASE for kernel targets

Currently IMAKE_INSTALL, which includes -M METALOG, is enough for the
sub-makes to work, but using kldxref for -DNO_ROOT builds will

Makefile.inc1: Forward on METALOG and DISTBASE for kernel targets

Currently IMAKE_INSTALL, which includes -M METALOG, is enough for the
sub-makes to work, but using kldxref for -DNO_ROOT builds will require
manually adding linker.hints to the METALOG, and thus both METALOG
itself and DISTBASE need to be exposed directly to the sub-makes, so do
so.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D43050

(cherry picked from commit e131d3f4f34fdbce465524f7b1093f01acc9de7c)

show more ...

Merge llvm-project main llvmorg-17-init-19304-gd0b54bb50e51

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvm-project main llvmorg-17-init-19304-gd0b54bb50e51, t

Merge llvm-project main llvmorg-17-init-19304-gd0b54bb50e51

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvm-project main llvmorg-17-init-19304-gd0b54bb50e51, the
last commit before the upstream release/17.x branch was created.

PR: 273753
MFC after: 1 month

(cherry picked from commit 06c3fb2749bda94cb5201f81ffdb8fa6c3161b2e)

show more ...

pkgbase: propagate SRCRELDATE to the packages correctly

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42892

(cherry picked from commit 5faaa602cee093269b

pkgbase: propagate SRCRELDATE to the packages correctly

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42892

(cherry picked from commit 5faaa602cee093269b1a73156c95c6892b4f098d)

show more ...

pkgbase: create source package

FreeBSD-src for all the sources but the kernel
FreeBSD-src-sys just for the kernel

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebs

pkgbase: create source package

FreeBSD-src for all the sources but the kernel
FreeBSD-src-sys just for the kernel

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42651

(cherry picked from commit 99b8c0c35b0fcc633649209621243d678a13542a)

show more ...

pkgbase: create package with SRCRELDATE not OSRELDATE as OSVERSION

In 188fe88ec50eac7c10e1d8350bf1180f8c16e463 OSVERSION has been set
to OSRELDATE which is the RELDATE of the building OS while we wa

pkgbase: create package with SRCRELDATE not OSRELDATE as OSVERSION

In 188fe88ec50eac7c10e1d8350bf1180f8c16e463 OSVERSION has been set
to OSRELDATE which is the RELDATE of the building OS while we wanted
to use SRCRELDATE which is the RELDATE of the target system

(cherry picked from commit 051d69d6f8b51dc1d7bba58af99be76185312222)

show more ...

pkgbase: force OSVERSION

This is used to the package annotation helping pkg to know about
backward compatibility is set to the version of the packages not
the version of the host building the packag

pkgbase: force OSVERSION

This is used to the package annotation helping pkg to know about
backward compatibility is set to the version of the packages not
the version of the host building the packages

(cherry picked from commit 188fe88ec50eac7c10e1d8350bf1180f8c16e463)

show more ...

pkgbase: set a default set of kernel for when PACKAGE_BUILDING=1

PACKAGE_BUILDING is already known in the ports tree as a variable
use to defined when the packages is being actually built in an
auto

pkgbase: set a default set of kernel for when PACKAGE_BUILDING=1

PACKAGE_BUILDING is already known in the ports tree as a variable
use to defined when the packages is being actually built in an
automation process, reuse that variable to define the default set
of kernel we plan to build for the default pkgbase.

Reviewed by: manu, jrtc27
Accepted by: manu
MFC After: 1 week
Differential Revision: https://reviews.freebsd.org/D42444

(cherry picked from commit df6e7e11c5f660cbd5381819718e3c947325fa2e)

show more ...

pkgbase: compress packages with zstandard

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42375

(cherry picked from commit 820fbcf25bb2c7f1c570c0d408b2ab7d05

pkgbase: compress packages with zstandard

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42375

(cherry picked from commit 820fbcf25bb2c7f1c570c0d408b2ab7d05bd771c)

show more ...

pkgbase: use <major>.<minor> format for alpha, beta and release package versions

This is consistent with version numbers used in releng/13.2.

PR: 275051
Reviewed by: bapt
MFC after: 3 days
Differe

pkgbase: use <major>.<minor> format for alpha, beta and release package versions

This is consistent with version numbers used in releng/13.2.

PR: 275051
Reviewed by: bapt
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42562

(cherry picked from commit 21e9018ae19662db643a21064150da866bc7beb4)

show more ...

ASan: Add asan_static to build

This is a partial fix for building with -DWITH_ASAN.

Reviewed by: Fangrui Song, dim, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D4238

ASan: Add asan_static to build

This is a partial fix for building with -DWITH_ASAN.

Reviewed by: Fangrui Song, dim, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42388

(cherry picked from commit 5c783a0b2826d1c6bbaa7d4b01a46d1dcdb39e8c)

show more ...