Revert "loader/lua: Remove workaround for command_error"

This reverts commit 552f3072af54820cf1805f712e2567bc1b7f046d.

loader.command_error was added just after 11.2, but appears to not have
been b

Revert "loader/lua: Remove workaround for command_error"

This reverts commit 552f3072af54820cf1805f712e2567bc1b7f046d.

loader.command_error was added just after 11.2, but appears to not have
been back ported to 11.x. 11.0 was the first lua loader release, so keep
this compat shim until we sort out what to do.

MFC After: 3 days
Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D45883

(cherry picked from commit 809edb3636a5494ad7d3dfff910e0aab89d83442)

show more ...

loader/lua: Remove workaround for command_error

loader.command_error was available prior to stable/12 branching. No need
to check if it is available or not.

Sponsored by: Netflix
Reviewed by: kev

loader/lua: Remove workaround for command_error

loader.command_error was available prior to stable/12 branching. No need
to check if it is available or not.

Sponsored by: Netflix
Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D44144

(cherry picked from commit 552f3072af54820cf1805f712e2567bc1b7f046d)

show more ...

stand/lua: always allow overriding with local config files

Loader now also read configuration files listed in local_loader_conf_files.
Files listed here are the last ones read. And /boot/loader.conf

stand/lua: always allow overriding with local config files

Loader now also read configuration files listed in local_loader_conf_files.
Files listed here are the last ones read. And /boot/loader.conf.local was
moved from loader_conf_files to local_loader_conf_files leaving only
loader.conf and device.hints in loader_conf_files by default.

The idea is to ensure local_loader_conf_files, i.e., /boot/loader.conf.local,
can always be used to override other user defined settings.

So the sequencing is now as follow:

1. Bootstrap:
/boot/defaults/loader.conf

2. Read loader_conf_files files:
/boot/device.hints
/boot/loader.conf

3. Read loader_conf_dirs files:
/boot/loader.conf.d/*.conf

4. And finally, rread local_loader_conf_files files:
/boot/loader.conf.local

Reviewed by: imp, kevans
Pull Request: https://github.com/freebsd/freebsd-src/pull/759

[[ The commit, revert, recommit has been squashed down to record the merge ]]
(cherry picked from commit c475e61f66fe8fe939e18ec7821c2340569f3271)
(cherry picked from commit 5fdf01dbeef1f64f8c446561498d662702451ac1)
(cherry picked from commit d3d0b735571d9562812ce5b343a6e91f7a795dbe)

show more ...

stand/lua: per-product conf if requested via product_vars

If product_vars is set, it must be a space separated list of environment
variable names to walk through to guess the product. Each time a pr

stand/lua: per-product conf if requested via product_vars

If product_vars is set, it must be a space separated list of environment
variable names to walk through to guess the product. Each time a product can be
guessed (i.e., the corresponding variable is defined), prepend
/boot/loader.conf.d/PRODUCT/ to loader_conf_dirs.

It can be typically used as follow:

smbios.system.planar.maker="PLANAR_MAKER"
smbios.system.planar.product="PLANAR_PRODUCT"
smbios.system.product="PRODUCT"
uboot.m_product="M_PRODUCT"
product_vars="smbios.system.planar.maker smbios.system.planar.product smbios.system.product uboot.m_product"

to read files found in the following directories, in that order:

/boot/loader.conf.d/PLANAR_MAKER
/boot/loader.conf.d/PLANAR_PRODUCT
/boot/loader.conf.d/PRODUCT
/boot/loader.conf.d/M_PRODUCT

Reviewed by: imp, kevans
Pull Request: https://github.com/freebsd/freebsd-src/pull/759

[[ The commit, revert, commit has been squashed dpwn ]]
(cherry picked from commit c343eedc6db381ac76e489f8ae0304898b71c5db)
(cherry picked from commit 994865caf250ea2a59b7b842e44680931e8b19f6)
(cherry picked from commit 754cac4b283eb024a3a6a194130199c860e32ebf)

show more ...

stand: lua: module names should permit more

The module entries should generally allow whatever is allowed as an
env_var in the pattern table. Notably, we're missing periods which
would allow proper

stand: lua: module names should permit more

The module entries should generally allow whatever is allowed as an
env_var in the pattern table. Notably, we're missing periods which
would allow proper entries for .dtb files in loader.conf that don't need
to specify a module_name entry for it.

%d in this expression is actually redundant as %w is actually
"all alphanumerics," but I've included it for now to match the env_var
entry. We should really remove it from both.

Reported by: "aribi" on the forums via allanjude@

(cherry picked from commit 5bc1e0c2285e73fe8455bb6c72b2b40e33f5477e)

show more ...

Remove $FreeBSD$: two-line lua tag

Remove /^--\n--\s*\$FreeBSD\$.*$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

lualoader: add support for .lua configuration files

If a file is specified in loader_conf_files that ends in '.lua', lualoader
will now load and execute that file. These may be used in place of a
tr

lualoader: add support for .lua configuration files

If a file is specified in loader_conf_files that ends in '.lua', lualoader
will now load and execute that file. These may be used in place of a
traditional loader.conf to use more complicated logic, where some values
may be set based on others or based on the environment that the C bits has
left us with.

Lua scripts are run in a limited environment. In particular, it does not get
access to any modules or, in-fact, anything except environment variable.

A config.buildenv hook has been added so that a local module can add
whatever it may need to to the environment.

When a global var is set in the lua script, it does not immediately alter
the loader environment. Instead, the script's environment is initially
empty and processed only if the whole script executes successfully.
Effectively, a lua configuration file either takes effect or it does not,
an error will not leave it in a half-baked state.

Reviewed by: bcr (manpages), imp
Differential Revision: https://reviews.freebsd.org/D28450

show more ...

lua: Export loaded modules env vars via environment

To make auditing and debugging easier, export foo_load=XXX foo_name=yyy
etc to the loader env that we export to the kernel.

Original by: dhw
Spo

lua: Export loaded modules env vars via environment

To make auditing and debugging easier, export foo_load=XXX foo_name=yyy
etc to the loader env that we export to the kernel.

Original by: dhw
Sponsored by: Netflix
Discussed with: kevans
Differential Revsion: https://reviews.freebsd.org/D38466

show more ...

lualoader: position hyphens at the beginning of character classes

According to the Lua 5.4 manual section 6.4.1 ("Patterns"), the interaction
between ranges and classes is not defined and hyphens mu

lualoader: position hyphens at the beginning of character classes

According to the Lua 5.4 manual section 6.4.1 ("Patterns"), the interaction
between ranges and classes is not defined and hyphens must be specified at
either the beginning or the end of a set if they are not escaped.

Move all such occurrences to the beginning.

Reported-by: _parv (twitter)
MFC-after: 3 days

show more ...

lualoader: improve loader.conf var processing

lualoader was previously not processing \ as escapes; this commit fixes
that and does better error checking on the value as well.

Additionally, loader.

lualoader: improve loader.conf var processing

lualoader was previously not processing \ as escapes; this commit fixes
that and does better error checking on the value as well.

Additionally, loader.conf had some odd restrictions on values that make
little sense. Previously, lines like:

kernel=foo

Would simply be discarded with a malformed line complaint you might not
see unless you disable beastie.

lualoader tries to process these as well as it can and manipulates the
environment, while forthloader did minimal processing and constructed a
`set` command to do the heavy lifting instead. The lua approach was
re-envisioned from building a `set` command so that we can appropriately
reset the environment when, for example, boot environments change.

Lift the previous restrictions to allow unquoted values on the right hand
side of an expression. Note that an unquoted value is effectively:

[A-Za-z0-9-][A-Za-z0-9-_.]*

This commit also stops trying to weirdly limit what it can handle in a
quoted value. Previously it only allowed spaces, alphanumeric, and
punctuation, which is kind of weird. Change it here to grab as much as it
can between two sets of quotes, then let processEnvVar() do the needful and
complain if it finds something malformed looking.

My extremely sophisticated test suite is as follows:

<<EOF
X_01_simple_string="simple"
X_02_escaped_string="s\imple"

X_03_unquoted_val=3
X_04_unquoted_strval=simple_test

X_05_subval="${X_03_unquoted_val}"
X_06_escaped_subval="\${X_03_unquoted_val}"

X_07_embedded="truth${X_03_unquoted_val}"
X_08_escaped_embedded="truth\${X_03_unquoted_val}"

X_09_unknown="${unknown_val}"
X_10_unknown_embedded="truth${unknown_val}"

X_11_crunchy="crunch$unknown_val crunch"
X_12_crunchy="crunch${unknown_val}crunch"

Y_01_badquote="te"lol"
Y_02_eolesc="lol\"
Y_02_noteolesc="lol\\"
Y_03_eolvar="lol$"
Y_03_noteolvar="lol\$"
Y_04_badvar="lol${"

exec="echo Done!"
EOF

Future work may provide a stub loader module in userland so that we can
formally test the loader scripts rather than sketchy setups like the above
in conjunction with the lua-* tools in ^/tools/boot.

show more ...

lualoader: add loader_conf_dirs support (loader.conf.d)

loader_conf_dirs is the supporting mechanism for the included
/boot/loader.conf.d directory. When lualoader finishes processing all of
the lo

lualoader: add loader_conf_dirs support (loader.conf.d)

loader_conf_dirs is the supporting mechanism for the included
/boot/loader.conf.d directory. When lualoader finishes processing all of
the loader_conf_files it finds after walking /boot/defaults/loader.conf,
it will now check any and all loader_conf_dirs and process files ending
in ".conf" as if they were a loader.conf.

Note that loader_conf_files may be specified in a loader.conf.d config
file, but loader_conf_dirs may *not*. It will only be processed as specified
in /boot/defaults/loader.conf and any loader_conf_files that were loaded
from there.

Reviewed by: allanjude, freqlabs, rpokala, tsoome
Includes suggestion from: imp
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D25608

show more ...

lualoader: cli: provide a show-module-options loader command

This effectively dumps everything lualoader knows about to the console using
the libsa pager; that particular lua interface was added in

lualoader: cli: provide a show-module-options loader command

This effectively dumps everything lualoader knows about to the console using
the libsa pager; that particular lua interface was added in r368591.

A pager stub implementation has been added that just dumps the output as-is
as a compat shim for older loader binaries that do not have lpager. This
stub should be moved into a more appropriate .lua file if we add anything
else that needs the pager.

show more ...

lualoader: config: fix module enabled check

A last minute rewrite left this logically wrong; if it's present in
modules_blacklist, then we do not load it.

lualoader: provide module-manipulation commands

Specifically, we have:
- enable-module
- disable-module
- toggle-module

These can be used to add/remove modules to be loaded or force modules to be
l

lualoader: provide module-manipulation commands

Specifically, we have:
- enable-module
- disable-module
- toggle-module

These can be used to add/remove modules to be loaded or force modules to be
loaded in spite of modules_blacklist. In the typical case, a user is
expected to use them to recover an issue happening due to a module directive
they've added to their loader.conf or because they discover that they've
under-specified what to load.

MFC after: 1 week

show more ...

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, t

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

show more ...

lualoader: config: improve readConfFiles, rename to readConf

The previous interface was pretty bad, and required the caller to get some
implementation details correct that it really shouldn't need t

lualoader: config: improve readConfFiles, rename to readConf

The previous interface was pretty bad, and required the caller to get some
implementation details correct that it really shouldn't need to (e.g.
loader_conf_files handling) and pass in an empty table for it to use.

The new and much improved interface, readConf, is much less of a hack;
hiding these implementation details and just doing the right thing.
config.lua will now use it to process /boot/defaults/loader.conf and the
subsequent loader_conf_files from there, and read-conf will also use it.

This improvement submitted by Olivier (cited below), loader_conf_files
handling from the original patch was changed to just clobber it before
processing and not bother restoring it after the fact following r360505
where it's now guaranteed to evade the loader environment.

PR: 244640
Submitted by: Olivier Certner (olivier freebsd free fr>

show more ...

lualoader: config: add a table for restricted environment vars

This new table should be used for transient values that don't need to end up
in the loader environment. Generally, these will be things

lualoader: config: add a table for restricted environment vars

This new table should be used for transient values that don't need to end up
in the loader environment. Generally, these will be things that are internal
details that really aren't needed or interesting outside of the config
module (e.g. if we changed how ${module}_* directives work, they might use
this instead).

To start, populate it with loader_conf_files. Any specific value of
loader_conf_files isn't all that interesting; if we're going to export it,
we should really instead export a loader_conf_files that indicates all of
the configuration files we processed. This will be used to reduce
bookkeeping overhead in a future commit that cleans up readConfFiles.

show more ...

lualoader: config: start exporting readConfFiles

In the process, change it slightly: readConfFiles will take a string like
loader_conf_files in addition to the loaded_files table that it normally
ta

lualoader: config: start exporting readConfFiles

In the process, change it slightly: readConfFiles will take a string like
loader_conf_files in addition to the loaded_files table that it normally
takes. This is to facilitate the addition of a read-conf CLI command, which
will just pass in the single file to read and an empty table.

MFC after: 3 days

show more ...

lualoader config: don't call loader.getenv() as much

We don't actually need to fetch loader_conf_files as much as we do; we've
already fetched it once at the beginning, we only really need to fetch

lualoader config: don't call loader.getenv() as much

We don't actually need to fetch loader_conf_files as much as we do; we've
already fetched it once at the beginning, we only really need to fetch it
again after each file we've processed. If it changes, then we can stash that
off into our local prefiles.

While here, drop a note about the recursion so that I stop trying to
change it. It may very well make redundant some of the work we're doing, but
that's OK.

MFC after: 3 days

show more ...

Fixup line lost from previous commit (r359072)

Reviewed by: kevans
Approved by: mav (mentor)
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D24115

loader: Add a "kernel.loaded" hook

This hook can be useful, for example to run a local function to choose
different modules to load when a user has picked a different kernel
from the menu.

Reviewed

loader: Add a "kernel.loaded" hook

This hook can be useful, for example to run a local function to choose
different modules to load when a user has picked a different kernel
from the menu.

Reviewed by: kevans
Approved by: mav (mentor)
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D24115

show more ...

lua: add modules.loaded hook

This may be used for the local module to hook in and load any additional
modules that it wants, since it can't modify the modules table internal to
config. We may consid

lua: add modules.loaded hook

This may be used for the local module to hook in and load any additional
modules that it wants, since it can't modify the modules table internal to
config. We may consider adding API to do so at a later time, but I suspect
it will be more complicated to use with little return.

status is captured but ignored for the purpose of loading the hook. status
will be false if *any* module failed to load, but we typically don't let
that halt the boot so there's no reason to let it halt hooks. Some vendors
or setups may have expected fails that would be actively thwarted by
checking it.

We may, at a later date, consider adding an API for letting non-config
modules check which modules have successfully (or not) loaded in case an
unexpected failure *should* halt whatever they are doing.

MFC after: 3 days

show more ...

Further normalize copyright notices

- s/C/c/ where I've been inconsistent about it
- +SPDX tags
- Remove "All rights reserved" where possible

Requested by: rgrimes (all rights reserved)

lualoader: Fix up some luacheck concerns

- Garbage collect an unused (removed because it was useless) constant
- Don't bother with vararg notation if args will not be used

MFC after: 1 week