lualoader: position hyphens at the beginning of character classes

According to the Lua 5.4 manual section 6.4.1 ("Patterns"), the interaction
between ranges and classes is not defined and hyphens mu

lualoader: position hyphens at the beginning of character classes

According to the Lua 5.4 manual section 6.4.1 ("Patterns"), the interaction
between ranges and classes is not defined and hyphens must be specified at
either the beginning or the end of a set if they are not escaped.

Move all such occurrences to the beginning.

(cherry picked from commit b24872cf7b13314669ed2136c0262bb2eb007695)

show more ...

lualoader: improve loader.conf var processing

lualoader was previously not processing \ as escapes; this commit fixes
that and does better error checking on the value as well.

Additionally, loader.

lualoader: improve loader.conf var processing

lualoader was previously not processing \ as escapes; this commit fixes
that and does better error checking on the value as well.

Additionally, loader.conf had some odd restrictions on values that make
little sense. Previously, lines like:

kernel=foo

Would simply be discarded with a malformed line complaint you might not
see unless you disable beastie.

lualoader tries to process these as well as it can and manipulates the
environment, while forthloader did minimal processing and constructed a
`set` command to do the heavy lifting instead. The lua approach was
re-envisioned from building a `set` command so that we can appropriately
reset the environment when, for example, boot environments change.

Lift the previous restrictions to allow unquoted values on the right hand
side of an expression. Note that an unquoted value is effectively:

[A-Za-z0-9-][A-Za-z0-9-_.]*

This commit also stops trying to weirdly limit what it can handle in a
quoted value. Previously it only allowed spaces, alphanumeric, and
punctuation, which is kind of weird. Change it here to grab as much as it
can between two sets of quotes, then let processEnvVar() do the needful and
complain if it finds something malformed looking.

My extremely sophisticated test suite is as follows:

<<EOF
X_01_simple_string="simple"
X_02_escaped_string="s\imple"

X_03_unquoted_val=3
X_04_unquoted_strval=simple_test

X_05_subval="${X_03_unquoted_val}"
X_06_escaped_subval="\${X_03_unquoted_val}"

X_07_embedded="truth${X_03_unquoted_val}"
X_08_escaped_embedded="truth\${X_03_unquoted_val}"

X_09_unknown="${unknown_val}"
X_10_unknown_embedded="truth${unknown_val}"

X_11_crunchy="crunch$unknown_val crunch"
X_12_crunchy="crunch${unknown_val}crunch"

Y_01_badquote="te"lol"
Y_02_eolesc="lol\"
Y_02_noteolesc="lol\\"
Y_03_eolvar="lol$"
Y_03_noteolvar="lol\$"
Y_04_badvar="lol${"

exec="echo Done!"
EOF

Future work may provide a stub loader module in userland so that we can
formally test the loader scripts rather than sketchy setups like the above
in conjunction with the lua-* tools in ^/tools/boot.

(cherry picked from commit 576562856efbec31520aca6a1f72f2b11298e9a7)

show more ...

lualoader: add loader_conf_dirs support (loader.conf.d)

loader_conf_dirs is the supporting mechanism for the included
/boot/loader.conf.d directory. When lualoader finishes processing all of
the lo

lualoader: add loader_conf_dirs support (loader.conf.d)

loader_conf_dirs is the supporting mechanism for the included
/boot/loader.conf.d directory. When lualoader finishes processing all of
the loader_conf_files it finds after walking /boot/defaults/loader.conf,
it will now check any and all loader_conf_dirs and process files ending
in ".conf" as if they were a loader.conf.

Note that loader_conf_files may be specified in a loader.conf.d config
file, but loader_conf_dirs may *not*. It will only be processed as specified
in /boot/defaults/loader.conf and any loader_conf_files that were loaded
from there.

Reviewed by: allanjude, freqlabs, rpokala, tsoome
Includes suggestion from: imp
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D25608

show more ...

lualoader: cli: provide a show-module-options loader command

This effectively dumps everything lualoader knows about to the console using
the libsa pager; that particular lua interface was added in

lualoader: cli: provide a show-module-options loader command

This effectively dumps everything lualoader knows about to the console using
the libsa pager; that particular lua interface was added in r368591.

A pager stub implementation has been added that just dumps the output as-is
as a compat shim for older loader binaries that do not have lpager. This
stub should be moved into a more appropriate .lua file if we add anything
else that needs the pager.

show more ...

lualoader: config: fix module enabled check

A last minute rewrite left this logically wrong; if it's present in
modules_blacklist, then we do not load it.

lualoader: provide module-manipulation commands

Specifically, we have:
- enable-module
- disable-module
- toggle-module

These can be used to add/remove modules to be loaded or force modules to be
l

lualoader: provide module-manipulation commands

Specifically, we have:
- enable-module
- disable-module
- toggle-module

These can be used to add/remove modules to be loaded or force modules to be
loaded in spite of modules_blacklist. In the typical case, a user is
expected to use them to recover an issue happening due to a module directive
they've added to their loader.conf or because they discover that they've
under-specified what to load.

MFC after: 1 week

show more ...

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, t

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

show more ...

lualoader: config: improve readConfFiles, rename to readConf

The previous interface was pretty bad, and required the caller to get some
implementation details correct that it really shouldn't need t

lualoader: config: improve readConfFiles, rename to readConf

The previous interface was pretty bad, and required the caller to get some
implementation details correct that it really shouldn't need to (e.g.
loader_conf_files handling) and pass in an empty table for it to use.

The new and much improved interface, readConf, is much less of a hack;
hiding these implementation details and just doing the right thing.
config.lua will now use it to process /boot/defaults/loader.conf and the
subsequent loader_conf_files from there, and read-conf will also use it.

This improvement submitted by Olivier (cited below), loader_conf_files
handling from the original patch was changed to just clobber it before
processing and not bother restoring it after the fact following r360505
where it's now guaranteed to evade the loader environment.

PR: 244640
Submitted by: Olivier Certner (olivier freebsd free fr>

show more ...

lualoader: config: add a table for restricted environment vars

This new table should be used for transient values that don't need to end up
in the loader environment. Generally, these will be things

lualoader: config: add a table for restricted environment vars

This new table should be used for transient values that don't need to end up
in the loader environment. Generally, these will be things that are internal
details that really aren't needed or interesting outside of the config
module (e.g. if we changed how ${module}_* directives work, they might use
this instead).

To start, populate it with loader_conf_files. Any specific value of
loader_conf_files isn't all that interesting; if we're going to export it,
we should really instead export a loader_conf_files that indicates all of
the configuration files we processed. This will be used to reduce
bookkeeping overhead in a future commit that cleans up readConfFiles.

show more ...

lualoader: config: start exporting readConfFiles

In the process, change it slightly: readConfFiles will take a string like
loader_conf_files in addition to the loaded_files table that it normally
ta

lualoader: config: start exporting readConfFiles

In the process, change it slightly: readConfFiles will take a string like
loader_conf_files in addition to the loaded_files table that it normally
takes. This is to facilitate the addition of a read-conf CLI command, which
will just pass in the single file to read and an empty table.

MFC after: 3 days

show more ...

lualoader config: don't call loader.getenv() as much

We don't actually need to fetch loader_conf_files as much as we do; we've
already fetched it once at the beginning, we only really need to fetch

lualoader config: don't call loader.getenv() as much

We don't actually need to fetch loader_conf_files as much as we do; we've
already fetched it once at the beginning, we only really need to fetch it
again after each file we've processed. If it changes, then we can stash that
off into our local prefiles.

While here, drop a note about the recursion so that I stop trying to
change it. It may very well make redundant some of the work we're doing, but
that's OK.

MFC after: 3 days

show more ...

Fixup line lost from previous commit (r359072)

Reviewed by: kevans
Approved by: mav (mentor)
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D24115

loader: Add a "kernel.loaded" hook

This hook can be useful, for example to run a local function to choose
different modules to load when a user has picked a different kernel
from the menu.

Reviewed

loader: Add a "kernel.loaded" hook

This hook can be useful, for example to run a local function to choose
different modules to load when a user has picked a different kernel
from the menu.

Reviewed by: kevans
Approved by: mav (mentor)
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D24115

show more ...

lua: add modules.loaded hook

This may be used for the local module to hook in and load any additional
modules that it wants, since it can't modify the modules table internal to
config. We may consid

lua: add modules.loaded hook

This may be used for the local module to hook in and load any additional
modules that it wants, since it can't modify the modules table internal to
config. We may consider adding API to do so at a later time, but I suspect
it will be more complicated to use with little return.

status is captured but ignored for the purpose of loading the hook. status
will be false if *any* module failed to load, but we typically don't let
that halt the boot so there's no reason to let it halt hooks. Some vendors
or setups may have expected fails that would be actively thwarted by
checking it.

We may, at a later date, consider adding an API for letting non-config
modules check which modules have successfully (or not) loaded in case an
unexpected failure *should* halt whatever they are doing.

MFC after: 3 days

show more ...

Further normalize copyright notices

- s/C/c/ where I've been inconsistent about it
- +SPDX tags
- Remove "All rights reserved" where possible

Requested by: rgrimes (all rights reserved)

lualoader: Fix up some luacheck concerns

- Garbage collect an unused (removed because it was useless) constant
- Don't bother with vararg notation if args will not be used

MFC after: 1 week

lualoader: Always return a proper dictionary for blacklist

If module_blacklist isn't specified, we have an empty blacklist; effectively
the same as if module_blacklist="" were specified in loader.co

lualoader: Always return a proper dictionary for blacklist

If module_blacklist isn't specified, we have an empty blacklist; effectively
the same as if module_blacklist="" were specified in loader.conf(5).

This was reported when switching to a BE that predated the module_blacklist
introduction, but the problem is valid all the same and likely to be tripped
over in other scenarios.

Reported by: bwidawsk
MFC after: 3 days

show more ...

lualoader: Improve module loading diagnostics

Some fixes:

- Maintain historical behavior more accurately w.r.t verbose_loading;
verbose_loading strictly prints "${module_name...}" and later "fail

lualoader: Improve module loading diagnostics

Some fixes:

- Maintain historical behavior more accurately w.r.t verbose_loading;
verbose_loading strictly prints "${module_name...}" and later "failed!"
or "ok" based on load success
- With or without verbose_loading, dump command_errbuf on load failure.
This usually happens prior to ok/failed if we're verbose_loading

Reviewed by: imp
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D17694

show more ...

lualoader: Create a module blacklist, add DRM modules to it

This is a step in the process of easing migration into the new world order
of DRM drivers. Strongly encourage users towards loading DRM mo

lualoader: Create a module blacklist, add DRM modules to it

This is a step in the process of easing migration into the new world order
of DRM drivers. Strongly encourage users towards loading DRM modules via
rc.conf(5) instead of loader.conf(5) by failing the load from loader(8).
Users so inclined may wipe out the blacklist via module_blacklist="" in
loader.conf(5), and it is expected that these modules will eventually be
removed from the blacklist. They may still be loaded as dependencies of
other modules or explicitly via the loader prompt, but this should not be a
major problem.

Approved by: re (rgrimes)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D16914

show more ...

lualoader: Fix override of module_path on loader prompt

Earlier changes setup a config.module_path variable that was populated upon
reading of loader.conf(5) and used for restoring module_path to pr

lualoader: Fix override of module_path on loader prompt

Earlier changes setup a config.module_path variable that was populated upon
reading of loader.conf(5) and used for restoring module_path to pristine
condition if multiple kernels are attempted. This broke the ability to
override module_path at the loader prompt in case of emergency.

Approved by: re (rgrimes)

show more ...

lualoader: Accept that people use unquoted values in loader.conf

While loader.conf(5) suggests that all values should be quoted, reality is
that this was never strictly enforced and it is used. We a

lualoader: Accept that people use unquoted values in loader.conf

While loader.conf(5) suggests that all values should be quoted, reality is
that this was never strictly enforced and it is used. We already make some
concession to this in number values, which aren't strictly quoted either.

The compromise here is that multi-word values must be quoted. This lets
things like `foo_load=YES` work, while denying more complex expressions on
the right hand side. This likely catches the vast majority of current usage.

A bit of a kludge is needed to accomplish this since Lua regex doesn't
support branching. I had considered splitting up expressions and generating
the right-hand side of the expressions completely in config.parse, but
deemed this too large of an overhaul to take given the current timing. This
should be re-worked shortly after the thaw.

Reported by: royger

show more ...

lualoader: Fix (add) Xen support

lualoader was not respecting the 'xen_kernel' environment variable, which
hints to the interpreter that it should load a Xen kernel prior to loading
any other kernel

lualoader: Fix (add) Xen support

lualoader was not respecting the 'xen_kernel' environment variable, which
hints to the interpreter that it should load a Xen kernel prior to loading
any other kernel that might be specified. If a Xen kernel is specified and
we fail to load it, we should not proceed to boot.

Reported by: royger
Tested by: royger

show more ...

lualoader: Fix loader.conf(5) EOL validation for 'exec' lines

This includes some light rework to simplify the line parsing, as well. If
we hit a line match, we'll always either use the line and mov

lualoader: Fix loader.conf(5) EOL validation for 'exec' lines

This includes some light rework to simplify the line parsing, as well. If
we hit a line match, we'll always either use the line and move on to the
next line, or we'll spew out malformed line errors.

We had multiple spots to output the error and set the status based on
whether we had a non-nil first capture group or failed EOL validation, but
it was always the same error. Light rework entails a small label jump to
skip error handling and elimination of 'found' local.

show more ...

lualoader: Refactor config line expressions

A couple of issues addressed:

1.) Modules with - in the name were not recognized as modules
2.) The module regex was repeated for each place a module nam

lualoader: Refactor config line expressions

A couple of issues addressed:

1.) Modules with - in the name were not recognized as modules
2.) The module regex was repeated for each place a module name may appear
3.) The 'strip leading space' bits were repeated for each expression
4.) The trailing 'comment validation' stuff was repeated every expression

#4 still has some more work to be done. exec lines, for instance, don't
capture a 'value' -- there's only one capture pattern. This throws off the
'c' value that we match, so the trailing bits aren't *actually* being
validated. This isn't a new issue, though, so a future comit will address
this.

show more ...

lualoader: Fix parsing of negative number loader.conf(5) variables

They would previously cause errors, as the regex for these did not tolerate
a leading negative sign, and the variable would simply

lualoader: Fix parsing of negative number loader.conf(5) variables

They would previously cause errors, as the regex for these did not tolerate
a leading negative sign, and the variable would simply not parse.

show more ...