scripts/nsdeps: get 'make nsdeps' working again

Since commit cdd30ebb1b9f ("module: Convert symbol namespace to string
literal"), when MODULE_IMPORT_NS() is missing, 'make nsdeps' inserts
pointless

scripts/nsdeps: get 'make nsdeps' working again

Since commit cdd30ebb1b9f ("module: Convert symbol namespace to string
literal"), when MODULE_IMPORT_NS() is missing, 'make nsdeps' inserts
pointless code:

MODULE_IMPORT_NS("ns");

Here, "ns" is not a namespace, but the variable in the semantic patch.
It must not be quoted. Instead, a string literal must be passed to
Coccinelle.

Fixes: cdd30ebb1b9f ("module: Convert symbol namespace to string literal")
Signed-off-by: Masahiro Yamada <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>

show more ...

kbuild: remove extmod_prefix, MODORDER, MODULES_NSDEPS variables

With the previous changes, $(extmod_prefix), $(MODORDER), and
$(MODULES_NSDEPS) are constant. (empty, modules.order, and
modules.nsde

kbuild: remove extmod_prefix, MODORDER, MODULES_NSDEPS variables

With the previous changes, $(extmod_prefix), $(MODORDER), and
$(MODULES_NSDEPS) are constant. (empty, modules.order, and
modules.nsdeps, respectively).

Remove these variables and hard-code their values.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Nicolas Schier <[email protected]>

show more ...

kbuild: change working directory to external module directory with M=

Currently, Kbuild always operates in the output directory of the kernel,
even when building external modules. This increases the

kbuild: change working directory to external module directory with M=

Currently, Kbuild always operates in the output directory of the kernel,
even when building external modules. This increases the risk of external
module Makefiles attempting to write to the kernel directory.

This commit switches the working directory to the external module
directory, allowing the removal of the $(KBUILD_EXTMOD)/ prefix from
some build artifacts.

The command for building external modules maintains backward
compatibility, but Makefiles that rely on working in the kernel
directory may break. In such cases, $(objtree) and $(srctree) should
be used to refer to the output and source directories of the kernel.

The appearance of the build log will change as follows:

[Before]

$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
CC [M] /path/to/my/externel/module/helloworld.o
MODPOST /path/to/my/externel/module/Module.symvers
CC [M] /path/to/my/externel/module/helloworld.mod.o
CC [M] /path/to/my/externel/module/.module-common.o
LD [M] /path/to/my/externel/module/helloworld.ko
make: Leaving directory '/path/to/my/linux'

[After]

$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
make[1]: Entering directory '/path/to/my/externel/module'
CC [M] helloworld.o
MODPOST Module.symvers
CC [M] helloworld.mod.o
CC [M] .module-common.o
LD [M] helloworld.ko
make[1]: Leaving directory '/path/to/my/externel/module'
make: Leaving directory '/path/to/my/linux'

Printing "Entering directory" twice is cumbersome. This will be
addressed later.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Nicolas Schier <[email protected]>

show more ...

scripts/nsdeps: adjust to the format change of *.mod files

Commit 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
changed the format of *.mod files to put one object per line, but mis

scripts/nsdeps: adjust to the format change of *.mod files

Commit 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
changed the format of *.mod files to put one object per line, but missed
to adjust scripts/nsdeps.

Fixes: 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

kbuild: replace LANG=C with LC_ALL=C

LANG gives a weak default to each LC_* in case it is not explicitly
defined. LC_ALL, if set, overrides all other LC_* variables.

LANG < LC_CTYPE, LC_COLLATE

kbuild: replace LANG=C with LC_ALL=C

LANG gives a weak default to each LC_* in case it is not explicitly
defined. LC_ALL, if set, overrides all other LC_* variables.

LANG < LC_CTYPE, LC_COLLATE, LC_MONETARY, LC_NUMERIC, ... < LC_ALL

This is why documentation such as [1] suggests to set LC_ALL in build
scripts to get the deterministic result.

LANG=C is not strong enough to override LC_* that may be set by end
users.

[1]: https://reproducible-builds.org/docs/locales/

Signed-off-by: Masahiro Yamada <[email protected]>
Acked-by: Michael Ellerman <[email protected]> (powerpc)
Reviewed-by: Matthias Maennich <[email protected]>
Acked-by: Matthieu Baerts <[email protected]> (mptcp)
Reviewed-by: Greg Kroah-Hartman <[email protected]>

show more ...

kbuild: do not use scripts/ld-version.sh for checking spatch version

scripts/ld-version.sh was, as its file name implies, originally intended
for the GNU ld version, but is (ab)used for the spatch v

kbuild: do not use scripts/ld-version.sh for checking spatch version

scripts/ld-version.sh was, as its file name implies, originally intended
for the GNU ld version, but is (ab)used for the spatch version too.

Use 'sort -CV' for the version comparison, then coccicheck does not need
to use scripts/ld-version.sh. Fix nsdeps as well.

Signed-off-by: Masahiro Yamada <[email protected]>
Signed-off-by: Julia Lawall <[email protected]>

show more ...

scripts: add dummy report mode to add_namespace.cocci

When running `make coccicheck` in report mode using the
add_namespace.cocci file, it will fail for files that contain
MODULE_LICENSE. Those matc

scripts: add dummy report mode to add_namespace.cocci

When running `make coccicheck` in report mode using the
add_namespace.cocci file, it will fail for files that contain
MODULE_LICENSE. Those match the replacement precondition, but spatch
errors out as virtual.ns is not set.

In order to fix that, add the virtual rule nsdeps and only do search and
replace if that rule has been explicitly requested.

In order to make spatch happy in report mode, we also need a dummy rule,
as otherwise it errors out with "No rules apply". Using a script:python
rule appears unrelated and odd, but this is the shortest I could come up
with.

Adjust scripts/nsdeps accordingly to set the nsdeps rule when run trough
`make nsdeps`.

Suggested-by: Julia Lawall <[email protected]>
Fixes: c7c4e29fb5a4 ("scripts: add_namespace: Fix coccicheck failed")
Cc: YueHaibing <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Signed-off-by: Matthias Maennich <[email protected]>
Reported-by: Shuah Khan <[email protected]>
Acked-by: Julia Lawall <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Greg Kroah-Hartman <[email protected]>

show more ...

scripts/nsdeps: support nsdeps for external module builds

scripts/nsdeps is written to take care of only in-tree modules.
Perhaps, this is not a bug, but just a design. At least,
Documentation/core-

scripts/nsdeps: support nsdeps for external module builds

scripts/nsdeps is written to take care of only in-tree modules.
Perhaps, this is not a bug, but just a design. At least,
Documentation/core-api/symbol-namespaces.rst focuses on in-tree modules.

Having said that, some people already tried nsdeps for external modules.
So, it would be nice to support it.

Reported-by: Steve French <[email protected]>
Reported-by: Jessica Yu <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>
Tested-by: Jessica Yu <[email protected]>
Acked-by: Jessica Yu <[email protected]>
Reviewed-by: Matthias Maennich <[email protected]>
Tested-by: Matthias Maennich <[email protected]>

show more ...

modpost: dump missing namespaces into a single modules.nsdeps file

The modpost, with the -d option given, generates per-module .ns_deps
files.

Kbuild generates per-module .mod files to carry module

modpost: dump missing namespaces into a single modules.nsdeps file

The modpost, with the -d option given, generates per-module .ns_deps
files.

Kbuild generates per-module .mod files to carry module information.
This is convenient because Make handles multiple jobs in parallel
when the -j option is given.

On the other hand, the modpost always runs as a single thread.
I do not see a strong reason to produce separate .ns_deps files.

This commit changes the modpost to generate just one file,
modules.nsdeps, each line of which has the following format:

<module_name>: <list of missing namespaces>

Please note it contains *missing* namespaces instead of required ones.
So, modules.nsdeps is empty if the namespace dependency is all good.

This will work more efficiently because spatch will no longer process
already imported namespaces. I removed the '(if needed)' from the
nsdeps log since spatch is invoked only when needed.

This also solves the stale .ns_deps problem reported by Jessica Yu:

https://lkml.org/lkml/2019/10/28/467

Signed-off-by: Masahiro Yamada <[email protected]>
Tested-by: Jessica Yu <[email protected]>
Acked-by: Jessica Yu <[email protected]>
Reviewed-by: Matthias Maennich <[email protected]>
Tested-by: Matthias Maennich <[email protected]>

show more ...

scripts/nsdeps: make sure to pass all module source files to spatch

The nsdeps script passes a list of the module source files to
generate_deps_for_ns() as a space delimited string named $mod_source

scripts/nsdeps: make sure to pass all module source files to spatch

The nsdeps script passes a list of the module source files to
generate_deps_for_ns() as a space delimited string named $mod_source_files,
which then passes it to spatch. But since $mod_source_files is not encased
in quotes, each source file in that string is treated as a separate shell
function argument (as $2, $3, $4, etc.). However, the spatch invocation
only refers to $2, so only the first file out of $mod_source_files is
processed by spatch.

This causes problems (namely, the MODULE_IMPORT_NS() statement doesn't
get inserted) when a module is composed of many source files and the
"main" module file containing the MODULE_LICENSE() statement is not the
first file listed in $mod_source_files. Fix this by encasing
$mod_source_files in quotes so that the entirety of the string is
treated as a single argument and can be referred to as $2.

In addition, put quotes in the variable assignment of mod_source_files
to prevent any shell interpretation and field splitting.

Reviewed-by: Masahiro Yamada <[email protected]>
Acked-by: Matthias Maennich <[email protected]>
Signed-off-by: Jessica Yu <[email protected]>

show more ...

scripts/nsdeps: use alternative sed delimiter

When doing an out of tree build with O=, the nsdeps script constructs
the absolute pathname of the module source file so that it can insert
MODULE_IMPOR

scripts/nsdeps: use alternative sed delimiter

When doing an out of tree build with O=, the nsdeps script constructs
the absolute pathname of the module source file so that it can insert
MODULE_IMPORT_NS statements in the right place. However, ${srctree}
contains an unescaped path to the source tree, which, when used in a sed
substitution, makes sed complain:

++ sed 's/[^ ]* *//home/jeyu/jeyu-linux\/&/g'
sed: -e expression #1, char 12: unknown option to `s'

The sed substitution command 's' ends prematurely with the forward
slashes in the pathname, and sed errors out when it encounters the 'h',
which is an invalid sed substitution option. To avoid escaping forward
slashes ${srctree}, we can use '|' as an alternative delimiter for
sed instead to avoid this error.

Reviewed-by: Masahiro Yamada <[email protected]>
Reviewed-by: Matthias Maennich <[email protected]>
Tested-by: Matthias Maennich <[email protected]>
Signed-off-by: Jessica Yu <[email protected]>

show more ...

nsdeps: make generated patches independent of locale

scripts/nsdeps automatically generates a patch to add MODULE_IMPORT_NS
tags, and what is nicer, it sorts the lines alphabetically with the
'sort'

nsdeps: make generated patches independent of locale

scripts/nsdeps automatically generates a patch to add MODULE_IMPORT_NS
tags, and what is nicer, it sorts the lines alphabetically with the
'sort' command. However, the output from the 'sort' command depends on
locale.

For example, I got this:

$ { echo usbstorage; echo usb_storage; } | LANG=en_US.UTF-8 sort
usbstorage
usb_storage
$ { echo usbstorage; echo usb_storage; } | LANG=C sort
usb_storage
usbstorage

So, this means people might potentially send different patches.

This kind of issue was reported in the past, for example,
commit f55f2328bb28 ("kbuild: make sorting initramfs contents
independent of locale").

Adding 'LANG=C' is a conventional way of fixing when a deterministic
result is desirable.

I added 'LANG=C' very close to the 'sort' command since changing
locale affects the language of error messages etc. We should respect
users' choice as much as possible.

Reviewed-by: Matthias Maennich <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>
Signed-off-by: Jessica Yu <[email protected]>

show more ...

nsdeps: fix hashbang of scripts/nsdeps

This script does not use bash-extension. I am guessing this hashbang
was copied from scripts/coccicheck, which really uses bash-extension.

/bin/sh is enough f

nsdeps: fix hashbang of scripts/nsdeps

This script does not use bash-extension. I am guessing this hashbang
was copied from scripts/coccicheck, which really uses bash-extension.

/bin/sh is enough for this script.

Reviewed-by: Matthias Maennich <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>
Signed-off-by: Jessica Yu <[email protected]>

show more ...

scripts: Coccinelle script for namespace dependencies.

A script that uses the '<module>.ns_deps' files generated by modpost to
automatically add the required symbol namespace dependencies to each
mo

scripts: Coccinelle script for namespace dependencies.

A script that uses the '<module>.ns_deps' files generated by modpost to
automatically add the required symbol namespace dependencies to each
module.

Usage:
1) Move some symbols to a namespace with EXPORT_SYMBOL_NS() or define
DEFAULT_SYMBOL_NAMESPACE
2) Run 'make' (or 'make modules') and get warnings about modules not
importing that namespace.
3) Run 'make nsdeps' to automatically add required import statements
to said modules.

This makes it easer for subsystem maintainers to introduce and maintain
symbol namespaces into their codebase.

Co-developed-by: Martijn Coenen <[email protected]>
Signed-off-by: Martijn Coenen <[email protected]>
Acked-by: Julia Lawall <[email protected]>
Reviewed-by: Greg Kroah-Hartman <[email protected]>
Signed-off-by: Matthias Maennich <[email protected]>
Signed-off-by: Jessica Yu <[email protected]>

show more ...