fixdep: use xmalloc()

When malloc() fails, there is not much userspace programs can do.
xmalloc() is useful to bail out on a memory allocation failure.

Signed-off-by: Masahiro Yamada <masahiroy@ker

fixdep: use xmalloc()

When malloc() fails, there is not much userspace programs can do.
xmalloc() is useful to bail out on a memory allocation failure.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

kbuild: implement CONFIG_TRIM_UNUSED_KSYMS without recursion

When CONFIG_TRIM_UNUSED_KSYMS is enabled, Kbuild recursively traverses
the directory tree to determine which EXPORT_SYMBOL to trim. If an

kbuild: implement CONFIG_TRIM_UNUSED_KSYMS without recursion

When CONFIG_TRIM_UNUSED_KSYMS is enabled, Kbuild recursively traverses
the directory tree to determine which EXPORT_SYMBOL to trim. If an
EXPORT_SYMBOL turns out to be unused by anyone, Kbuild begins the
second traverse, where some source files are recompiled with their
EXPORT_SYMBOL() tuned into a no-op.

Linus stated negative opinions about this slowness in commits:

- 5cf0fd591f2e ("Kbuild: disable TRIM_UNUSED_KSYMS option")
- a555bdd0c58c ("Kbuild: enable TRIM_UNUSED_KSYMS again, with some guarding")

We can do this better now. The final data structures of EXPORT_SYMBOL
are generated by the modpost stage, so modpost can selectively emit
KSYMTAB entries that are really used by modules.

Commit f73edc8951b2 ("kbuild: unify two modpost invocations") is another
ground-work to do this in a one-pass algorithm. With the list of modules,
modpost sets sym->used if it is used by a module. modpost emits KSYMTAB
only for symbols with sym->used==true.

BTW, Nicolas explained why the trimming was implemented with recursion:

https://lore.kernel.org/all/[email protected]/

Actually, we never achieved that level of optimization where the chain
reaction of trimming comes into play because:

- CONFIG_LTO_CLANG cannot remove any unused symbols
- CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is enabled only for vmlinux,
but not modules

If deeper trimming is required, we need to revisit this, but I guess
that is unlikely to happen.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: do not parse *.rlib, *.rmeta, *.so

fixdep is designed only for parsing text files. read_file() appends
a terminating null byte ('\0') and parse_config_file() calls strstr()
to search for CON

fixdep: do not parse *.rlib, *.rmeta, *.so

fixdep is designed only for parsing text files. read_file() appends
a terminating null byte ('\0') and parse_config_file() calls strstr()
to search for CONFIG options.

rustc outputs *.rlib, *.rmeta, *.so to dep-info. fixdep needs them in
the dependency, but there is no point in parsing such binary files.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Miguel Ojeda <[email protected]>
Tested-by: Miguel Ojeda <[email protected]>

show more ...

fixdep: avoid parsing the same file over again

The dep files (*.d files) emitted by C compilers usually contain the
deduplicated list of included files.

One exceptional case is when a header is inc

fixdep: avoid parsing the same file over again

The dep files (*.d files) emitted by C compilers usually contain the
deduplicated list of included files.

One exceptional case is when a header is included by the -include
command line option, and also by #include directive.

For example, the top Makefile adds the command line option,
"-include $(srctree)/include/linux/kconfig.h". You do not need to
include <linux/kconfig.h> in every source file.

In fact, include/linux/kconfig.h is listed twice in many .*.cmd files
due to include/linux/xarray.h having "#include <linux/kconfig.h>".
I did not fix that since it is a small redundancy.

However, this is more annoying for rustc. rustc emits the dependency
for each emission type.

For example, cmd_rustc_library emits dep-info, obj, and metadata.
So, the emitted *.d file contains the dependency for those 3 targets,
which makes fixdep parse the same file 3 times.

$ grep rust/alloc/raw_vec.rs rust/.alloc.o.cmd
rust/alloc/raw_vec.rs \
rust/alloc/raw_vec.rs \
rust/alloc/raw_vec.rs \

To skip the second parsing, this commit adds a hash table for parsed
files, just like we did for CONFIG options.

Signed-off-by: Masahiro Yamada <[email protected]>
Acked-by: Miguel Ojeda <[email protected]>
Tested-by: Miguel Ojeda <[email protected]>
Reviewed-by: Vincenzo Palazzo <[email protected]>

show more ...

fixdep: refactor hash table lookup

Change the hash table code so it will be easier to add the second table.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Miguel Ojeda <ojeda@ke

fixdep: refactor hash table lookup

Change the hash table code so it will be easier to add the second table.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Miguel Ojeda <[email protected]>
Tested-by: Miguel Ojeda <[email protected]>

show more ...

fixdep: parse Makefile more correctly to handle comments etc.

fixdep parses dependency files (*.d) emitted by the compiler.

*.d files are Makefiles describing the dependencies of the main source
fi

fixdep: parse Makefile more correctly to handle comments etc.

fixdep parses dependency files (*.d) emitted by the compiler.

*.d files are Makefiles describing the dependencies of the main source
file.

fixdep understands minimal Makefile syntax. It works well enough for
GCC and Clang, but not for rustc.

This commit improves the parser a little more for better processing
comments, escape sequences, etc.

My main motivation is to drop comments. rustc may output comments
(e.g. env-dep). Currentyly, rustc build rules invoke sed to remove
comments, but it is more efficient to do it in fixdep.

Signed-off-by: Masahiro Yamada <[email protected]>
Acked-by: Miguel Ojeda <[email protected]>
Tested-by: Miguel Ojeda <[email protected]>

show more ...

kbuild: rename cmd_$@ to savedcmd_$@ in *.cmd files

The cmd-check macro compares $(cmd_$@) and $(cmd_$1), but a pitfall is
that you cannot use cmd_<target> as the variable name for the command.

For

kbuild: rename cmd_$@ to savedcmd_$@ in *.cmd files

The cmd-check macro compares $(cmd_$@) and $(cmd_$1), but a pitfall is
that you cannot use cmd_<target> as the variable name for the command.

For example, the following code will not work in the top Makefile
or ./Kbuild.

quiet_cmd_foo = GEN $@
cmd_foo = touch $@

targets += foo
foo: FORCE
$(call if_changed,foo)

In this case, both $@ and $1 are expanded to 'foo', so $(cmd_check)
is always empty.

We do not need to use the same prefix for cmd_$@ and cmd_$1.
Rename the former to savedcmd_$@.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Nicolas Schier <[email protected]>

show more ...

fixdep: remove unneeded <stdarg.h> inclusion

This is unneeded since commit 69304379ff03 ("fixdep: use fflush() and
ferror() to ensure successful write to files").

Signed-off-by: Masahiro Yamada <ma

fixdep: remove unneeded <stdarg.h> inclusion

This is unneeded since commit 69304379ff03 ("fixdep: use fflush() and
ferror() to ensure successful write to files").

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: use fflush() and ferror() to ensure successful write to files

Currently, fixdep checks the return value from (v)printf(), but it does
not ensure the complete write to the .cmd file.

printf(

fixdep: use fflush() and ferror() to ensure successful write to files

Currently, fixdep checks the return value from (v)printf(), but it does
not ensure the complete write to the .cmd file.

printf() just writes data to the internal buffer, which usually succeeds.
(Of course, it may fail for another reason, for example when the file
descriptor is closed, but that is another story.)

When the buffer (4k?) is full, an actual write occurs, and printf() may
really fail. One of typical cases is "No space left on device" when the
disk is full.

The data remaining in the buffer will be pushed out to the file when
the program exits, but we never know if it is successful.

One straight-forward fix would be to add the following code at the end
of the program.

ret = fflush(stdout);
if (ret < 0) {
/* error handling */
}

However, it is tedious to check the return code in all the call sites
of printf(), fflush(), fclose(), and whatever can cause actual writes
to the end device. Doing that lets the program bail out at the first
failure but is usually not worth the effort.

Instead, let's check the error status from ferror(). This is 'sticky',
so you need to check it just once. You still need to call fflush().

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: David Laight <[email protected]>
Reviewed-by: Nick Desaulniers <[email protected]>

show more ...

kbuild: redo fake deps at include/config/*.h

Make include/config/foo/bar.h fake deps files generation simpler.

* delete .h suffix
those aren't header files, shorten filenames,

* delete tolower()

kbuild: redo fake deps at include/config/*.h

Make include/config/foo/bar.h fake deps files generation simpler.

* delete .h suffix
those aren't header files, shorten filenames,

* delete tolower()
Linux filesystems can deal with both upper and lowercase
filenames very well,

* put everything in 1 directory
Presumably 'mkdir -p' split is from dark times when filesystems
handled huge directories badly, disks were round adding to
seek times.

x86_64 allmodconfig lists 12364 files in include/config.

../obj/include/config/
├── 104_QUAD_8
├── 60XX_WDT
├── 64BIT
...
├── ZSWAP_DEFAULT_ON
├── ZSWAP_ZPOOL_DEFAULT
└── ZSWAP_ZPOOL_DEFAULT_ZBUD

0 directories, 12364 files

Signed-off-by: Alexey Dobriyan <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

modpost,fixdep: Replace zero-length array with flexible-array

The current codebase makes use of the zero-length array language
extension to the C90 standard, but the preferred mechanism to declare
v

modpost,fixdep: Replace zero-length array with flexible-array

The current codebase makes use of the zero-length array language
extension to the C90 standard, but the preferred mechanism to declare
variable-length types such as these ones is a flexible array member[1][2],
introduced in C99:

struct foo {
int stuff;
struct boo array[];
};

By making use of the mechanism above, we will get a compiler warning
in case the flexible array does not occur last in the structure, which
will help us prevent some kind of undefined behavior bugs from being
inadvertently introduced[3] to the codebase from now on.

Also, notice that, dynamic memory allocations won't be affected by
this change:

"Flexible array members have incomplete type, and so the sizeof operator
may not be applied. As a quirk of the original implementation of
zero-length arrays, sizeof evaluates to zero."[1]

sizeof(flexible-array-member) triggers a warning because flexible array
members have incomplete type[1]. There are some instances of code in
which the sizeof operator is being incorrectly/erroneously applied to
zero-length arrays and the result is zero. Such instances may be hiding
some bugs. So, this work (flexible-array member conversions) will also
help to get completely rid of those sorts of issues.

This issue was found with the help of Coccinelle.

[1] https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html
[2] https://github.com/KSPP/linux/issues/21
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by: Gustavo A. R. Silva <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: remove redundant null character check

If *q is '\0', the condition (isalnum(*q) || *q == '_') is false anyway.

It is redundant to ensure non-zero *q.

Signed-off-by: Masahiro Yamada <masahi

fixdep: remove redundant null character check

If *q is '\0', the condition (isalnum(*q) || *q == '_') is false anyway.

It is redundant to ensure non-zero *q.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: remove unneeded code and comments about *.ver files

This is probably stale code. In old days (~ Linux 2.5.59), Kbuild made
genksyms generate include/linux/modules/*.ver files.

The currenct

fixdep: remove unneeded code and comments about *.ver files

This is probably stale code. In old days (~ Linux 2.5.59), Kbuild made
genksyms generate include/linux/modules/*.ver files.

The currenct Kbuild does not generate *.ver files at all.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: check return value of printf() and putchar()

When there is not enough space on your storage device, the build will
fail with 'No space left on device' error message.

The reason is obvious f

fixdep: check return value of printf() and putchar()

When there is not enough space on your storage device, the build will
fail with 'No space left on device' error message.

The reason is obvious from the message, so you will free up some disk
space, then you will resume the build.

However, sometimes you may still see a mysterious error message:

unterminated call to function 'wildcard': missing ')'.

If you run out of the disk space, fixdep may end up with generating
incomplete .*.cmd files.

For example, if the disk-full error occurs while fixdep is running
print_dep(), the .*.cmd might be truncated like this:

$(wildcard include/config/

When you run 'make' next time, this broken .*.cmd will be included,
then Make will terminate parsing since it is a wrong syntax.

Once this happens, you need to run 'make clean' or delete the broken
.*.cmd file manually.

Even if you do not see any error message, the .*.cmd files after any
error could be potentially incomplete, and unreliable. You may miss
the re-compilation due to missing header dependency.

If printf() cannot output the string for disk shortage or whatever
reason, it returns a negative value, but currently fixdep does not
check it at all. Consequently, fixdep *successfully* generates a
broken .*.cmd file. Make never notices that since fixdep exits with 0,
which means success.

Given the intended usage of fixdep, it must respect the return value
of not only malloc(), but also printf() and putchar().

This seems a long-standing issue since the introduction of fixdep.

In old days, Kbuild tried to provide an extra safety by letting fixdep
output to a temporary file and renaming it after everything is done:

scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).tmp;\
rm -f $(depfile); \
mv -f $(dot-target).tmp $(dot-target).cmd)

It was no help to avoid the current issue; fixdep successfully created
a truncated tmp file, which would be renamed to a .*.cmd file.

This problem should be fixed by propagating the error status to the
build system because:

[1] Since commit 9c2af1c7377a ("kbuild: add .DELETE_ON_ERROR special
target"), Make will delete the target automatically on any failure
in the recipe.

[2] Since commit 392885ee82d3 ("kbuild: let fixdep directly write to
.*.cmd files"), .*.cmd file is included only when the corresponding
target already exists.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

kbuild: simplify dependency generation for CONFIG_TRIM_UNUSED_KSYMS

My main motivation of this commit is to clean up scripts/Kbuild.include
and scripts/Makefile.build.

Currently, CONFIG_TRIM_UNUSED

kbuild: simplify dependency generation for CONFIG_TRIM_UNUSED_KSYMS

My main motivation of this commit is to clean up scripts/Kbuild.include
and scripts/Makefile.build.

Currently, CONFIG_TRIM_UNUSED_KSYMS works with a tricky gimmick;
possibly exported symbols are detected by letting $(CPP) replace
EXPORT_SYMBOL* with a special string '=== __KSYM_*===', which is
post-processed by sed, and passed to fixdep. The extra preprocessing
is costly, and hacking cmd_and_fixdep is ugly.

I came up with a new way to find exported symbols; insert a dummy
symbol __ksym_marker_* to each potentially exported symbol. Those
dummy symbols are picked up by $(NM), post-processed by sed, then
appended to .*.cmd files. I collected the post-process part to a
new shell script scripts/gen_ksymdeps.sh for readability. The dummy
symbols are put into the .discard.* section so that the linker
script rips them off the final vmlinux or modules.

A nice side-effect is building with CONFIG_TRIM_UNUSED_KSYMS will
be much faster.

Signed-off-by: Masahiro Yamada <[email protected]>
Reviewed-by: Nicolas Pitre <[email protected]>

show more ...

fixdep: suppress consecutive / from file paths in dependency list files

Underscores in symbol names are translated into slashes for path names.
Filesystems treat consecutive slashes as if there was

fixdep: suppress consecutive / from file paths in dependency list files

Underscores in symbol names are translated into slashes for path names.
Filesystems treat consecutive slashes as if there was only one, so
let's do the same in the dependency list for easier grepping, etc.

Signed-off-by: Nicolas Pitre <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

kbuild: move include/config/ksym/* to include/ksym/*

The idea of using fixdep was inspired by Kconfig, but autoksyms
belongs to a different group. So, I want to move those touched
files under inclu

kbuild: move include/config/ksym/* to include/ksym/*

The idea of using fixdep was inspired by Kconfig, but autoksyms
belongs to a different group. So, I want to move those touched
files under include/config/ksym/ to include/ksym/.

The directory include/ksym/ can be removed by 'make clean' because
it is meaningless for the external module building.

Signed-off-by: Masahiro Yamada <[email protected]>
Acked-by: Nicolas Pitre <[email protected]>

show more ...

fixdep: do not ignore kconfig.h

kconfig.h was excluded from consideration by fixdep by
6a5be57f0f00 (fixdep: fix extraneous dependencies) to avoid some false
positive hits

(1) include/config/.h
(2)

fixdep: do not ignore kconfig.h

kconfig.h was excluded from consideration by fixdep by
6a5be57f0f00 (fixdep: fix extraneous dependencies) to avoid some false
positive hits

(1) include/config/.h
(2) include/config/h.h
(3) include/config/foo.h

(1) occurred because kconfig.h contains the string CONFIG_ in a
comment. However, since dee81e988674 (fixdep: faster CONFIG_ search), we
have a check that the part after CONFIG_ is non-empty, so this does not
happen anymore (and CONFIG_ appears by itself elsewhere, so that check
is worthwhile).

(2) comes from the include guard, __LINUX_KCONFIG_H. But with the
previous patch, we no longer match that either.

That leaves (3), which amounts to one [1] false dependency (aka stat() call
done by make), which I think we can live with:

We've already had one case [2] where the lack of include/linux/kconfig.h in
the .o.cmd file caused a missing rebuild, and while I originally thought
we should just put kconfig.h in the dependency list without parsing it
for the CONFIG_ pattern, we actually do have some real CONFIG_ symbols
mentioned in it, and one can imagine some translation unit that just
does '#ifdef __BIG_ENDIAN' but doesn't through some other header
actually depend on CONFIG_CPU_BIG_ENDIAN - so changing the target
endianness could end up rebuilding the world, minus that small
TU. Quoting Linus,

... when missing dependencies cause a missed re-compile, the resulting
bugs can be _really_ subtle.

[1] well, two, we now also have CONFIG_BOOGER/booger.h - we could change
that to FOO if we care

[2] https://lkml.org/lkml/2018/2/22/838

Cc: Linus Torvalds <[email protected]>
Signed-off-by: Rasmus Villemoes <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: remove some false CONFIG_ matches

The string CONFIG_ quite often appears after other alphanumerics,
meaning that that instance cannot be referencing a Kconfig
symbol. Omitting these means ma

fixdep: remove some false CONFIG_ matches

The string CONFIG_ quite often appears after other alphanumerics,
meaning that that instance cannot be referencing a Kconfig
symbol. Omitting these means make has fewer files to stat() when
deciding what needs to be rebuilt - for a defconfig build, this seems to
remove about 2% of the (wildcard ...) lines from the .o.cmd files.

Signed-off-by: Rasmus Villemoes <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: remove stale references to uml-config.h

uml-config.h hasn't existed in this decade (87e299e5c750 - x86, um: get
rid of uml-config.h). The few remaining UML_CONFIG instances are defined
direc

fixdep: remove stale references to uml-config.h

uml-config.h hasn't existed in this decade (87e299e5c750 - x86, um: get
rid of uml-config.h). The few remaining UML_CONFIG instances are defined
directly in terms of their real CONFIG symbol in common-offsets.h, so
unlike when the symbols got defined via a sed script, anything that uses
UML_CONFIG_FOO now should also automatically pick up a dependency on
CONFIG_FOO via the normal fixdep mechanism (since common-offsets.h
should at least recursively be a dependency). Hence I believe we should
actually be able to ignore the HELLO_CONFIG_BOOM cases.

Cc: Al Viro <[email protected]>
Cc: Richard Weinberger <[email protected]>
Cc: [email protected]
Signed-off-by: Rasmus Villemoes <[email protected]>
Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: use existing helper to check modular CONFIG options

str_ends_with() tests if the given token ends with a particular string.
Currently, it is used to check file paths without $(srctree).

Act

fixdep: use existing helper to check modular CONFIG options

str_ends_with() tests if the given token ends with a particular string.
Currently, it is used to check file paths without $(srctree).

Actually, we have one more place where this helper is useful. Use it
to check if CONFIG option ends with _MODULE.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: refactor parse_dep_file()

parse_dep_file() has too much indentation, and puts the code far to
the right. This commit refactors the code and reduces the one level
of indentation.

strrcmp()

fixdep: refactor parse_dep_file()

parse_dep_file() has too much indentation, and puts the code far to
the right. This commit refactors the code and reduces the one level
of indentation.

strrcmp() computes 'slen' by itself, but the caller already knows the
length of the token, so 'slen' can be passed via function argument.
With this, we can swap the order of strrcmp() and "*p = \0;"

Also, strrcmp() is an ambiguous function name. Flip the logic and
rename it to str_ends_with().

I added a new helper is_ignored_file() - this returns 1 if the token
represents a file that should be ignored.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: move global variables to local variables of main()

I do not mind global variables where they are useful enough. In this
case, I do not see a good reason to use global variables since they
a

fixdep: move global variables to local variables of main()

I do not mind global variables where they are useful enough. In this
case, I do not see a good reason to use global variables since they
are just referenced in shallow places. It is easy to pass them via
function arguments.

I squashed print_cmdline() into main() since it is just one line code.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: remove unneeded memcpy() in parse_dep_file()

Each token in the depfile is copied to the temporary buffer 's' to
terminate the token with zero. We do not need to do this any more
because the

fixdep: remove unneeded memcpy() in parse_dep_file()

Each token in the depfile is copied to the temporary buffer 's' to
terminate the token with zero. We do not need to do this any more
because the parsed buffer is now writable. Insert '\0' directly in
the buffer without calling memcpy().

<limits.h> is no longer necessary. (It was needed for PATH_MAX).

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...

fixdep: factor out common code for reading files

Now, do_config_files() and print_deps() are almost the same. Only
the difference is the parser function called (parse_config_file vs
parse_dep_file)

fixdep: factor out common code for reading files

Now, do_config_files() and print_deps() are almost the same. Only
the difference is the parser function called (parse_config_file vs
parse_dep_file).

We can reduce the code duplication by factoring out the common code
into read_file() - this function allocates a buffer and loads a file
to it. It returns the pointer to the allocated buffer. (As before,
it bails out by exit(2) for any error.) The caller must free the
buffer when done.

Having empty source files is possible; fixdep should simply skip them.
I deleted the "st.st_size == 0" check, so read_file() allocates 1-byte
buffer for an empty file. strstr() will immediately return NULL, and
this is what we expect.

On the other hand, an empty dep_file should be treated as an error.
In this case, parse_dep_file() will error out with "no targets found"
and it is a correct error message.

Signed-off-by: Masahiro Yamada <[email protected]>

show more ...