The 'local' site method is easily confused with the 'file' site method,
making people create packages like this:
FOO_SITE_METHOD = local
FOO_SOURCE = foo.tar.gz
$(eval $(generic-package))
Due to the intricacies of the generic package infra, this does not
cause an error; instead, the foo.tar.gz tarball that happens to be
present in the download directory will be used. This behaviour differs
greatly from what is specified in the manual.
Instead, error out immediately if a package specifies the 'local' site
method but does not specify a _SITE.
We check for _OVERRIDE_SRCDIR rather than checking for _SITE, just
after _OVERRIDE_SRCDIR has been set to _SITE. Indeed, a package that
sets _OVERRIDE_SRCDIR but not _SITE currently works correctly. There is
no reason to make it fail.
See also
https://stackoverflow.com/questions/50364655/including-patches-to-build-root
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 775929c988)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When rebuilding a package, simply appending the package's file list to
the global list means that the package list grows for every rebuild, as
does the time taken to check for files installed by multiple packages.
Furthermore, we get false positives where a file is reported as being
installed by multiple copies of the same package.
With this approach we may end up with orphaned files in the target
filesystem if a package that has been updated and rebuilt no longer
installs the same set of files, but we know that only a clean build will
produce reliable results. In fact it may be helpful to identify these
orphaned files as evidence that the build is not clean.
Signed-off-by: John Keeping <john@metanate.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit d3dca1e993)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In current Buildroot, clashes occur between the variables _NAME and
_BASE_NAME for two packages called foo and foo-base, i.e.
Package foo:
FOO_NAME = foo
FOO_BASE_NAME = foo-1.2.3
Package foo-base:
FOO_BASE_NAME = foo-base
FOO_BASE_BASE_NAME = foo-base-4.5.6
where variable FOO_BASE_NAME is clashing between these two packages.
Specific cases where this clash is already existing are:
- alljoyn-base
- alljoyn-tcl-base
- perl-xml-sax-base
The problem is generic and can occur for a number of variables in Buildroot.
A non-exhaustive list:
<pkg>_BASE and <pkg>_BASE_NAME
<pkg>_BASE_NAME and <pkg>_RAW_BASE_NAME
<pkg>_DIR and <pkg>_DL_DIR
<pkg>_VERSION and <pkg>_DL_VERSION
<pkg>_SOURCE and <pkg>_TARGET_SOURCE
<pkg>_INSTALL_IMAGES and <pkg>_TARGET_INSTALL_IMAGES (same for _STAGING and _TARGET)
<pkg>_LICENSE_FILES and <pkg>_MANIFEST_LICENSE_FILES
<pkg>_DEPENDENCIES and <pkg>_FINAL_DEPENDENCIES
One solution is to use another separator than '_' to separate the
package name from the rest of the variable name. For example, a double
underscore:
FOO__NAME
FOO__BASE_NAME
FOO_BASE__NAME
FOO_BASE__BASE_NAME
However, making that change for only this case means that the variable
naming is no longer consistent. And making the change for all variables has
a large impact, also on certain user scripts.
For now, keep it simple, and rename FOO_BASE_NAME into FOO_BASENAME, so that
the variables become:
FOO_NAME
FOO_BASENAME
FOO_BASE_NAME
FOO_BASE_BASENAME
For consistency, also adapt FOO_RAW_BASE_NAME. Since FOO_RAW_BASENAME would
still pose a conflict with a package called 'foo-raw', take the opportunity
to rename it into FOO_BASENAME_RAW instead, which does not pose a conflict
as we have no variable called FOO_RAW.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Sam Voss <sam.voss@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 83d2644b11)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
As part of the build, we run some instrumentation hooks to gather
statistics about the usage of the target/, staging/ and host/
directories, so that we can generate reports for the user, that
shows:
- for each file, what package installed it,
- for each package,the size that it installed.
In so doing, we run a double md5 pass on all files of the affected
directories (before/after installation). These passes were mostly invisible
when we were only scanning target/, but has greatly increased in time now
that we also scan staging/ and host/ (but only in the corresponding _CMDS,
of course).
This md5 was mostly aimed at catching packages that would "cheat" with
mtime/atime/ctime somehow. They can't really cheat on md5, though [0].
Timings however speak for themselves, with this defconfig (slightly
biggish-but-still-manageable build) [1].
host/ 20965 files 1.2GiB
staging/ 4715 files 333MiB
target/ 1801 files 44MiB
All instrumentation steps, using md5: 19min 27s
All instrumentation steps, using mtime: 14min 45s
No instrumentation step at all: 14min 31s
So, using mtime is an almost-5min improvement, i.e. about 25% faster,
while removing all instrumentation steps does not gain that much more...
So, we switch to using mtime, because in the end that's still good-enough
for our use-case: generating some graphs. It is not mission-critical, and
if a graph is slightly off, that's not a biggy. It can anyway be attributed
to a broken package's buildsystem, which should get fixed.
However, we lose the ability to track directories. Non-empty directories
can be tracked back by a bit of scripting, but empty directories are
simply not caught. If we were to also look for directories using mtime,
we would catch parents of installed files:
- /foo/bar/ exists
- a package installs /foo/bar/buz
- mtime of /foo/bar/ is changed to account for the new file in it.
So we do not track directories at all, and we lose empty directories.
The existing tracking was mostly happenstance, with the original
submission and comments not really accounting for a real use-case.
Now, we also change the way we handle symlinks. Previously, we would
hash the file pointed to by the symlink. Now, we only look at the mtime
of the symlink itself, which still detects modifications.
Eventually, this also means that we now no longer need to establish a
list before the install step; we can now simply run after the install
step, finding any files newer than the build stamp.
[0] Yeah, md5 is very weak, but we're not guarding against malicious
attacks, just about careless modifications.
[1] defconfig used for tests:
BR2_arm=y
BR2_cortex_a7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_INIT_SYSTEMD=y
BR2_PACKAGE_MESA3D=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_ETNAVIV=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_VC4=y
BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_VIRGL=y
BR2_PACKAGE_MESA3D_DRI_DRIVER_SWRAST=y
BR2_PACKAGE_MESA3D_OSMESA=y
BR2_PACKAGE_MESA3D_OPENGL_ES=y
BR2_PACKAGE_SYSTEMD_JOURNAL_GATEWAY=y
BR2_PACKAGE_SYSTEMD_BACKLIGHT=y
BR2_PACKAGE_SYSTEMD_BINFMT=y
BR2_PACKAGE_SYSTEMD_COREDUMP=y
BR2_PACKAGE_SYSTEMD_FIRSTBOOT=y
BR2_PACKAGE_SYSTEMD_HIBERNATE=y
BR2_PACKAGE_SYSTEMD_IMPORTD=y
BR2_PACKAGE_SYSTEMD_LOCALED=y
BR2_PACKAGE_SYSTEMD_LOGIND=y
BR2_PACKAGE_SYSTEMD_MACHINED=y
BR2_PACKAGE_SYSTEMD_POLKIT=y
BR2_PACKAGE_SYSTEMD_QUOTACHECK=y
BR2_PACKAGE_SYSTEMD_RANDOMSEED=y
BR2_PACKAGE_SYSTEMD_RFKILL=y
BR2_PACKAGE_SYSTEMD_SMACK_SUPPORT=y
BR2_PACKAGE_SYSTEMD_SYSUSERS=y
BR2_PACKAGE_SYSTEMD_VCONSOLE=y
[Peter: tweak commit message, use find -type l]
Reported-by: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Trent Piepho <tpiepho@impinj.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 7fb6e78254)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some packages (mostly, out-of-tree) may want to install binary blobs for
another architecture, outside the locations we currently exclude, like
in /opt or whatever...
Add support in check-bin-arch to accept any arbitrary location, that
individual package can each request to excude from the check, when they
are installed.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 01d90f0d09)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we store the list of files installed in target/ and associate
each of them to the package that installed it.
However, we sometimes may need to know what package installed which file
in staging/, for example to debug header collision, or in host/, to
debug what package installed what host tool.
Enhance the step instrumentation to also generate the list for staging/
and host/.
We maintain backward compatibility, for external scripts that wanted to
parse the previously existing list, by not renaming the target-related
package list. Only the staging- and host-related lists are named after
staging and host.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Thomas: fix missing word in .mk comment.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To compute the list of files added by a package, we first store the list
of files before the install, do the install, list the files after the
install, and finally compare the two lists. The two lists are stored in
dot-files, hidden in the package's build dir.
We currently keep those two files, and only list the files installed in
target/
In followup patches, we'll also list files installed in staging/ as well
as files installed in host/.
Rather than add even more internal, hidden files in the package build
dir, we'll just re-use the same two temporary files to store the before
and after lists.
So, remove them after the comparison is done.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Using the package instrumentation hooks, a file packages-file-list.txt is
created containing the list of files installed by a package with their size.
Due to the use of 'find -type f', symbolic links to files are not included
in this file list. Since the original purpose of this file was to calculate
the total size of a package and symbolic links have file size 0, this was
not a problem.
However, if packages-file-list.txt is reused for other purposes, for example
to get a complete list of files installed by a package regardless of size,
symbolic links to files are important too.
Likewise, to get a complete view of what a package installs, directories
should be included too.
Update the instrumentation hook accordingly.
Although for files an md5sum is taken, we cannot do this for directories.
Instead, mimic the output of md5sum with a fake hash string.
Note: for directories that are used by several packages, e.g. /etc/init.d/,
the package that created the directory will be treated as the 'owner' of
that directory. This gives a somewhat distorted view of ownership.
Similarly, the package size reported by 'make graph-size' (based on
packages-file-list.txt) will include the 'size' of a directory for that
owner, which is reported as e.g. 4096 bytes.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Before and after the building of each package, the instrumentation hooks are
run. One of these hooks obtains the list of files installed by a package.
The code to obtain this list is currently duplicated in the start and end
part of the hook. While the amount of duplication is currently small, a
subsequent patch will make more changes to this code, increasing the
duplication.
Therefore, split off into a helper function.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In some cases, the directory structure we want in the filesystem is not
exactly what we have in target/
For example, when systemd is used on a read-only rootfs, /var must be a
tmpfs. However, we may have packages that install stuff in there, and
set important rights (via the permission-table). So, at build time, we
need /var to be a symlink to the remanent location (/usr/share/factory)
while at runtime we need /var to be a directory.
One option would have been to have /var as a real directory even during
build time, and in a target-finalize hook, move everything out of there
and into the "factory" location. However, that's not possible because
it's too early: some packages may want to set ownership and/or acces
rights on directories or files in /var, and this is only done in the
fakeroot script, which is called only later during the assembling of the
filesystem images.
Also, there would have been no way to undo the tweak (i.e. we need to
restore the /var symlink so that subsequent builds continue to work) if
it were done as a target-finalize hook.
The only solution is to allow packages to register pre- and post-hooks
that are called right before and right after the rootfs commands are
executed, and inside in the fakeroot script.
We can however not re-use the BR2_ROOTFS_POST_FAKEROOT_SCRIPT feature
either because it is done before the filesystem command, but there is
nothing that is done after. Also, we don't want to add to, and modify a
user-supplied variable.
So, we introduce two new variables that packages can set to add the
commands they need to run to tweak the filesystem right at the last
moment.
Those hooks are not documented on-purpose; they are probably going to
only ever be used by systemd.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This variable is currently not used by anyone.
The value is changed to match the path to DL_DIR.
The next patch will introduce the use of PKG_DL_DIR for packages that
use specific EXTRACT_CMDS.
And then we will be able to change the value of PKG_DL_DIR to create a
new directory structure in DL_DIR with a subdirectory for each package.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
We currently skip the skeleton dependency by checking if the current
package is the skeleton.
We are going to introduce more skeleton-related packages, so we
need a way to exclude the skeleton dependency for those, or we'd
get a circular dependency, for the same reason we need to skip
the toolchain dependency.
Instead of checking for all the skeleton-providing packages in the core
infra, add a new package options so that packages can express they do
not need the dependency on the skeleton, like we have an option to avoid
the depednency on the toolchain. The only packages that will use that
option are probably the skeletons, so we need not document this
variable, like we did not document the option to exclude the dependency
on the toolchain.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, the per-package legal-info is mostly silent, but we're soon
to add a check for the hashes of the license files.
In that case, and when there is a hash mis-match, we want a user to know
what package had a changed license file.
So, we add a call to MESSAGE to display the package we're currently
saving the legal-info of, like so:
>>> busybox 1.26.2 Collecting legal info
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Rahul Bedarkar <rahulbedarkar89@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Running foo-source-check on packages retrieved from git fails.
This is because there is no associated stamp file, so we do not have a
rule-assignment that sets PKG for foo-source-check.
But it does not make sense to have a stamp file at all, because
source-check is not supposed to change anything: the status after is
exactly the same as before; nothing is downlaoded, so there is no
progress (whatsoever) to memorise.
Fix that by just defining PKG in the source-check rule definition.
Fixes#9796.
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When debugging hidden dependencies, the build order is very important.
Most notably, it is interesting to identify potential culprits.
Add a new top-level rule, show-biuld-order, that dumps all the packages
in the order they would get built.
Note that there are a few differences with show-targets:
- more packages are reported, becasue show-targets does not report
host packages that have no prompt;
- the output is line-based, because we're using $(info $(1)); getting
a single output line like show-targets would require we use an
actual command, like printf '%s ' $(1); but that takes a lot of
time, while $(info $(1)) is almost instantaneous (the time to parse
the Makefiles);
- rootfs targets are not reported.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Whitespaces were searched using the following regex:
[ ]{1,}\t
and then manually removed in most of the cases. For
xserver_xorg-server.mk, tabs before backslashes were removed.
Signed-off-by: Bernd Kuhls <bernd.kuhls@t-online.de>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
As shown recently by the firejail example, it is easy to miss that a
package builds and installs binaries without actually cross-compiling
them: they are built for the host architecture instead of the target
architecture.
This commit adds a small helper script, check-bin-arch, called as a
GLOBAL_INSTRUMENTATION_HOOKS at the end of the target installation of
each package, to verify that the files installed by this package have
been built for the correct architecture.
Being called as a GLOBAL_INSTRUMENTATION_HOOKS allows the build to error
out right after the installation of the faulty package, and therefore
get autobuilder error detection properly assigned to this specific
package.
Example output with the firejail package enabled, when building for an
ARM target:
ERROR: architecture for ./usr/lib/firejail/libconnect.so is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/bin/firejail is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/lib/firejail/libtrace.so is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/lib/firejail/libtracelog.so is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/lib/firejail/ftee is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/lib/firejail/faudit is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/bin/firemon is Advanced Micro Devices X86-64, should be ARM
ERROR: architecture for ./usr/bin/firecfg is Advanced Micro Devices X86-64, should be ARM
Many thanks to Yann E. Morin and Arnout Vandecappelle for their reviews
and suggestions.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Don't special case $(XZCAT) when constructing DL_TOOLS_DEPENDENCIES. The next
commit will introduce another extractor that automatically builds when not
installed. Introduce EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS that lists
archive extensions for which the extractor is already checked in
support/dependencies/check-host-foo.mk. Use this in the newly introduced
extractor-dependency to populate DL_TOOLS_DEPENDENCIES.
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Thomas: add missing space after "firstword", as noticed by Thomas DS.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some packages misbehave, and install files in either $(STAGING_DIR)/$(O)
or in $(TARGET_DIR)/$(O) .
One common reason for that is that pkgconf now prepends the sysroot path
to all the paths it returns. Other reasons vary, but are mostly due to
poorly writen generic-packages.
And a new step hooks to check that no file gets installed in either
location, called after the install-target and install-staging steps.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Now that we can dump the reverse dependencies of a package, add the
ability to graph those.
It does not make sense to do a full reverse graph, as it would be
semantically equivalent to the direct graph. So we only provide a
per-package reverse graph.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Finding the packages that select another one in a specific configuration
is not very trivial:
- when optional, the dependency is not expressed in Kconfig
- looking at the .mk files is not very nice.
Introduce a way to dump reverse dependencies of packages, i.e. the list
of packages that directly depend on that package. Like for direct
dependencies, we limit the list to the first-order reverse dependencies.
Document it in the main help; use the opportunity to also document
foo-show-depends.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we only support at most one br2-external tree. Being able
to use more than one br2-external tree can be very useful.
A use-case would be for having a br2-external to contain the basic
packages, basic board defconfigs and board files, provided by one team
responsible for the "board-bringup", while other teams consume that
br2-external as a base, and complements it each with their own set of
packages, defconfigs and extra board files.
Another use-case would be for third-parties to provide their own
Buildroot packaging in a br2-external tree, along-side the archives for
their stuff.
Finally, another use-case is to be able to add FLOSS packages in a
br2-external tree, and proprietary packages in another. This allows
to not touch the Buildroot tree at all, and still be able to get in
compliance by providing only that br2-external tree(s) that contains
FLOSS packages, leaving aside the br2-external tree(s) with the
proprietary bits.
What we do is to treat BR2_EXTERNAL as a colon-separated (space-
separated also work, and we use that internally) list of paths, on which
we iterate to construct:
- the list of all br2-external names, BR2_EXTERNAL_NAMES,
- the per-br2-external tree BR2_EXTERNAL_$(NAME) variables, which
point each to the actual location of the corresponding tree,
- the list of paths to all the external.mk files, BR2_EXTERNAL_MKS,
- the space-separated list of absolute paths to the external trees,
BR2_EXTERNAL_DIRS.
Once we have all those variables, we replace references to BR2_EXTERNAL
with either one of those.
This cascades into how we display the list of defconfigs, so that it is
easy to see what br2-external tree provides what defconfigs. As
suggested by Arnout, tweak the comment from "User-provided configs" to
"External configs", on the assumption that some br2-external trees could
be provided by vendors, so not necessarily user-provided. Ditto the menu
in Kconfig, changed from "User-provided options" to "External options".
Now, when more than one br2-external tree is used, each gets its own
sub-menu in the "User-provided options" menu. The sub-menu is labelled
with that br2-external tree's name and the sub-menu's first item is a
comment with the path to that br2-external tree.
If there's only one br2-external tree, then there is no sub-menu; there
is a single comment that contains the name and path to the br2-external
tree.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@openwide.fr>
Cc: Julien CORJON <corjon.j@ecagroup.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This unique NAME is used to construct a per br2-external tree variable,
BR2_EXTERNAL_$(NAME)_PATH, which contains the path to the br2-external
tree.
This variable is available both from Kconfig (set in the Kconfig
snippet) and from the .mk files.
Also, display the NAME and its path as a comment in the menuconfig.
This will ultimately allow us to support multiple br2-external trees at
once, with that NAME (and thus BR2_EXTERNAL_$(NAME)) uniquely defining
which br2-external tree is being used.
The obvious outcome is that BR2_EXTERNAL should now no longer be used to
refer to the files in the br2-external tree; that location is now known
from the BR2_EXTERNAL_$(NAME)_PATH variable instead. This means we no
longer need to expose, and must stop from from exposing BR2_EXTERNAL as
a Kconfig variable.
Finally, this also fixes a latent bug in the pkg-generic infra, where we
would so far always refer to BR2_EXTERNAL (even if not set) to filter
the names of packages (to decide whether they are a bootloader, a
toolchain or a simple package).
Note: since the variables in the Makefile and in Kconfig are named the
same, the one we computed early on in the Makefile will be overridden by
the one in .config when we have it. Thus, even though they are set to
the same raw value, the one from .config is quoted and, being included
later in the Makefile, will take precedence, so we just re-include the
generated Makefile fragment a third time before includeing the
br2-external's Makefiles. That's unfortunate, but there is no easy way
around that as we do want the two variables to be named the same in
Makefile and Kconfig (and we can't ask the user to un-quote that variable
himself either), hence this little dirty triple-inclusion trick.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
By aggregating all the license files into a single big text-only file
means we have no way to use license files that are binary blobs (e.g.
pdf, rtf...).
Just do not generate that big file; if the user still wants it, it is
very easy to create it afterwards.
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, packages using target finalize hooks must remember that they
need to register their hook in TARGET_FINALIZE_HOOKS
conditionally (otherwise their hook will be triggered even if the
package is disabled).
In order to avoid this potential mistake, this commit introduces a
per-package target-finalize hook variable, in which packages can
register their target-finalize hooks, with the guarantee that they will
only be triggered if the package is enabled.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
[Thomas: rework commit log.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, when an user forgets to define a SITE, he gets some fairly
weird download failure. In order to make things easier to diagnose, this
commit adds a check in the package infrastructure that verifies that if
SOURCE has a value, SITE is not empty.
Cc: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, when the package defines no _SOURCE and no _VERSION
variables, we end up trying to download <PACKAGE>-undefined.tar.gz.
This is utterly pointless.
Change the logic to not attempt a download when neither _SOURCE
nor _VERSION is set:
- do not set an undefined _VERSION to "undefined"
- do not append an empty (or "undefined") version to the package
rawname
Consequently, for packages that have no _VERSION, the build
directory will be just the package name (which to some may look
nicer than the current "package-undefined").
Furthermore, the message trace is also a bit leaner (to the same
people that find "package" nicer than "package-undefined").
Signed-off-by: Alex Suykov <alex.suykov@gmail.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This feature consists in automatically deducing dependencies of a host
package from the dependencies of the target variant.
However, it causes some issues, and many host packages need different
dependencies than their target variants.
Now that host dependencies are explicitly set for all packages, we can
remove it.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Almost all packages which are saved for legal-info have their source
archives downloaded as part of 'make source', which makes an off-line
build completely possible [0].
However, for the pre-configured external toolchains, the source tarball
is different, as the main tarball is a binary package. And that source
tarball is only downloaded during the legal-info phase, which makes it
inconvenient for full off-line builds.
We fix that by adding a new rule, $(1)-legal-source which only
$(1)-all-source depends on, so that we only download it for a top-level
'make source', not as part of the standard download mechanism (i.e. only
what is really needed to build).
This new rule depends, like the normal download mechanism, on a stamp
file, so that we do not emit a spurious hash-check message on successive
runs of 'make source'.
This way, we can do a complete [0] off-line build and are still able to
generate legal-info, while at the same time we do not incur any download
overhead during a simple build.
Also, we previously downloaded the _ACTUAL_SOURCE_TARBALL when it was
not empty. However, since _ACTUAL_SOURCE_TARBALL defaults to the value
of _SOURCE, it can not be empty when _SOURCE is not. Thus, we'd get a
spurious report of a missing hash for the tarball, since it was not in
a standard package rule (configure, build, install..) and thus would
miss the PKG and PKGDIR variables to find the .hash file.
We fix that in this commit as well, by:
- setting PKG and PKGDIR just for the -legal-source rule;
- only downloading _ACTUAL_SOURCE_TARBALL if it is not empty *and* not
the same as _SOURCE (to avoid a second report about the hash).
[0] Save for nodejs which invarriably wants to download stuff at build
time. Sigh... :-( Fixing that is work for another time...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Move the declarations of _ACTUAL_SOURCE and _ACTUAL_SITE earlier, so
that they are close to where _SOURCE and _SITE are handled.
This looks so far like a purely cosmetic change, but makes more sense
with the follow-up patch, where we'll need them earlier in the file.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some packages, like perl, download extra files that end up as part of
the source that Buildroot builds. Up until now, those files were not
saved in the legal-info output.
Add those files to the legal-info output.
The unfortunate side-effect is that we will also save the secondary
archive for the external blackfin toolchains; however, we already do
save the binary release of some external toolchains when they do not
provide actual source archives.
This is inherently bad, as those are not source archives, but solving
this is a bigger concern, for another series...
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, the legal-info infra only saves the source archive of a
package. However, that's not enough as we may apply some patches on
packages sources.
We do suggest users to also redistribute the Buildroot sources as part
of their compliance distribution, so the patches bundled in Buildroot
would indeed be included in the compliance distribution.
However, that's still not enough, since we may download some patches, or
the user may use a global patch directory. Patches in there might not
end up in the compliance distribution, and there are risks of
non-conformity.
So, always include patches alongside the source archive.
To ensure reproducibility, we also generate a series file, so patches
can be re-applied in the correct order.
We get the list of patches to include from the list of patches that were
applied by the package infrastructure (via the apply-patches support
script). So, we need to get packages properly extracted and patched
before we can save their legal-info, not just in the case they define
_LICENSE_FILES.
Update the legal-info header accordingly.
Note: this means that, when a package is not patched and defines no
LICENSE_FILES, we will extract and patch it for nothing. There is no
easy way to know whether we have to patch a package or not. We can only
either duplicate the logic to detect patches (bad) or rely on the infra
actually patching the package. Also, a vast majority of packages are
either patched, or define _LICENSE_FILES, so it is best and easiest to
always extract and patch them prior to legal-info.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Now that we save the source archives in a directory named after the
package and its version, do the same for the license files, for
consistency.
It has a not-so-bad side-effect of also saving the version string in
the all-licenses list.
The only (small) side-effect, is that the warnings about undefined
_LICENSE_FILES now contains the version string, too. That's unavoidable,
since that's what is stored in the legal report.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add a package-variable to store the package-specific make rules.
Although this variable would be seldom used, we still document it.
However, we make sure the documentation explicitly states that this
variable should not be used (if it needs to be, the submitter of a
package will be told so during reviews).
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we put all source archives side-by-side in the same
directory.
Since we're about to also save individual patches that were applied
on those sources, we don't want to make that directory a complete
mess of unassorted files.
So, we install each source archive in its own sub-directory, where
we'll later store the patches too. Store that location in a variable,
so it can be re-used later on (to install patches in a future commit).
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The ifneq ($$($(2)_SITE_METHOD),override) test will always match
because SITE_METHOD is never set to "override". The result is that
some packages are downloaded and extracted unnecessarily, although
they are (correctly) not saved in output/legal-info/.
Fix by simply testing $(2)_OVERRIDE_SRCDIR instead: when empty, it
indicates that the packages is neither a "local" site-method package
nor a package that has its source directory overridden.
Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, we generate the dependency graph in a single command, piping
the stdout of support/scripts/.graph-depends to the stdin of dot.
Unfortunately, this means we can't catch a failure of graph-depends, as
the shell can only treturn the exit code of the last command in a pipe.
Still, we do want to keep the output of graph-depends, and we in fact do
keep it by mean of a tee.
graph-depends has just gained the ability to generate its output to a
file, so we break the pipe in two differnet commands, so we can bail out
on graph-depends errors.
Do that for the two call sites.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, the check that packages we build are indeed enabled is done
at the time a package is configured.
This can come quite late in the build process, and does not provide
direct knowledge of the real culprit for the incorrect dependency.
However, we can improve these two issues quite easily, albeit at the
expense of a very slightly more complicated make code.
First, the check can not be done at the time we define the package, i.e.
in the inner-generic-pacakge, because all its dependencies might have
not been parsed yet, so we can't yet know whether it is enabled or not
(because we can't match the package name of the dependency to its
Kconfig variable yet).
But then, we know we have all packages definitions after we scanned the
the bundled packages, kernel, bootloaders and toolchains, as well as the
br2-external tree (if any).
So, at this location, we iterate through the list of enabled packages,
and check that the packages they each depend on are indeed enabled.
This allows us to:
1- do the check very early, before any build action,
2- report on the exact offending package very easily.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
