Commit Graph

1028 Commits

Author SHA1 Message Date
Baruch Siach
1e063bef46 support/dependencies: clarify intended use of host bison/flex
We should not rely on host installed bison/flex for target code. This
ensures better reproducibility of generated code.

http://lists.busybox.net/pipermail/buildroot/2020-November/296786.html

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 1b1c049af2)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-12-07 21:36:04 +01:00
Peter Korsgaard
a4832641bc Update for 2020.02.8
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-17 00:04:03 +01:00
Romain Naour
dd8bada803 support/testing/test_hardening: add missing Kconfig symbol
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y is needed to use the
custom external toolchain x86-i686--glibc--bleeding-edge-2018.11-1.tar.bz2

Otherwise the symbol BR2_TOOLCHAIN_EXTERNAL_URL is lost.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981738
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981739
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981740
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981741
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981742
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981743

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 7b9762f4ab)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-14 12:23:06 +01:00
Romain Naour
31c336fb8e support/testing: TestInitSystemSystemdRwIfupdown test expect a RW rootfs
When running the TestInitSystemSystemdRwIfupdown test, the rootfs must
be in read-write mode. The commit log [1] introducing systemd tests say
so:

"basic systemd, read-write, network w/ ifupdown"

With systemd 246.5, the service systemd-update-done return an error code
when it can't write on the filesystem (/etc)

[1] 117835d5fc
[2] 8019995e9a

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/830981813

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 14ed65e3a6)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-11-13 23:36:07 +01:00
Fabrice Fontaine
150a9c4024 support/scripts/apply-patches.sh: manage uncompression
Extract from bug report:

"Code line 120 to line 128 is to check whether the patch containing
"rename from" and "rename to". But it directly use grep to find,
ignoring the patch may be a tar file or else. It can only work on patch
of textfile form."

Fixes:
 - https://bugs.buildroot.org/show_bug.cgi?id=11931

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 35c3a1e693)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-10-28 23:28:37 +01:00
Peter Korsgaard
d8082db677 Update for 2020.02.7
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-10-12 23:33:22 +02:00
Peter Korsgaard
3a813a6a52 support/scripts/apply-patches.sh: do not blindly remove *.orig files
apply-patches currently blindly removes *.orig / .*.orig files as GNU patch
by default writes these as backup files when patches only apply with fuzz.

This is unfortunate as package sources may contain files ending in .orig as
well, breaking the build.  Luckily GNU patch can be told to not write these
backup files using the --no-backup-if-mismatch option, so used that instead
of the .orig removal step.

--no-backup-if-mismatch is supported since GNU patch 2.3.8 (1997-06-17) and
busybox patch if built with CONFIG_DESKTOP, but E.G.  isn't supported by the
BSD patch, so add logic to dependencies.sh to error out if patch doesn't
support the flag.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 42f61e759a)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-10-10 22:43:06 +02:00
Thomas De Schampheleire
5b59dc0354 support/scripts/setlocalversion: fix/improve Mercurial output
Commit 9e4ffdc8cf modified the output of
'setlocalversion' so that the Buildroot version tag is included in the
output, the version part was added in Makefile.

Due to differences in behavior of the used git and Mercurial commands, this
caused different output for the Mercurial case, in BR2_VERSION_FULL and thus
/etc/os-release and 'make print-version'. Assuming the official Buildroot
releases are tagged and no project-specific tags are present, the output
after commit 9e4ffdc8cf is:
    -hg<commit>
whereas it is expected to be something like:
    2020.02.6-hg<commit>

Change the Mercurial case in setlocalversion to behave similar to git,
looking up the latest tag if the current revision is not itself tagged.

The number of commits after the latest tag is not added, unlike in git, as
this value is not commonly present in Mercurial output, and its added value
can be disputed in this context. Even one commit could bring a huge change
to the sources, so in order to interpret the number one has to look at the
repository anyhow, in which case the commit ID can just be used.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 32eb5a1d16)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-10-05 08:09:33 +02:00
Robin Jarry
a25c190df4 support/scripts/pycompile: fix .pyc original source file paths
When generating a .pyc file, the original .py source file path is
encoded in it. It is used for various purposes: traceback generation,
.pyc file comparison with its .py source, and code inspection.

By default, the source path used when invoking compileall is encoded in
the .pyc file. Since we use paths relative to TARGET_DIR, we end up with
paths that are only valid when relative to '/' encoded in the installed
.pyc files on the target.

This breaks code inspection at runtime since the original source path
will be invalid unless the code is executed from '/'.

Unfortunately, compileall cannot be forced to use the proper path. It
was not written with cross-compilation usage in mind.

Rework the script to call py_compile.compile() directly with pertinent
options:

- The script now has a new --strip-root argument. This argument is
  optional but will always be specified when compiling py files in
  buildroot.
- All other (non-optional) arguments are folders in which all
  "importable" .py files will be compiled to .pyc.
- Using --strip-root=$(TARGET_DIR), the future runtime path of each .py
  file is computed and encoded into the compiled .pyc.

No need to change directory before running the script anymore.

The trickery used to handle error reporting was only applicable with
compileall. Since we implement our own "compileall", error reporting
becomes trivial.

Previously, we had a --force option to tell compileall.compiledir() to
forcibly recompile files if they had changed. Now, we would have to
handle it ourselves. It turns out to not be easy and would need us to
delve into the format of bytecompiled files to extract metadata and
compare it with the expected values, that being even dependent on the
python version being used (fortunately, only two for us: python 2.7 and
the latext 3.x).

Still, this is deemed too complex, and byte-compiling is pretty fast, so
much so that it should be eclipsed by the build duration anyway.

So we just drop support for --force, and instead we always byte-compile.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
[yann.morin.1998@free.fr:
  - always byte-compile
  - drop --force
  - expand commit log to state so and explain why
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit c566f5206a)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-09-15 20:06:11 +02:00
Robin Jarry
8673646dcc support/scripts/pycompile: sort imports
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 4c77dca550)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-09-15 20:06:07 +02:00
Robin Jarry
416bb105d8 support/scripts/pycompile: add main entry point
Only run code when the script is executed directly (not imported).
Factorize command description by using the script's __doc__ variable.
Fix typo in --force help message.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 7b3025f93e)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-09-15 20:05:58 +02:00
Thomas Petazzoni
fc24b95723 support/scripts/pkg-stats: drop erroneous "break" in CVE.affects()
Commit 7d2779ecbb
("support/script/pkg-stats: handle exception when version comparison
fails") erroneousy introduced a "break" within a try/expect block.

This break has the unfortunate consequence that every CVE that was
using the <= operator was skipped, and according to the current
CVE statistics, made us miss 74 CVEs out of 141 CVEs.

Here is for reference the complete list of CVEs we missed:

 - gnupg
   CVE-2006-3082
   CVE-2019-13050

 - jhead
   CVE-2020-6624
   CVE-2020-6625

 - patch
   CVE-2018-6952
   CVE-2019-20633

 - json-c
   CVE-2020-12762

 - git
   CVE-2018-1000110
   CVE-2018-1000182
   CVE-2019-1003010
   CVE-2020-2136

 - iperf2
   CVE-2016-4303

 - libtorrent
   CVE-2009-1760
   CVE-2016-5301

 - lua
   CVE-2020-15888
   CVE-2020-15889
   CVE-2020-15945
   CVE-2020-24342

 - openvpn
   CVE-2020-7224

 - smack
   CVE-2016-10027

 - bashtop
   CVE-2019-18276

 - links
   CVE-2008-3319

 - argus
   CVE-2011-3332

 - libraw
   CVE-2020-15503

 - netcat
   CVE-2008-5727
   CVE-2008-5728
   CVE-2008-5729
   CVE-2008-5730
   CVE-2008-5742
   CVE-2015-2214

 - subversion
   CVE-2017-1000085
   CVE-2018-1000111
   CVE-2020-2111

 - python
   CVE-2013-1753
   CVE-2015-5652
   CVE-2017-17522
   CVE-2017-18207
   CVE-2019-20907
   CVE-2019-9674

 - cereal
   CVE-2020-11104
   CVE-2020-11105

 - opencv
   CVE-2017-1000450
   CVE-2017-12597
   CVE-2017-12598
   CVE-2017-12599
   CVE-2017-12600
   CVE-2017-12601
   CVE-2017-12602
   CVE-2017-12603
   CVE-2017-12604
   CVE-2017-12605
   CVE-2017-12606
   CVE-2017-12862
   CVE-2017-12863
   CVE-2017-12864
   CVE-2019-15939

 - docker
   CVE-2015-1843
   CVE-2015-3627
   CVE-2015-3630
   CVE-2015-3631
   CVE-2016-3697
   CVE-2017-14992
   CVE-2019-16884

 - trousers
   CVE-2020-24330
   CVE-2020-24331
   CVE-2020-24332

 - libcroco
   CVE-2020-12825

 - libpupnp
   CVE-2020-13848

 - openjpeg
   CVE-2020-15389

 - flex
   CVE-2015-1773

 - libesmtp
   CVE-2019-19977

 - ed
   CVE-2015-2987

 - libmad
   CVE-2018-7263

 - grub
   CVE-2020-15705

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit b3f959fe96)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 19:29:38 +02:00
Thomas Petazzoni
9cd8ee158f support/docker: drop Python 2.x modules
Since commit 4a40d36f13
("support/testing: switch to Python 3 only") our runtime testing
infrastructure is Python 3.x only.

Therefore, it is no longer needed to have python-nose2 and
python-pexpect in the Docker container used to run our Gitlab CI jobs.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 23f7fa874b)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 18:38:04 +02:00
Thomas Petazzoni
ddf41cf9a7 support/docker: use python3-flake8
support/scripts/pkg-stats now uses some Python 3.x only constructs
("async" and related keywords), so we must use the Python 3.x flake8.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 385c4da3dd)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 18:35:49 +02:00
Thomas Petazzoni
d917bf4727 support/testing/tests/core/test_timezone.py: fix indentation
Fixes:

support/testing/tests/core/test_timezone.py:7:9: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit cc061128de)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 18:32:58 +02:00
Thomas Petazzoni
76bbb43441 support/testing: consistently use raw strings for re.compile
Otherwise Python 3.x flake8 complains with:

W605 invalid escape sequence '\s'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit b0078c058a)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 18:32:16 +02:00
Thomas Petazzoni
be53bbbdd0 support/scripts/pkg-stats: show progress of upstream URL and latest version
This commit slightly improves the output of pkg-stats by showing the
progress of the upstream URL checks and latest version retrieval, on a
package basis:

Checking URL status
[0001/0062] curlpp
[0002/0062] cmocka
[0003/0062] snappy
[0004/0062] nload
[...]
[0060/0062] librtas
[0061/0062] libsilk
[0062/0062] jhead
Getting latest versions ...
[0001/0064] libglob
[0002/0064] perl-http-daemon
[0003/0064] shadowsocks-libev
[...]
[0061/0064] lua-flu
[0062/0064] python-aiohttp-security
[0063/0064] ljlinenoise
[0064/0064] matchbox-lib

Note that the above sample was run on 64 packages. Only 62 packages
appear for the URL status check, because packages that do not have any
URL in their Config.in file, or don't have any Config.in file at all,
are not checked and therefore not accounted.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 5fea2e3997)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:48:32 +02:00
Thomas Petazzoni
bc51041e6b support/scripts/pkg-stats: use aiohttp for upstream URL checking
This commit reworks the code that checks if the upstream URL of each
package (specified by its Config.in file) using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 5c3221ac20)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:48:25 +02:00
Thomas Petazzoni
8e0b8bcbd3 support/scripts/pkg-stats: use aiohttp for latest version retrieval
This commit reworks the code that retrieves the latest upstream
version of each package from release-monitoring.org using the aiohttp
module. This makes the implementation much more elegant, and avoids
the problematic multiprocessing Pool which is causing issues in some
situations.

Since we're now using some async functionality, the script is Python
3.x only, so the shebang is changed to make this clear.

Suggested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 68093f4778)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:48:13 +02:00
Thomas Petazzoni
7cc13f49af support/scripts/pkg-stats: fix flake8 warning
This fixes the following flake8 warning:

support/scripts/pkg-stats:1005:9: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 204d03ae43)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:46:17 +02:00
Gregory CLEMENT
44f78b11de support/script/pkg-stats: handle exception when version comparison fails
With python 3, when a package has a version number x-y-z instead of
x.y.z, then the version returned by LooseVersion can't be compared
which raises a TypeError exception:

Traceback (most recent call last):
  File "./support/scripts/pkg-stats", line 1062, in <module>
    __main__()
  File "./support/scripts/pkg-stats", line 1051, in __main__
    check_package_cves(args.nvd_path, {p.name: p for p in packages})
  File "./support/scripts/pkg-stats", line 613, in check_package_cves
    if pkg_name in packages and cve.affects(packages[pkg_name]):
  File "./support/scripts/pkg-stats", line 386, in affects
    return pkg_version <= cve_affected_version
  File "/usr/lib64/python3.8/distutils/version.py", line 58, in __le__
    c = self._cmp(other)
  File "/usr/lib64/python3.8/distutils/version.py", line 337, in _cmp
    if self.version < other.version:
TypeError: '<' not supported between instances of 'str' and 'int'

This patch handles this exception by adding a new return value when
the comparison can't be done. The code is adjusted to take of this
change. For now, a return value of CVE_UNKNOWN is handled the same way
as a CVE_DOESNT_AFFECT return value, but this can be improved later
on.

Signed-off-by: Gregory CLEMENT <gregory.clement@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 7d2779ecbb)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:46:10 +02:00
Heiko Thiery
94725bb729 support/scripts/pkg-stats: add tilde '~' expansion for pathes
When the 'nvd-path', 'json' and 'html' are used like this:

  --html ~/foo

then the tilde expansion is properly done by the shell. However, when
they are used like this:

  --html=~/foo

The shell doesn't do the tilde expansion, and pkg-stats doesn't do
it. This commit modifies pkg-stats to ensure that tilde expansion is
done when parsing the 'nvd-path', 'json' and 'html' arguments.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[Thomas: improve commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit f41056ec4b)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:45:21 +02:00
Thomas Petazzoni
6a8e31a27a support/scripts/pkg-stats: fix flake8 E722 warning
flake8 complains with:

  support/scripts/pkg-stats:339:13: E722 do not use bare 'except'

Due to the construct:

  try:
     something
  except:
     print("some message")
     raise

Which is in fact OK because the exception is re-raised. This issue is
discussed at https://github.com/PyCQA/pycodestyle/issues/703, and the
general agreement is that these "bare except" are OK, and should be
ignored from flake8 using a noqa statement.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 3b5bc480a5)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:45:16 +02:00
Thomas Petazzoni
48d9765054 support/scripts/pkg-stats: fix flake8 E501 warning
Fixes:

support/scripts/pkg-stats:281:133: E501 line too long (139 > 132 characters)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit f7f33771b3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:45:12 +02:00
Thomas Petazzoni
903b4fa920 support/scripts/pkg-stats: fix flake8 E117 warning
Fixes:

  support/scripts/pkg-stats:146:17: E117 over-indented

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 198d76efb3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:45:07 +02:00
Thomas Petazzoni
9f21e32341 support/scripts/pkg-stats: fix flake8 E302 warning
Fixes:

  support/scripts/pkg-stats:57:1: E302 expected 2 blank lines, found 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit e03bdef0ec)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:45:03 +02:00
Thomas Petazzoni
0ab030f188 support/scripts/pkg-stats: fix flake8 E402 warning
flake8 complains with:

pkg-stats:38:1: E402 module level import not at top of file

This is due to sys.path.append() being before the import from
getdeveloperlib, but we really need this sys.path.append() to be
before, so let's ignore this flake8 warning.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 769f98c18c)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:44:57 +02:00
Heiko Thiery
7c87a94ef8 support/scripts/pkg-stats: add list of status checks to the json output
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 759521dae6)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:42:26 +02:00
Heiko Thiery
ef3585d6b8 support/scripts/pkg-stats: set status to 'na' for virtual packages
If there is no infra set or infra is virtual the status is set to 'na'.

This is done for the follwing checks:
 - license
 - license-files
 - hash
 - hash-license
 - patches
 - version

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit fb879c1954)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:42:14 +02:00
Heiko Thiery
75d97f2a8f support/scripts/pkg-stats: add defconfig support
Scan configs directory and create Defconfig objects.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 8d77ecbad0)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:42:09 +02:00
Heiko Thiery
3c98107873 support/scripts/pkg-stats: store pkg dir path
This value can be used for later processing.

In the buildroot-stats application this is used to create links pointing
to the git repo of buildroot.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit d31fadfbf5)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:42:04 +02:00
Heiko Thiery
3a8dd28bc5 support/scripts/pkg-stats: add package count to stats
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 0e267518cb)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:59 +02:00
Heiko Thiery
af549ab68f support/scripts/pkg-stats: add package status
Unify the status check information. The status is stored in a tuple. The
first entry is the status that can be 'ok', 'warning' or 'error'. The
second entry is a verbose message.

The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check

With that status information the following variables are replaced:
has_license, has_license_files, has_hash, url_status

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit f422fa991f)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:55 +02:00
Heiko Thiery
8ca8c1572c support/scripts/pkg-stats: store licences of package
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 5b7278e5f1)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:51 +02:00
Heiko Thiery
743b013a56 support/scripts/pkg-stats: set developers info
Use the function 'parse_developers' function from getdeveloperlib that
collect the information about the developers and the files they
maintain. Then set the maintainer(s) to each package.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit c1fc827934)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:46 +02:00
Heiko Thiery
7f72379943 support/scripts/pkg-stats: store patch files for the package
Remove the patch_count attribute and use a class property instead.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit b1916b0a8d)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:42 +02:00
Heiko Thiery
922cd9e7f7 support/scripts/pkg-stats: store latest version in a dict
This patch changes the type of the latest_version variable to a dict.
This is for better readability/usability of the data. With this the json
output is more descriptive in later processing of the json output.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit c46e707182)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-28 17:41:35 +02:00
Peter Korsgaard
3957a66827 support/testing: drop explicit CGROUPFS_MOUNT from docker test
Since commit 4f8229653 (package/docker-engine: needs more runtime
dependencies), docker-engine now automatically pulls in cgroupfs-mount, so
drop the explicit handling of it in TestDockerCompose.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 4726cf9517)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-18 17:23:27 +02:00
Thomas Petazzoni
9e4ffdc8cf Makefile: properly account for custom tags in BR2_VERSION_FULL
BR2_VERSION_FULL is currently defined as follows:

  BR2_VERSION_FULL := $(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)

This BR2_VERSION_FULL value then gets used as the "VERSION" variable
in the /etc/os-release file.

The logic of "setlocalversion" is that if it is exactly on a tag, it
returns nothing.

If it is on a tag + a number of commits, then it returns only
-XYZ-gABC where XYZ is the number of commits since the last tag, and
ABC the git commit hash (these are extracted from git describe).

This output then gets concatenated to BR2_VERSION which gives
something like 2020.05 or 2020.05-00123-g5bc6a.

The issue is that when you're on a tag specific to your project, which
is not a Buildroot YYYY.MM tag, then the output of setlocalversion is
empty, and all you get as VERSION in os-release is $(BR2_VERSION)
which is not really nice. Worse, if you have another non-official
Buildroot tag between the last official Buildroot tag/version and
where you are, you will get $(BR2_VERSION)-XYZ-gABC, but XYZ will not
correspond to the number of commits since BR2_VERSION, but since the
last tag that "git describe" as found, which is clearly incorrect.

Here is an example: you're on master, "make print-version" (which
displays BR2_VERSION_FULL) will show:

$ make print-version
2020.08-git-00758-gc351877a6e

So far so good. Now, you create a tag say 5 commits "before" master,
and show BR2_VERSION_FULL again:

$ git tag -a -m "dummy tag" dummy-tag HEAD~5
$ make print-version
2020.08-git-00005-gc351877a6e

This makes you believe you are 5 commits above 2020.08, which is
absolutely wrong.

So this commit simplifies the logic of setlocalversion to simply
return what "git describe" provides, and not prepend $(BR2_VERSION) in
the main Makefile. Since official Buildroot tags match official
Buildroot version names, you get the same output when you're on an
official Buildroot tag, or some commits above a Buildroot tag. An in
other cases, you get a sensible output. The logic is also adjusted for
the Mercurial case.

In the above situation, with this commit applied, we get:

$ make print-version
dummy-tag-6-g6258cdddeb

(6 commits instead of 5 as we have this very commit applied, but at
least it's 6 commits on top of the dummy-tag)

Finally, if you're not using a version control system, setlocalversion
was already returning nothing, so in this case, the Makefile simply
sets BR2_VERSION_FULL to BR2_VERSION to preserve this behavior.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 98c99556e3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-08-13 20:06:23 +02:00
Adam Duskett
8e7a3bf68f package/openjdk-bin: install to host/usr/lib/jvm
Buildroot currently installs openjdk-bin to $(HOST_DIR)/ instead of the more
traditional (for java installations) $(HOST_DIR)/usr/lib/jvm.

As described in https://bugs.busybox.net/show_bug.cgi?id=13001

"Openjdk-bin provides it's own libfreetype.so and places it into
$(HOST_DIR)/lib/. This library causes build failures with the
host-xapp_mkfontscale package due to the overwritten libfreetype.so.

mkfontscale.o: In function `doDirectory':
mkfontscale.c:(.text+0x1a80): undefined reference to `FT_Get_BDF_Property'
collect2: error: ld returned 1 exit status

Reproducing the error is done by repeating the following steps.
make host-freetype
make host-openjdk-bin
make host-xapp_mkfontscale"

There are two options for fixing this problem:

 1) add host-freetype and host-lksctp-tools as dependencies to host-openjdk-bin
    and then remove the provided libfreetype.so and libsctp.so libraries
    in a post_extract_hook.

 2) change the installation directory from $(HOST_DIR)/ to
    $(HOST_DIR)/usr/lib/jvm just like the target OpenJDK package and
    copy the entire source directories contents to the above location.

The second option provides the following advantages:
  - the directory structure is consistent with how we handle the target OpenJDK.

  - the HOST_OPENJDK_BIN_INSTALL_CMDS step is simplified.

  - packages such as Maven require directories of which we are currently not
    copying. These missing directories cause programs such as Maven to crash
    when running with an error such as
    "Can't read cryptographic policy directory: unlimited."

  - does not miss any other libraries that solution 1 would not cope with
    (e.g. libzip.so from host-libzip, or libnet.so from not-yet existing
    host-libnet, or libsctp.so from not-yet existing host-lksctp-tools)

Because the second option is both simple, easier to implement, is low-impact,
and fixes the problems described above wholly, it is the best to implement.

To implement the above changes, we must also modify the following files in the
same patch to match the host's new directory paths:

 - openjdk.mk
 - openjdk-jni-test.mk
 - openjdk-hello-world.mk

To avoid having to change all those packages in the future, expose two
new variables, HOST_OPENJDK_BIN_ROOT_DIR which contains the path where
the openjdk-bin was installed in, and JAVAC, which contains the path to
the javac compiler (modeled after the way the autoconf et al. variables
are set and exposed).

Tested with:
./support/testing/run-tests -o out -d dl tests.package.test_openjdk.TestOpenJdk

Fixes: https://bugs.busybox.net/show_bug.cgi?id=13001

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[yann.morin.1998@free.fr:
  - introduce HOST_OPENJDK_BIN_ROOT_DIR and JAVAC
  - expand and tweak the commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit e9a02417f3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-15 21:54:03 +02:00
Yann E. MORIN
845d560371 support/download: fix git wrapper with submodules on older git versions
Older versions of git store the absolute path of the submodules'
repository as stored in the super-project, e.g.:

    $ cat some-submodule/.git
    gitdir: /path/to/super-project/.git/modules/some-submodule

Obviously, this is not very reproducible.

More recent versions of git, however, store relative paths, which
de-facto makes it reproducible.

Fix older versions by replacing the absolute paths with relative ones.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 8fe9894f65)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-15 21:50:38 +02:00
Yann E. MORIN
4bf0794bc3 core/br2-external: report better error messages
The error is misleading: it reports that no name was provided,
when in fact the external.desc file is missing.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>p
Reviewed-by: Romain Naour <romain.naour@gmail.com>
(cherry picked from commit c62e78a85b)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-15 21:27:38 +02:00
Yann E. MORIN
87f9527b55 core/br2-external: fix reporting errors
When a br2-external tree has an issue, e.g. a missing file, or does not
have a name, or the name uses invalid chars, we report that condition by
setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external,
which outputs it on stdout, and checked by the Makefile.

Before d027cd75d0, stdout was explicitly redirected to the generated
.mk file, with   exec >"${ofile}"   as the Makefile and Kconfig
fragments were generated each with their own call to the script, and
the validation phase would emit the BR2_EXTERNAL_ERROR variable in the
Makefile fragment.

But with d027cd75d0, both the Makefile and Kconfig fragments were now
generated with a single call to the script, and as such the semantics of
the scripts changed, and only each of the actual generators, do_mk and
do_kconfig, had their out put redirected. Which left do_validate with
the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

In turn, the stdout of the script would be interpreted by as part of the
Makefile. But this does not end up very well when a br2-external tree
indeed has an error:

  - missing a external.desc file:

    Makefile:184: *** multiple target patterns.  Stop.

  - empty external.desc file:

    Config.in:22: can't open file "output/.br2-external.in.paths"

So we must redirect the output of the validation step to the
Makefile fragment, so that the error message is correctly caught by the
top-level Makefile.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no
error, and thus the fragment is empty.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Romain Naour <romain.naour@gmail.com>
(cherry picked from commit 0ac7dcb73e)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-07-15 21:26:54 +02:00
Yann E. MORIN
8b31a049b1 support/gnuconfig: reference the correct sha1 we're using
In commit ff9f778c66 (support/gnuconfig: update to 2019-05-28), we
forgot to update the README to reference the sha1 we're using, keeping
the old one from 2016...

Update it now.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 3bf545da78)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-05-10 21:55:47 +02:00
Romain Naour
8cfa5004d0 support/testing: test_python_django: use the timeout_multiplier value
As reported by our gitlab runtime test [1] and on the mailing list
[2], the test_python_django is failing due to django server taking a
lot of time to start. Since the django server is started in background
through pexpect, we can't easily wait for the last startup line:

"January 01, 1970 - 00:00:41

Django version 3.0.4, using settings 'testsite.settings'

Starting development server at http://0.0.0.0:1234/

Quit the server with CONTROL-C."

In the failing gitlab job, we don't see such lines.

If we increase a lot the timout, the test passes.

Use timeout_multiplier introduced by [3] in order to adjust the
timeout.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/488816222

Tested:
https://gitlab.com/kubu93/buildroot/-/jobs/507458355

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/488816222
[2] http://lists.busybox.net/pipermail/buildroot/2020-April/279598.html
[3] 6e45e33f27

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 3e6b97e09e)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-04-29 23:42:22 +02:00
Yaroslav Syrytsia
453352c32a Makefile: make-4.3 now longer un-escapes \# in macros
make-4.3 shipped with a backward incompatible change in how sharp signs
are handled in macros. Previously, up to make 4.2, the sharp sign would
always start a comment, unless backslash-escaped, even in a macro or a
fucntion call.

Now, the sharp sign is no longer starting a comment when it appears
inside such a macro or function call. This behaviour was supposed to be
in force since 3.81, but was not; 4.3 fixed the code to match the doc.

As such, use of external toolchains is broken, as we use the sharp sign
in the copy_toolchain_sysroot macro, in shell variable expansion to
strip off any leading /: ${target\#/}.

Fix that by applying the workaround suggested in the release annoucement
[0], by using a variable to hold a sharp sign.

[0] https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html

Signed-off-by: Yaroslav Syrytsia <me@ys.lc>
[yann.morin.1998@free.fr:
  - move the SHARP_SIGN definition out of Makefile and into support/
  - expand the commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit 35c5cf56d2)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-04-08 13:25:28 +02:00
Thomas Petazzoni
6bba85671b support/scripts/check-kernel-headers.sh: do not print error for loose checks
The C program inside check-kernel-headers.sh has two checking mode: a
strict and a loose one.

In strict mode, we want the kernel headers version declared by the
user to match exactly the one of the toolchain.

In loose mode, we want the kernel headers version of the toolchain to
be greater than or equal to the one declared by the user: this is used
when we have a toolchain that has newer headers than the latest
version known by Buildroot.

However, in loose mode, we continue to show the "Incorrect kernel
headers version" message, even though we then return a zero error
code. This is very confusing: you see an error displayed on the
terminal, but the build goes on.

We fix that by first doing the loose check first, and returning 0 if
it succeeds. And then we move on with the strict check where we want
the version to be identical.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit c3c4b3dfa8)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-04-06 17:18:25 +02:00
Peter Korsgaard
5f5477b0ab Update for 2020.02
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-08 22:23:03 +01:00
Peter Korsgaard
22e833af5e Config.in: drop BR2_NEEDS_HOST_{JAVAC,JAR}
With classpath removed, no packages select these symbols any more - So drop
them and their corresponding logic in dependencies.sh / genrandconfig.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-03 23:55:48 +01:00
Titouan Christophe
54645c0b39 support/scripts/pkg-stats: clear multiprocessing pools after use
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-03-02 23:35:39 +01:00