For details, see https://github.com/snyk/zip-slip-vulnerability
Older python versions do not validate that the extracted files are inside
the target directory. Detect and error out on evil paths before extracting
.zip / .tar file.
Given the scope of this (zip issue was fixed in python 2.7.4, released
2013-04-06, scanpypi is only used by a developer when adding a new python
package), the security impact is fairly minimal, but it is good to get it
fixed anyway.
Reported-by: Bas van Schaik <security-reports@semmle.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit a83e30ad63)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This script currently uses "/usr/bin/env python" as shebang but it does
not really support Python3. Instead of limiting the script to Python2,
fix it to support both versions.
So change all imports to absolute imports because Python3 follows PEP328
and dropped implicit relative imports.
In order to avoid errors when decoding files with the default 'utf-8'
codec, use errors="surrogateescape" when opening files, the docs for
open() states: "This is useful for processing files in an unknown
encoding.". This argument is not compatible with Python2 open() so
import 'six' to use it only when running in Python3.
As a consequence the file handler becomes explicit, so use it to close()
the file after it got processed.
This "surrogateescape" is a simple alternative to the complete solution
of opening files with "rb" and changing all functions in the lib*.py
files to use bytes objects instead of strings. The only case we can have
non-ascii/non-utf-8 files being checked by the script are for patch
files when the upstream file to be patched is not ascii or utf-8. There
is currently one case in the tree:
package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 02b165dc71)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 62d5558f76 (utils/get-developers: make it callable from elsewhere
than the toplevel directory) tried to fix this by passing in the toplevel
directory when the DEVELOPERS file is parsed.
Unfortunately this is not enough, as E.G. also the paths listed in the
patches are relative to the toplevel directory, causing it to not match the
entries in the DEVELOPERS file.
In concept this can be fixed by also passing the toplevel directory to the
Developers class, but the simplest solution is just to chdir to the toplevel
Buildroot directory before calling any of the getdeveloperlib functions.
This does require us to finish parsing command line arguments (which opens
the provided patch files) to not get into trouble with relative paths to
patches before chdir'ing / initializing getdeveloperlib.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 45aabcddc5)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This reverts commit 62d5558f76.
This actually does not work, as patches contain paths relative to the
toplevel directory as well.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 74fbc20504)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When adding the custom BR2_WGET value in the configuration,
genrandconfig forgets to add a newline. Due to this, the next option
that is added is printed on the same line as BR2_WGET="", which causes
it to be ignored.
Due to this, in all builds, the line right after BR2_WGET was
ignored. It could have been BR2_ENABLE_DEBUG, BR2_INIT_BUSYBOX,
BR2_INIT_SYSTEMD, BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV,
BR2_STATIC_LIBS or BR2_PACKAGE_PYTHON_PY_ONLY depending on the
randomization.
Fix that by adding a proper newline at the end of the BR2_WGET option.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 3eb49f59d6)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some packages have archive name that is different from package name.
For example websocket-client's archive name is websocket_client-*.tar.gz.
scanpypi expects the temporary extract folder to be:
/tmp-folder/BR-package-name/PyPI-packagename-and-version
In the case of websocket-client package the real extraction folder
will be different from the expected one because of the '_' in the
archive file name.
Use archive file name instead of package name to specify the extraction
folder. As the version is already part of this file, we don't need to
specify it.
Bonus: remove obsolete "return None, None" as the function doesn't return
anything. OSError class doesn't provide "message" member, so replace it
with "strerror".
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=11251
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Asaf Kahlon <asafka7@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit fd29797f65)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The utils/diffconfig script works only on variables with the BR2_
prefix. This is OK for Buildroot [def]configs since this is the prefix
for all user-facing variables, but it prevents using the same script
to compare configs from kconfig-based packages.
Remove the BR2_ restriction, allowing usage such as:
./utils/diffconfig \
board/qemu/xtensa-lx60/linux.config \
board/qemu/xtensa-lx60/linux-nommu.config
Signed-off-by: Marcel Patzlaff <m.patzlaff@pilz.de>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit bf9ccfc37b)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When used without spdx_lookup the BSD licence cannot be
detected correctly because many Python packages just specify
BSD without the exact version in their metadata. So add a
special message warning the user instead of the licence id.
Bonus: fix typo.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit d05e41eb1a)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
get-developers tries to open DEVELOPERS in the current directory, so it
breaks when calling it from elsewhere than the toplevel Buildroot directory.
Traceback (most recent call last):
File "../utils/get-developers", line 107, in <module>
__main__()
File "../utils/get-developers", line 26, in __main__
devs = getdeveloperlib.parse_developers(os.path.dirname()
File "/home/peko/source/buildroot/utils/getdeveloperlib.py", line 161, in parse_developers
with open(os.path.join(basepath, "DEVELOPERS"), "r") as f:
IOError: [Errno 2] No such file or directory: '/home/peko/source/buildroot/output-foo/DEVELOPERS'
Fix it by instead figuring out where the DEVELOPERS file is relative to the
location of get-developers (E.G. one level up).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[Arnout:
- add realpath to support a symlinked get-developers script;
- pass devs_dir argument to check_developers() to support -c in subdir;
- convert basepath to absolute path to support -f option.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(cherry picked from commit 62d5558f76)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This fixes a syntax error introduced in bcf2ed5cc3.
Output before the patch:
$ ./utils/get-developers outgoing/*
File "./utils/get-developers", line 97
print dev
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean
print(dev)?
Output after the patch:
$ ./utils/get-developers outgoing/*
git send-email --to buildroot@buildroot.org
Signed-off-by: Grégoire Delattre <gregoire.delattre@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
(cherry picked from commit 8320ad3341)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When called with a list of patches, get-developers prints the entire git
send-email invocation line:
./utils/get-developers 0001-git-security-bump-to-version-2.16.5.patch
git send-email --to buildroot@buildroot.org --cc "Matt Weber <matthew.weber@rockwellcollins.com>"
This may be handy when creating an entire patch series and editing a cover
letter, but it does mean that this has to be explicitly executed and
get-developers cannot be used directly by the --cc-cmd option of git
send-email to automatically CC affected developers.
So add an -e flag to only let get-developers print the email addresses of
the affected developers in the one-email-per-line format expected by git
send-email, similar to how get_maintainer.pl works in the Linux kernel.
With this and a suitable git configuration:
git config sendemail.to buildroot@buildroot.org
git config sendemail.ccCmd "$(pwd)/utils/get-developers -e"
You can simply do:
git send-email master
To automatically mail the buildroot list and CC affected developers on
patches.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit bcf2ed5cc3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, the stdout is consigned to oblivion, while the stderr is not
redirected at all. So, when the configuration under test redefines a
symbol, like is the case when testing busybox for example, there is an
ugly warning.
So, just redirect both stdout and stderr to the logfile.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(cherry picked from commit 2664022787)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
A number of autobuilder failures are due to the fact that autobuilder
instances use old distributions, with old SSL certificates, and
therefore wget aborts with an error "The certificate of `xyz.org' is
not trusted.".
In order to avoid such failures that are not very interesting in the
context of the autobuilders, we pass --no-check-certificate to
wget. The integrity of the downloaded files is anyway verified by the
hashes, and this is only meant to be used in the context of
testing/CI, not in production.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(cherry picked from commit 0866a280e4)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fix these warnings:
E101 indentation contains mixed spaces and tabs
E128 continuation line under-indented for visual indent
E231 missing whitespace after ','
E261 at least two spaces before inline comment
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
W191 indentation contains tabs
Ignore these warnings:
E402 module level import not at top of file
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Fix these warnings:
E202 whitespace before ']'
E203 whitespace before ':'
E302 expected 2 blank lines, found 1
E305 expected 2 blank lines after class or function definition, found 1
E711 comparison to None should be 'if cond is None:'
E741 ambiguous variable name 'l'
F401 'sys' imported but unused
W391 blank line at end of file
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Ignore these warnings:
F401 'lib.ConsecutiveEmptyLines' imported but unused
And remove comments that are not needed anymore.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Depending on how setup.py reads requirements files empty elements can occur.
This patch takes care, that such elements will be ignored and don't crash
the scanpypi script.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Store the list of detected licence files in the main object and
automatically add their sha256 checksums when creating *.hash file.
Bonus: fix wrong indentation.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Use spdx_lookup package to compare packages' license file texts
with SPDX database.
This feature is optional.
Bonus: fix wrong indentation.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Just AUTORECONF = NO is redundant.
Just HOST_AUTORECONF = NO is redundant.
But the combination of AUTORECONF = YES + HOST_AUTORECONF = NO is valid.
So basically for all variables that have inheritance between target and
host, having the host variant of the variable set the variable value
back to its default is correct if the target variable is set.
Instead of increasing complexity of the script to fully detect this
case, ignore the host flag set to its default value as it can be
overriding a non-default value inherited from the equivalent target
flag.
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reported-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Diffconfig is a simple utility for comparing two configuration files.
See usage in the script for more info.
Borrowed from the Linux kernel source code and adapted to Buildroot.
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Default prefix is set to `BR2_` but may be overidden by setting
BR2_PREFIX.
Example usage:
Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --enable GNUPG
Check state of config option `BR2_PACKAGE_GNUPG`:
./utils/config --package --state GNUPG
y
Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --disable GNUPG
Set `BR2_TARGET_GENERIC_ISSUE` to "Welcome":
./utils/config --set-str TARGET_GENERIC_ISSUE "Welcome"
Copied from the Linux kernel (4.13-rc6) source code and adapted to
Buildroot.
Thanks to Andi Kleen who is the original author of this script.
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
[Arnout: merge the two tr invications]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The DEVELOPERS file lists file patterns, but the get-developers script
only supports getting packages and CPU architectures. Some files are
neither of those, e.g. defconfigs, support files, package infras, ...
Add a '-f' option that allows to give a list of files for which to get
the developer(s).
Note that the DEVELOPERS file contains directories as well as files,
and the directories are not expanded into individual files. Therefore,
we have to use f.startswith(devfile) to match a directory. This assumes
that the directory name ends with /, otherwise 'package/python' would
also match 'package/python-foo'. Since we make sure that directories do
end with / in the DEVELOPERS file, and since false positives are not
much of an issue in this case, this isn't a problem.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When testing a package with test-pkg, it may be useful to override the
set of toolchains used. For example:
- to test with toolchains used in your company;
- to test against a subset that is known to be problematic;
- to use only toolchains you already have available locally when you
have no network access.
Add an option to use an alternate CSV file containing the config
fragments of toolchains to try.
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.
The --toolchains-url option is renamed to --toolchains-csv.
The paths in the toolchains_csv file should be either absolute, or
relative to buildrootdir.
After this change, the script should be called from autobuild-run as:
subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
"-o", outputdir, "-b", srcdir,
"--toolchains-csv", kwargs['toolchains_csv']],
stdout=devnull, stderr=log)
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Now we have the toolchain config fragments in the buildroot directory
itself, it is no longer necessary to fetch it from the toolchain URL.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This has a number of side-effects which must be handled.
The lines in minimal.config may be overridden by the random lines added
by amending the configuration, so is_toolchain_usable() shouldn't take
those into account, or indeed the random lines added. Therefore, make
a copy of the config before appending minimal.config and the random
lines. While we're at it, rename the variable to the more appropriate
toolchainconfig.
minimal.config sets BR2_INIT_NONE=y, but we really also want to test
with BR2_INIT_BUSYBOX=y. Therefore, add a random line to use the
busybox init system. We set its probability rather high. The
probabilities of systemd and eudev are increased since they're now
in the else branch of BR2_INIT_BUSYBOX, which halves the probability
that we even get there.
We now also generate configurations without busybox. Previously,
busybox was almost always selected due to BR2_INIT_BUSYBOX=y. Only if
systemd is selected there was a chance to build without busybox.
We now set BR2_SYSTEM_BIN_SH_NONE=y, the other /bin/sh options are
never tested. However, this is not really something that is relevant
to test in the autobuilders.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When outputdir is 'output' (the default), genrandconfig didn't work
correctly because it expects the configfile in outputdir, while
Buildroot puts it in the buildroot directory.
Fix this by explicitly checking if outputdir == buildrootdir/output.
Because abspath is used for both paths, string comparison works
reliably.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The path to the .config file is calculated in several places - replace
it with a single calculation, and pass configfile as an argument
to is_toolchain_usable and fixup_config. These functions also don't
need outputdir any more.
This makes it easier to fix the case when configfile is not in
outputdir.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The --instance argument is just an artifact of genrandconfig's
history as part of autobuild-run. It is much more logical to pass
the output directory and the buildroot directory as arguments, with
sane defaults.
This also allows us to remove the hack of creating a symlink in the
instance directory if it doesn't exist yet.
Note that the default outputdir 'output' doesn't work yet, because in
that case Buildroot will put the config file in the buildroot directory
instead of the output directory. This will be fixed in a follow-up
patch.
After this change, the script should be called from autobuild-run as:
subprocess.call([os.path.join(srcdir, "utils/genrandconfig"),
"-o", outputdir, "-b", srcdir,
"--toolchains-url", kwargs['toolchains_url']],
stdout=devnull, stderr=log)
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The output of genrandconfig is currently very terse, which is annoying
for debugging the script or generally seeing what is going on. Also the
timing information added by log_write isn't very useful when the script
is used stand-alone.
In the new setup, (verbose) output goes to stdout and error output goes
to stderr. Also the "INFO: generate the configuration" message is
eliminated - it should go in the autobuild-run script.
We also add an explicit message when a toolchain can't be used after
the first defconfig, otherwise autobuild-run will just silently
restart.
Note that, since the output of make is no longer redirected to
/dev/null, we get one more message on stderr that will be recorded in
the autobuilder's log file: KCONFIG_SEED=0xXXXXXXXX.
This approach allows us to optimise the error handling to use
exceptions, where appropriate, which can be caught at the top level and
converted to an error message to stderr. This, in turn, allows us to use
subprocess.check_call, which eliminates a lot of conditions.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The SystemInfo class is instantiated globally and passed down to all
functions, but it is really only used in fixup_config. So instead,
instantiate it there.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
kwargs is a left-over from the use of docopt, it's better to use
argparse's Namespace object directly.
In addition, most functions use just one or two fields of args, so
these can just as well be passed directly as arguments to the function.
Particularly for outputdir it doesn't make sense to reconstruct it all
the time.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Warnings fixed:
E731 do not assign a lambda expression, use a def
-> urlopen_closing is defined with a def. urlopen is not used
elsewhere so inlined.
E302 expected 2 blank lines
E501 line too long
-> long lines due to a long string are NOT split
E701 multiple statements on one line (colon)
E722 do not use bare except'
-> use "except Exception", so KeyInterrupt and SystemExit are still
passed. We never intended to catch those.
E741 ambiguous variable name 'l'
-> variable name is replaced with the much more descriptive
toolchains_csv
E271 multiple spaces after keyword
E231 missing whitespace after ','
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Popen is more complicated and more difficult to understand.
check_output raises an exception if the exit code is non-zero, but
that's probably what we want if ldd can't be executed.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
