Commit Graph

1161 Commits

Author SHA1 Message Date
Bence Szépkúti
340352ad74 Track upstreaming task in an issue
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
7ccbea6e47 Document the values in EXPECTED_FAILURES
Including the issues where the corresponding defects are tracked.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
355f8050cc Move to an updated fork of psa-arch-tests
The new fork was rebased on top of the upstream master, removing the
need for most of the downstream patches we carried.

On the other hand, the new fork includes a couple of fixes to problems
that were not addressed by the original fork, or were introduced with the
new version of psa-arch-tests.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
d6cf089b37 Explain why support_test_psa_compliance is needed
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
bd66d184ff Keep local clone around even if the test succeeds
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
eda2fb9583 Make directory creation code more compact
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
c2ca135f82 Add licence header to script
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:35 +01:00
Bence Szépkúti
559f1ce0a3 Make main() suitable to being called from python
Don't call sys.exit(), and don't clobber the working directory.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
19a124d677 Fix pylint errors
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
6025655598 Simplify regex and use named capture groups
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
faf7f1b554 Use print(end='') to silence double newline
Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
f54a9d2adf Indicate errors interleaved with test suite output
Indicate whether a success or failure is unexpected, or expected and
ignored as they happen.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
ab796e656b Make the changes easier to backport
The code replaced in this patch was not compatible with the
development_2.x branch.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Bence Szépkúti
9f84911d55 Run the PSA Compliance test suite in all.sh
This commit adds a component to all.sh which clones, builds and runs the
compliance test suite.

Signed-off-by: Bence Szépkúti <bence.szepkuti@arm.com>
2021-11-10 14:56:34 +01:00
Manuel Pégourié-Gonnard
0c81ba7856
Merge pull request #4944 from gilles-peskine-arm/remove-greentea-2.x
Backport 2.x: Remove on-target testing
2021-10-29 09:33:26 +02:00
Gilles Peskine
ff30bd0111 Always set a build type for cmake when building for testing
Set the build type to Release (-O2) when running CPU-intensive tests (ssl-opt,
or unit tests with debug features). A build type of Check (-Os) would be best
when the main objective of the build is to check for build errors or warnings
and there aren't many tests to run; in this commit there are no such test
cases to change. Only use cmake with no build type (which results in not
passing a -O option, and thus missing some GCC warnings) when exercising cmake
features.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Gilles Peskine
31fdda1262 Fix cmake invocation syntax
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Gilles Peskine
2531970772 Switch cmake -O2 builds around to where we test a lot
Use Release mode (-O2) for component_test_full_cmake_clang which runs SSL
tests.

To have some coverage with Check mode (which enables more compiler warnings
but compiles with -Os), change a few other builds that only run unit tests
at most to Check mode.

Don't add any new builds, to keep the total build volume down. We don't need
extensive coverage of all combinations, just a reasonable set.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Gilles Peskine
a0c51fb53e Clarify a comment
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Gilles Peskine
3daa83e99a Correct support function name
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Gilles Peskine
6fa69864a2 Build with -O2 when running ssl-opt
SSL testing benefits from faster executables, so use -O2 rather than -O1.
Some builds use -O1, but that's intended for jobs that only run unit tests,
where the build takes longer than the tests.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-28 14:28:53 +02:00
Manuel Pégourié-Gonnard
d599dc7f1b
Merge pull request #4932 from tom-daubney-arm/all.sh-subshells-2.x
Backport 2.x: Run all.sh components in a subshell
2021-10-28 09:33:22 +02:00
Gilles Peskine
a686bf46f7
Merge pull request #5083 from mprse/issue_5065_2x
backport 2.x: backport only new test in all.sh from #5072
2021-10-18 17:51:53 +02:00
Przemyslaw Stekiel
7ce979a4f5 all.sh: tests/scripts/all.sh full minus MBEDTLS_CHACHAPOLY_C without PSA_WANT_ALG_GCM and PSA_WANT_ALG_CHACHA20_POLY1305
Signed-off-by: Przemyslaw Stekiel <przemyslaw.stekiel@mobica.com>
2021-10-18 11:47:32 +02:00
Przemyslaw Stekiel
d6ead7ca84 Address review comments
Signed-off-by: Przemyslaw Stekiel <przemyslaw.stekiel@mobica.com>
2021-10-11 16:39:32 +02:00
Gilles Peskine
8eb3c95590 Add --restore option to clean up but not necessarily run components
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
6702ce9f73 Documentation improvements
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
a25c5675e2 Documentation improvement
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
4fa948feb7 Documentation improvements
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
8bfe15f291 Disable wildcards when checking for unsupported components
Otherwise $COMMAND_LINE_COMPONENTS would try to expand wildcard patterns
based on files in the current directory.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
76d40fa977 Improve the detection of keep-going commands
Have simpler patterns related to 'test' (the central objective being to keep
going if 'make test' or 'tests/...' fails, but not if 'make tests' fails).

Add 'cd' as a can't-keep-going command.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
202e9b4ed4 Improve --error-test reporting
Count invocations from 1 to n instead of n to 1.

Explain how changing the loop variable would cause an error if the function
was not executed in a subshell.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
cb4bfac119 Clarify some comments
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
ef64e7fb3b Don't unconditionally restore **/Makefile
all.sh restores **/Makefile from git in case the version in the worktree was
from doing a cmake in-tree build. Instead of doing this unconditionally, do
it only if the toplevel Makefile seems to have been automatically
generated (by cmake or otherwise, e.g. by mbedtls-prepare-build). This way
all.sh no longer silently wipes changes made to Makefile but not committed yet.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
24bdf02e4e Don't restore *config.h before backing it up
Back up the config files at the beginning of all.sh, rather than before each
component. In particular, create the backup before running cleanup for the
first time. This fixes #3139 (all.sh using a config.h.bak from a previous
job), and makes all.sh more robust against accidentally using a modified
config.h midway through because a component messed with the backup.

Use a different extension (*.all.bak rather than *.bak) for the backups.
This is necessary to ensure that auxiliary scripts such as depends*.pl that
make their own backup don't remove all.sh's backup, which the code from this
commit does not support.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-11 14:29:16 +01:00
Gilles Peskine
9b35640e3f Generate cpp_dummy_build.cpp dynamically
Generate programs/test/cpp_dummy_build.cpp dynamically instead of
maintaining it manually. This removes the need to update it when the list of
headers changes.

Include all the headers unconditionally except for the ones that cannot be
included directly.

Support this dynamic generation both with make and with cmake.

Adapt all.sh accordingly. Remove the redundant C build from
component_build_default_make_gcc_and_cxx (it was also done in
component_test_default_out_of_box), leaving a component_test_make_cxx. Also
run the C++ program, because why not. Do this in the full configuration
which may catch a bit more problems in headers.

Fixes #2570 for good.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Thomas Daubney <>
2021-10-11 14:29:16 +01:00
Przemyslaw Stekiel
81be6e275c enerate_psa_tests.py fix format
Signed-off-by: Przemyslaw Stekiel <przemyslaw.stekiel@mobica.com>
2021-10-08 15:40:16 +02:00
Przemyslaw Stekiel
330e460ef3 Addapt psa_generate_key() tests
Signed-off-by: Przemyslaw Stekiel <przemyslaw.stekiel@mobica.com>
2021-10-08 12:56:45 +02:00
Gilles Peskine
85229acf04 Limit make parallelism to the number of CPUs
Don't default to unbridled -j, which causes a load spike and isn't really
faster.

"Number of CPUs" is implemented here as a reasonable compromise between
portability, correctness and simplicity. This is just a default that can be
overridden by setting MAKEFLAGS in the environment.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-10-06 17:21:57 +02:00
Gilles Peskine
f315d6a718
Merge pull request #4957 from gilles-peskine-arm/make-tests-v-2.2x
Backport 2.x: Facilitate reproducing unit tests from all.sh
2021-09-30 16:08:56 +02:00
Gilles Peskine
f313336c01 Remove barely-used redirect functions
redirect_out was no longer used and redirect_err was only used to
quiet dd. Change the dd invocation to only print diagnostics on
error (on platforms where this is possible).

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
7c220d7d37 Keep going after a shell "[" a.k.a. "test" fails
This is necessary to actually keep going and finish the
component-specific cleanup in component_test_cmake_out_of_source if
ssl-opt.err is non-empty.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
1d55995d32 Remove code that is useless now that components run in a subshell
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
e8056c5e31 Complain if an unsupported component is explicitly requested
In all.sh, when an explicit list of components is specified, error out
if one of the components is not known or not supported. Patterns that
happen to match zero components are still effectively ignored.

Fix #2783

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
39a3b11006 Better not function
In the `not` function, in keep-going mode, arrange to report the
failing command (rather than `"$@"`).

Note that the `!` keyword should not be used, because failures with
`!` are not reported properly.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
4848d7bb45 Fix double reporting when the last command of a function fails
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:59:00 +01:00
Gilles Peskine
f6d29c6a9e Stop dispatching through obsolete functions
Remove the obsolete functions record_status and if_build_succeeded.
They didn't affect error detection, but they made error reporting
worse since $BASH_COMMAND would be the unexpanded "$@".

Keep the function definitions for the sake of pull requests using them
that may still be in flight.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:58:55 +01:00
Gilles Peskine
0a7984f1dd component_test_cmake_out_of_source: simplify and fix error handling
Remove ssl-opt.err even if it's empty.

Call cat unconditionally: it'll have no visible effect if the file is
empty.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:55:21 +01:00
Gilles Peskine
8ab2994eb5 Detect errors on the left-hand side of a pipeline
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:55:21 +01:00
Gilles Peskine
7105a33906 Run each component in a subshell and handle errors more robustly
This commit completely rewrites keep-going mode. Instead of relying
solely on "set -e", which has some subtle limitations (such as being
off anywhere inside a conditional), use an ERR trap to record errors.

Run each component in a subshell. This way a component can set
environment variables, change the current directory, etc., without
affecting other components.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2021-09-29 14:55:21 +01:00