Look for changes that remove X509/SSL functionality.
```
git diff 'HEAD^{/^Merge}~1' HEAD --diff-filter=M -- . ':!library/error.c' ':!library/version_features.c' ':!programs/test/query_config.c' ':!visualc' ':!*.pdf' ':!*.der' | grep -E "^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_"
```
All of these removals are in `config.h` or `check_config.h`. Selectively revert the differences in these two files.
```
git diff 'HEAD^{/^Merge}~1' 'HEAD^{/^Merge}' include/mbedtls/config.h include/mbedtls/check_config.h | git apply -p1 -R
```
* `include/mbedtls/check_config.h`:
* ARIA for GCM: don't remove it.
* `MBEDTLS_PSA_CRYPTO_SE_C`: don't remove it.
* `MBEDTLS_SHA512_NO_SHA384`: don't remove it.
* `MBEDTLS_SSL_DTLS_CONNECTION_ID`: restore it.
* `include/mbedtls/config.h`:
* warning about `MBEDTLS_ECDSA_SIGN_ALT`: don't remove it.
* `MBEDTLS_CTR_DRBG_USE_128_BIT_KEY` full paragraph: don't remove it.
* `MBEDTLS_PSA_CRYPTO_KEY_FILE_ID_ENCODES_OWNER`: don't remove it.
* `MBEDTLS_SHA512_SMALLER`…: don't remove it.
* `MBEDTLS_SSL_RECORD_CHECKING`: restore it.
* `MBEDTLS_SSL_CONTEXT_SERIALIZATION`: restore it.
* `MBEDTLS_USE_PSA_CRYPTO` note: don't restore the tls version.
* `MBEDTLS_USE_PSA_CRYPTO` warning: restore the tls version.
* `MBEDTLS_CMAC_C`: restore it to being disabled by default. It's a minor API change in Mbed TLS because it changes the layout of `mbedtls_cipher_context_t`.
* `MBEDTLS_CTR_DRBG_C`: don't restore the older version of the description from tls.
* `MBEDTLS_GCM_C`: don't restore the older description from tls.
* `MBEDTLS_PSA_CRYPTO_C`: don't restore `crypto/`.
* `MBEDTLS_PSA_CRYPTO_SE_C`: don't remove it.
* `MBEDTLS_PSA_CRYPTO_STORAGE_C`: don't restore `crypto/`. Don't disable it by default.
* `MBEDTLS_PSA_ITS_FILE_C`: don't restore, like for ``MBEDTLS_PSA_CRYPTO_STORAGE_C``.
* `MBEDTLS_CTR_DRBG_USE_128_BIT_KEY` single line: don't restore it since there is now a full paragraph in the proper section above.
* `MBEDTLS_SSL_CID_IN_LEN_MAX`…: restore it.
* `MBEDTLS_TLS_DEFAULT_ALLOW_SHA1_IN_CERTIFICATES`: restore the version without a space to format the line like the others.
Look for any other invocation of test scripts that was removed: look for a change that removes a line (`^-.*…`) containing one of the names of a test script (without its path because it may be used with a relative path). Look for `ssl-opt.sh` and `compat.sh`, as well any file in `tests/scripts` that only exists in tls.
```
git diff 'HEAD^{/^Merge}~1' HEAD --diff-filter=M -- . ':!library/error.c' ':!library/version_features.c' ':!programs/test/query_config.c' ':!visualc' ':!*.pdf' ':!*.der' | grep -E "^-.*($(comm -23 <(git ls-tree -r --name-only 'HEAD^{/^Merge}~1') <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development 'HEAD^{/^Merge}^2')) | sed -n 's!^tests/scripts/!!p' | sed 's/\./\\./g' | tr '\n' '|')ssl-opt\.sh|compat\.sh)" | grep -v '^---'
```
This only turns up changes in `basic-in-docker.sh`.
The merge of mbed-crypto removed some tls coverage. Restore it. Also
remove references to the `crypto` subdirectory brought by the mbedtls
side of the merge. In more detail:
* `tests/scripts/all.sh`:
* `fuzz` in comments (×2): restore it.
* `CTEST_OUTPUT_ON_FAILURE=1`: don't remove it.
* `cd crypto` for `make clean`: don't restore it.
* `cleanup`: do restore `programs/fuzz/Makefile`. Don't go into `crypto`. Keep only one copy of the calls to `rm` in `cmake_subproject`.
* Comment legacy options: don't remove it.
* `crypto/Makefile` and `pre_check_seedfile`: don't restore either. See below regarding the lack of need for `pre_check_seedfile`.
* blank line in `pre_print_configuration`: restore it.
* blank line before `#### Build and test`: restore it.
* SSL tests in `component_test_full_cmake_gcc_asan` and zlib components: restore it.
* `component_test_no_pem_no_fs` (×2): the merge placed two copies in different locations. Reconcile them: unset PSA storage like in crypto, and call `ssl-opt.sh` like in tls. Put the merged version at the tls location.
* `component_test_everest`: do add it at the tls location.
* `component_test_small_mbedtls_ssl_dtls_max_buffering`: restore the tls value.
* `component_test_new_ecdh_context`…: move `component_test_new_ecdh_context` before `component_test_everest` and add a calls to `compat.sh` and `ssl-opt.sh` like in `component_test_everest`. Remove the redundant crypto-only `component_test_everest`. Don't remove `component_test_psa_collect_statuses`.
* `component_test_full_cmake_clang`: don't remove `clang` in the `msg` call. Don't remove the call to `test_psa_constant_names.py`.
* `component_test_full_make_gcc_o0`: remove it. It's subsumed by `component_test_gcc_opt`.
* `component_build_deprecated`: don't remove anything.
* `component_test_memory_buffer_allocator`: restore `ssl-opt.sh`.
* `component_test_when_no_ciphersuites_have_mac`: restore it.
* `component_test_platform_calloc_macro`: don't restore `unset MBEDTLS_MEMORY_BUFFER_ALLOC_C` which is now redundant. Don't restore explicit flags instead of `$ASAN_CFLAGS`.
* `component_test_aes_fewer_tables`…: don't remove it.
* `component_test_m32_o1`: restore SSL testing.
* `component_test_m32_everest`: restore SSL testing.
* `component_test_min_mpi_window_size`…: don't remove it.
* `component_test_valgrind`: do restore the tls version of the comment.
* `run_component`: don't remove the seedfile creation. This is better than `pre_check_seedfile` (see below).
* `pre_check_seedfile`: don't restore it. `pre_check_seedfile` (from tls) creates a seedfile once and for all. This is not good enough if a component fails in such a way as to leave a broken seedfile, or if a component leaves a seedfile with a size that's wrong for the next component to run. Instead (from crypto), `run_component` creates a sufficiently large seedfile before each component.
Merge `unremove-non-crypto` into `mbedtls/development`. The branch
`unremove-non-crypto` was obtained by starting from `mbed-crypto/development`,
then reverting many commits that removed X.509 and TLS functionality when Mbed
Crypto forked from Mbed TLS (the “unremoval”), then make a few tweaks to
facilitate the merge.
The unremoval step restored old versions of some tls files. If a file doesn't
exist in mbed-crypto, check out the mbedtls version, regardless of what
happened during the unremoval of tls files in the crypto tree. Also
unconditionally take the mbedtls version of a few files where the
modifications are completely project-specific and are not relevant in
mbed-crypto:
* `.github/issue_template.md`: completely different. We may want to reconcile
them independently as a follow-up.
* `.travis.yml`: would only be reverted to an earlier tls version.
* `README.md`: completely different. We may want to reconcile them
independently as a follow-up.
* `doxygen/input/doc_mainpage.h`: the changes in crypto were minimal and not
relevant except as a stopgap as mbed-crypto did not have its own product
versioning in the Doxygen documentation.
* `tests/.jenkins/Jenkinsfile`: completely different.
* `tests/data_files/Makefile`: there were no changes in mbed-crypto,
but the unremoval step restored an old version.
Shell script for everything to do after the merge apart from the conflict
resolution:
```
tls_files=($(comm -23 <(git ls-tree -r --name-only HEAD) <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development MERGE_HEAD))))
tls_files+=($tls_files .github/issue_template.md .travis.yml README.md doxygen/input/doc_mainpage.h tests/.jenkins/Jenkinsfile tests/data_files/Makefile)
git checkout --theirs HEAD -- $tls_files
git add -- $tls_files
```
Resolve the remaining conflicts:
* `library/CMakeLists.txt`:
* Keep the TLS definition of `src_crypto`
* `USE_SHARED_MBEDTLS_LIBRARY`: keep all three libraries, with both
`include` and `crypto/include` in `target_include_directories`, all with
version `2.21.0`.
* `programs/Makefile`:
* Reconcile the APPS lists (add/add from a differently-formatted common
ancestor): insert the `psa/*` from crypto into the tls list.
* Keep the `fuzz` target defined only in tls version.
* Keep the recipe (only in tls version) cleaning `ssl_pthread_server`
stuff for the `clean` target.
* `scripts/config.py`:
* `include_in_full`: add/add conflict. Keep both.
* `tests/scripts/all.sh`:
* `component_test_no_use_psa_crypto_full_cmake_asan`: partially old
version in crypto. Take the tls version.
* `component_test_malloc_0_null` and more: take
`component_test_malloc_0_null` from crypto (with `config.py` rather than
`config.pl`, and with `$ASAN_FLAGS` rather than an explicit list), but
add the call to `ssl-opt.sh` from tls. Take the other components from
crypto.
With this commit, building and running the unit tests with both `make ` and
`cmake` work in the default configuration on Linux. Other platforms, build
systems and configurations are likely not to work, and there is some
regression in test coverage.
There is some loss of functionality because the unremoval step restored older
versions of tls content. This commit contains the latest tls version of
tls-only files, but some changes from the tls side in files that existed on
both sides have regressed. Most problematic changes are hunks that remove some
tls-specific feature and contain either a C preprocessor symbol identifying a
tls-specific module or option, or the name of a tls-specific file. Hunks
that remove a tls-specific preprocessor symbol can be identified with the
regular expression `^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_`.
Subsequent commits will revert a few parts of the patch from this merge commit
in order to restore the tls functionality that it removes, ensure that the
test coverage includes what was covered in either branch, and fix test
failures.
Even if other higher-level libraries were added, these programs would
only link with the crypto library, which is the one that contains
platform functions.
This reverts commit bea98b4581.
Conflicts:
* programs/Makefile:
* APPS: the layout of the definition has changed. re-add dh_client
and dh_server appropriately.
Run scripts/generate_visualc_files.pl to account for the added programs.
This reverts commit 9b90f2e294.
Conflicts:
* tests/scripts/all.sh: do the same changes, dancing around the new
outcome file feature and components added in the same places.
Make sure that the components that are getting added back are at the
same locations as where they are now in mbedtls.
This reverts commit 9afb2e9921.
Conflicts:
* include/CMakeLists.txt
* "Make config.h available" comment: there has been a change
adjacent to where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* compat.sh: there has been a change immediately before where it was
removed. Just re-add what was removed.
This reverts commit 120d571e8e.
Conflicts:
* tests/CMakeLists.txt:
* target_include_directories: the instruction whose addition is to
be reverted has changed. Remove what is there now.
This reverts commit b478bb6ddb.
Conflicts:
* tests/CMakeLists.txt: revert the introduction of exe_name, but keep
the addition of ${CMAKE_SOURCE_DIR}/crypto/library/ to
target_include_directories.
This reverts commit d832f187f7.
Conflicts:
* CMakeLists.txt:
* USE_PKCS11_HELPER_LIBRARY: there has been a change immediately before
where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* USE_PKCS11_HELPER_LIBRARY: there has been a change immediately before
where it was removed. Just re-add what was removed.
This reverts commit d874a1fd14.
Conflicts:
* CMakeLists.txt:
* ENABLE_ZLIB_SUPPORT: there has been a change immediately after
where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* ENABLE_ZLIB_SUPPORT: there has been a change immediately after
where it was removed. Just re-add what was removed.
This reverts commit 8298d70bee.
Conflicts:
* library/Makefile: removal of SOEXT_X509 and SOEXT_TLS vs change of
value of SOEXT_CRYPTO. Keep all, with the new value of SOEXT_CRYPTO.
This reverts commit 1ad37309e4.
Conflicts:
* tests/scripts/test-ref-configs.pl:
* config-ccm-psk-tls1_2.h: there has been an addition of
"config-symmetric-only.h" at the place where the configurations
that are added back were removed. Keep this configuration.
This reverts commit 1c66e48670.
Conflicts:
* include/mbedtls/check_config.h:
* MBEDTLS_SSL_PROTO_SSL3: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed. Re-add it
after (alphabetical order).
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* include/mbedtls/config.h:
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* library/version_features.c: re-generate by running
scripts/generate_features.pl.
* programs/test/query_config.c: re-generate by running
scripts/generate_query_config.pl.
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
* Revert removing one symbol (MBEDTLS_NET_C) from the list of symbols
to exclude from baremetal.
* scripts/footprint.sh:
* Re-add the line to unset MBEDTLS_NET_C, but with config.py instead of
config.pl.
* tests/scripts/all.sh:
* component_test_no_platform: re-add the line to unset MBEDTLS_NET_C, but
with config.py instead of config.pl.
* component_build_arm_none_eabi_gcc,
component_build_arm_none_eabi_gcc_no_udbl_division,
component_build_arm_none_eabi_gcc_no_64bit_multiplication,
component_build_armcc: these components now use the baremetal
configuration, so they do not need to turn off MBEDTLS_NET_C explicitly.
This reverts commit bb1f701212.
* include/mbedtls/check_config.h:
* MBEDTLS_X509_RSASSA_PSS_SUPPORT: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed.
Re-add it before MBEDTLS_SHA512_NO_SHA384 to keep it grouped
with MBEDTLS_RSA_C.
Conflicts:
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
This reverts commit 356acc82ad.
Conflicts:
* scripts/generate_errors.pl: a line adjacent to a changed line has
independently changed in the meantime. Just revert the change done
in the commit that's being reverted.
Although the 'flags' variable is not checked or used after a call to
mbedtls_ssl_check_cert_usage, it might be in the future. With this fix, after
each iteration, the flags will apply only to the most recent certificate, not
to any of the previous ones checked. This fix also stops any reads and
writes via a '|=' from/to an uninitialized variable happening.
This commit fixes#2444.
Signed-off-by: Andrzej Kurek <andrzej.kurek@arm.com>
Add a description of MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY to the
documentation, as suggested by hanno-arm.
Signed-off-by: Andrzej Kurek <andrzej.kurek@arm.com>
Pylint when installed as a distro package can be installed as pylint3, whilst as
a PEP egg, it can be installed as pylint.
This commit changes the scripts to first use pylint if installed, and optionally
look for pylint3 if not installed. This is to allow a preference for the PEP
version over the distro version, assuming the PEP one is more likely to be
the correct one.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
