Investigate running CI with hardened dependencies vs running CI with valgrind
In legacy/trac#30674 (moved), we investigated why running with --enable-fragile-hardening had missed a memory leak that valgrind could successfully catch. The answer turned out to be that we had not compiled our dependencies with sanitizers enabled -- so they didn't catch memory leaks that happened inside our dependencies.
Assuming we want CI to catch this kind of bug (and we do!) the alternatives seem to be: build our dependencies with sanitizers, or run with valgrind.
Teor made the following notes about deployment and evaluations:
Hardened dependencies:
- We know we can harden dependencies
- Hardened dependencies may cause CI failures due to bugs in dependencies
- Hardened dependencies may be slower
- We probably won't rebuild libc and other large libraries in hardened mode
- We don't know if valgrind or hardened builds provide better coverage of the kinds of coding errors we typically make
- It might be complicated to configure builds for all our dependencies
- We can't harden our chutney, stem, and sbws CIs, because they use pre-built binaries
Valgrind:
- We don't know if valgrind runs well in Travis CI
- Valgrind may cause CI failures due to bugs in dependencies
- Valgrind may be slower
- Valgrind instruments all the code, no matter which library it's in
- We don't know if valgrind or hardened builds provide better coverage of the kinds of coding errors we typically make
- Valgrind is simple to configure
- We can run valgrind on the pre-built binaries in our chutney, stem, and sbws CIs
We should come to a decision here and take action.