keep an inventory of installed software
We should have an inventory of programs and software installed and managed by TPA (and service admins, ideally).
This was an issue in the handling of the log4j security issue (tpo/tpa/team#40551), as we didn't actually know where the software was installed. It was actually relatively easy to tell if we had the debian package installed everywhere (basically by running Cumin), but even that takes some time to run.
Ideally, we'd have a full inventory of all Debian packages, but also JARs, Python wheels, Ruby gems, and so on, installed everywhere, including dependencies, in one location, alongside version information. Container images are also probably something to take into account.
It's a hard problem, especially with our bespoke deployment systems, but one worth fixing. This should be part of our security policy (tpo/team#41).
Prior art:
- debian upgrade procedures
- upgrade automation (#31957 (closed))
- reboot automation (#33406 (closed))
- limoncelli test (#40944 (closed))