Loading service/password-manager.md +71 −3 Original line number Diff line number Diff line Loading @@ -208,9 +208,9 @@ automate this than to manually perform each reset using the above. ### LUKS Next, full disk encryption keys. TODO: fabric task? Next, full disk encryption keys. Those are currently handled manually (with `pass update`) as well, but we are hoping to automate this as well, see [issue 41537](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41537) for details. ### lists Loading @@ -223,6 +223,74 @@ Mailman 3 is deployed, all those will go away anyway. Those can probably be left alone; it's unclear if they have any relevance left and should probably be removed. ### Trocla Some passwords are stored in Trocla, on the Puppet server (currently `pauli.torproject.org`). If we worry about lateral movement of an hostile attacker or a major compromise, it might be worth resetting all some of Trocla's password. This is currently not automated. In theory, deleting the entire Trocla database (its path is configured in `/etc/troclarc.yaml`) and running Puppet everywhere *should* reset all passwords, but this hides a *lot* of complexity, namely: 1. IPSec tunnels will collapse until Puppet is ran on both ends, which could break lots of things (e.g. CiviCRM, Ganeti) 2. application passwords are sometimes manually set, for example the CiviCRM IMAP and MySQL passwords are *not* managed by Puppet and would need to be reset by hand Here's a non-exhaustive list of passwords that need manual resets: * CiviCRM IMAP and MySQL * Dangerzone WebDAV * Grafana user accounts * KGB bot password (used in GitLab) * Prometheus CI password (used in GitLab's prometheus-alerts CI) * metrics DB, Tagtor, victoria metrics, weather * network health relay * probetelemetry/v2ray * rdsys frontend/backend Run `git grep trocla` in `tor-puppet.git` for the list. Note that it will match secrets that *are* correctly managed by Puppet. Automation could be built to incrementally perform those rotations, interactively. Alternatively, some password expiry mechanism could be used, especially for secrets that are managed in one Puppet run (e.g. the Dovecot mail passwords in GitLab). ### GitLab secrets In case of a full compromise, an attacker could have sucked the secrets out of GitLab projects. The `gitlab-tokens-audit.py` script in [gitlab-tools](https://gitlab.torproject.org/tpo/tpa/gitlab-tools/) provides a view of all the group and project access tokens and CI/CD variables in a set of groups or projects. Those tokens are currently rotated manually, but there could be more automation here as well: the above Python script could be improved to allow rotating tokens and resetting the associated CI/CD variable. A lot of CI/CD secret variables are SSH deploy keys, those would need coordination with the Puppet repository, maybe simply modifying the YAML files at first, but eventually those could be generated by Trocla and (why not) automatically populated in GitLab as well. ### S3 Object storage uses secrets extensively to provide access to buckets. In case of a compromise, some or all of those tokens need to be reset. The [authentication section of the object storage documentation](service/object-storage#authentication) has some more information. Basically, all access keys need to be rotated, which means expiring the existing one and creating a new one, then copying the configuration over to the right place, typically Puppet, but GitLab runners need manual configuration. The bearer token also needs to be reset for Prometheus monitoring. ## Pager playbook This service is likely not going to alert or require emergency Loading Loading
service/password-manager.md +71 −3 Original line number Diff line number Diff line Loading @@ -208,9 +208,9 @@ automate this than to manually perform each reset using the above. ### LUKS Next, full disk encryption keys. TODO: fabric task? Next, full disk encryption keys. Those are currently handled manually (with `pass update`) as well, but we are hoping to automate this as well, see [issue 41537](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41537) for details. ### lists Loading @@ -223,6 +223,74 @@ Mailman 3 is deployed, all those will go away anyway. Those can probably be left alone; it's unclear if they have any relevance left and should probably be removed. ### Trocla Some passwords are stored in Trocla, on the Puppet server (currently `pauli.torproject.org`). If we worry about lateral movement of an hostile attacker or a major compromise, it might be worth resetting all some of Trocla's password. This is currently not automated. In theory, deleting the entire Trocla database (its path is configured in `/etc/troclarc.yaml`) and running Puppet everywhere *should* reset all passwords, but this hides a *lot* of complexity, namely: 1. IPSec tunnels will collapse until Puppet is ran on both ends, which could break lots of things (e.g. CiviCRM, Ganeti) 2. application passwords are sometimes manually set, for example the CiviCRM IMAP and MySQL passwords are *not* managed by Puppet and would need to be reset by hand Here's a non-exhaustive list of passwords that need manual resets: * CiviCRM IMAP and MySQL * Dangerzone WebDAV * Grafana user accounts * KGB bot password (used in GitLab) * Prometheus CI password (used in GitLab's prometheus-alerts CI) * metrics DB, Tagtor, victoria metrics, weather * network health relay * probetelemetry/v2ray * rdsys frontend/backend Run `git grep trocla` in `tor-puppet.git` for the list. Note that it will match secrets that *are* correctly managed by Puppet. Automation could be built to incrementally perform those rotations, interactively. Alternatively, some password expiry mechanism could be used, especially for secrets that are managed in one Puppet run (e.g. the Dovecot mail passwords in GitLab). ### GitLab secrets In case of a full compromise, an attacker could have sucked the secrets out of GitLab projects. The `gitlab-tokens-audit.py` script in [gitlab-tools](https://gitlab.torproject.org/tpo/tpa/gitlab-tools/) provides a view of all the group and project access tokens and CI/CD variables in a set of groups or projects. Those tokens are currently rotated manually, but there could be more automation here as well: the above Python script could be improved to allow rotating tokens and resetting the associated CI/CD variable. A lot of CI/CD secret variables are SSH deploy keys, those would need coordination with the Puppet repository, maybe simply modifying the YAML files at first, but eventually those could be generated by Trocla and (why not) automatically populated in GitLab as well. ### S3 Object storage uses secrets extensively to provide access to buckets. In case of a compromise, some or all of those tokens need to be reset. The [authentication section of the object storage documentation](service/object-storage#authentication) has some more information. Basically, all access keys need to be rotated, which means expiring the existing one and creating a new one, then copying the configuration over to the right place, typically Puppet, but GitLab runners need manual configuration. The bearer token also needs to be reset for Prometheus monitoring. ## Pager playbook This service is likely not going to alert or require emergency Loading