cupani's IP address hardcoded in many places
just in terms of SSH keys, the IP address of the cupani server is hardcoded in a lot of places:
anarcat@curie:tor-puppet(master)$ cumin-all 'grep -e 78.47.38.228 -e 2a01:4f8:211:6e8:0:823:4:1 /etc/ssh/userkeys/*'
77 hosts will be targeted:
alberti.torproject.org,archive-01.torproject.org,bacula-director-01.torproject.org,build-arm-10.torproject.org,build-x86-[05-06,08-09].torproject.org,bungei.torproject.org,cache01.torproject.org,cache-02.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,check-01.torproject.org,chives.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,fsn-node-[None..None](../compare/None...None).torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-[None..None](../compare/None...None).torproject.org,henryi.torproject.org,hetzner-hel1-[None..None](../compare/None...None).torproject.org,hetzner-nbg1-[None..None](../compare/None...None).torproject.org,kvm[None..None](../compare/None...None).torproject.org,listera.torproject.org,loghost01.torproject.org,macrum.torproject.org,majus.torproject.org,mandos-01.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nutans.torproject.org,omeiense.torproject.org,onionbalance-01.torproject.org,onionoo-backend-01.torproject.org,onionoo-frontend-01.torproject.org,oo-hetzner-03.torproject.org,orestis.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rouyi.torproject.org,rude.torproject.org,scw-arm-par-01.torproject.org,static-master-fsn.torproject.org,staticiforme.torproject.org,submit-01.torproject.org,subnotabile.torproject.org,tbb-nightlies-master.torproject.org,troodi.torproject.org,unifolium.torproject.org,vineale.torproject.org,web-cymru-01.torproject.org,web-fsn-[None..None](../compare/None...None).torproject.org,web-hetzner-01.torproject.org
Confirm to continue [y/n]? y
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) staticiforme.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/torhelp:command="/srv/help-master.torproject.org/bin/update",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-user-rc,from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) vineale.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/gitweb:command="/srv/gitweb.torproject.org/bin/gitweb-ssh-wrap",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) troodi.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/tracweb:command="/srv/trac.torproject.org/bin/trigger-from-githost",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-user-rc,from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) rouyi.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/jenkins:command="/srv/jenkins.torproject.org/bin/update",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-user-rc,from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) nevii.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/dnsadm:command="/srv/dns.torproject.org/bin/from-git-rw",from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
/etc/ssh/userkeys/letsencrypt:command="/srv/letsencrypt.torproject.org/bin/from-githost",from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) gitlab-01.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
grep: /etc/ssh/userkeys/dip-git: No such file or directory |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) hetzner-hel1-01.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/nagiosadm:command="/home/nagiosadm/bin/from-git-rw",from="78.47.38.228,2a01:4f8:211:6e8:0:823:4:1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAw9/GOnsO3yhv8RRbIMTUEbFY3og+bzbz4vxaehMxddkVUkJDyDJZnwJRFr/1+q1kkasZoaYN19gfs8XaJnh9cSRNOAWquDqKjpu4z27aycQ3Pc3CtgJROvqTbgIdrq9UoWiFmXPJMCPepCe3T7y5kOmD5spSHTyVunT4fYNwGhILLkHPaDKXYRUIB4MrChuBW/tpzALTbRyWsXa9Ec3DBqSVrlKUXL4cZisa79mqJPmcfRphJFUFLwci1tifNlwOXtTTUqQTGLmt8GXJUEyU/6/QWDvkuq6B/paFnIdAAeH3mdOpXwbY7f+Y5uKl9+c7+i+9BxlJe6XB6S3H18Lbbw== git@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
===== NODE GROUP ===== |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
(1) alberti.torproject.org |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
----- OUTPUT of 'grep -e 78.47.38...c/ssh/userkeys/*' ----- |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
/etc/ssh/userkeys/sshdist:command="flock -s /var/cache/userdir-ldap/hosts//ud-generate.lock -c 'rsync --server --sender -pr . /var/cache/userdir-ldap/hosts/cupani.torproject.org'",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,from="2a01:4f8:211:6e8:0:823:4:1,78.47.38.228" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqKk7DdcughgnjqwLCQBtd5vJueu0xPXONvYFfMAWJYvSLylV7CEAqkCmDN1PUXffH76PGG+X9LrTtQGtG9WrV6Y1lGyYMkR82fkYeXPL3nLdLE+IvSkxKUg3r4qgQ/CsaFKmz8DpfdOqipnKwamncZVemplUDxaC750hCJhacGFtGaM5TbEG+B6Ykx5PXlFPjXJQ8i0tNdwhIq5nfxrUizJzWioTA8LSJ8zb+VrC9/8HaaRnOEIugDC1DJth6pjODmAO+M2aQjbpzBu0CtegIUcW/T76Tt+X3GBFV4uYR+YNA7VKaoI/xxqWku85Tx9G/6E6FUOMhD8QxdIuc968T root@cupani
/etc/ssh/userkeys/sshdist:command="flock -s /var/cache/userdir-ldap/hosts//ud-generate.lock -c 'rsync --server --sender -pr . /var/cache/userdir-ldap/hosts/cupani.torproject.org'",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,from="2a01:4f8:211:6e8:0:823:4:1,78.47.38.228" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIVn+MFJptnxYAGSBSmD06c8Aj2h0zSdde+HK7wHN3Rq root@cupani
|█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
================ PASS |███ | 9% (7/77) [00:58<05:39, 4.85s/hosts]
FAIL |█████████████████████████████ | 91% (70/77) [00:58<00:06, 1.12hosts/s]
90.9% (70/77) of nodes failed to execute command 'grep -e 78.47.38...c/ssh/userkeys/*': archive-01.torproject.org,bacula-director-01.torproject.org,build-arm-10.torproject.org,build-x86-[05-06,08-09].torproject.org,bungei.torproject.org,cache01.torproject.org,cache-02.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,check-01.torproject.org,chives.torproject.org,chiwui.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,eugeni.torproject.org,fallax.torproject.org,forrestii.torproject.org,fsn-node-[None..None](../compare/None...None).torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-[None..None](../compare/None...None).torproject.org,henryi.torproject.org,hetzner-hel1-[None..None](../compare/None...None).torproject.org,hetzner-nbg1-[None..None](../compare/None...None).torproject.org,kvm[None..None](../compare/None...None).torproject.org,listera.torproject.org,loghost01.torproject.org,macrum.torproject.org,majus.torproject.org,mandos-01.torproject.org,materculae.torproject.org,meronense.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nutans.torproject.org,omeiense.torproject.org,onionbalance-01.torproject.org,onionoo-backend-01.torproject.org,onionoo-frontend-01.torproject.org,oo-hetzner-03.torproject.org,orestis.torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rude.torproject.org,scw-arm-par-01.torproject.org,static-master-fsn.torproject.org,submit-01.torproject.org,subnotabile.torproject.org,tbb-nightlies-master.torproject.org,unifolium.torproject.org,web-cymru-01.torproject.org,web-fsn-[None..None](../compare/None...None).torproject.org,web-hetzner-01.torproject.org
9.1% (7/77) success ratio (>= 0.0% threshold) for command: 'grep -e 78.47.38...c/ssh/userkeys/*'.: alberti.torproject.org,hetzner-hel1-01.torproject.org,nevii.torproject.org,rouyi.torproject.org,staticiforme.torproject.org,troodi.torproject.org,vineale.torproject.org
9.1% (7/77) success ratio (>= 0.0% threshold) of nodes successfully executed all commands.: alberti.torproject.org,hetzner-hel1-01.torproject.org,nevii.torproject.org,rouyi.torproject.org,staticiforme.torproject.org,troodi.torproject.org,vineale.torproject.org
those keys should be deployed by Puppet instead. for now they have been renumbered by hand as part of #33446 (moved) but it would be important to change those if we ever want to rebuild that service on another host.