fsn-node-07 SSH misconfiguration
The Nagios Ganeti cluster check fails with this mysterious failure:
CHECK_NRPE: Receive underflow - only -1 bytes received (4 expected).
... but when i run the verify command on the master, the problem is clearer:
root@fsn-node-01:~# gnt-cluster verify
Submitted jobs 103518, 103519
Waiting for job 103518 ...
Mon Jul 6 14:19:45 2020 * Verifying cluster config
Mon Jul 6 14:19:45 2020 * Verifying cluster certificate files
Mon Jul 6 14:19:45 2020 * Verifying hypervisor parameters
Mon Jul 6 14:19:45 2020 * Verifying all nodes belong to an existing group
Waiting for job 103519 ...
Mon Jul 6 14:19:45 2020 * Verifying group 'default'
Mon Jul 6 14:19:45 2020 * Gathering data (7 nodes)
Mon Jul 6 14:19:45 2020 * Gathering information about nodes (7 nodes)
Mon Jul 6 14:19:49 2020 * Gathering disk information (7 nodes)
Mon Jul 6 14:19:50 2020 * Verifying configuration file consistency
Mon Jul 6 14:19:50 2020 * Verifying node status
Mon Jul 6 14:19:50 2020 - ERROR: node fsn-node-03.torproject.org: ssh communication with node 'fsn-node-07.torproject.org': ssh problem: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\'r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\'r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\'r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\'r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\'r\nIt is also possible that a host key has just been changed.\'r\nThe fingerprint for the RSA key sent by the remote host is\nSHA256:F4j2N2fSMYDYxjGf/Clj8W5moFZNhdx4mgz5zXFTPi0.\'r\nPlease contact your system administrator.\'r\nUpdate the SSHFP RR in DNS with the new host key to get rid of this message.\'r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\'r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\'r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\'r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\'r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\'r\nIt is also possible that a host key has just been changed.\'r\nThe fingerprint for the RSA key sent by the remote host is\nSHA256:F4j2N2fSMYDYxjGf/Clj8W5moFZNhdx4mgz5zXFTPi0.\'r\nPlease contact your system administrator.\'r\nAdd correct host key in /dev/null to get rid of this message.\'r\nOffending RSA key in /var/lib/ganeti/known_hosts:1\'r\n remove with:\'r\n ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "fsngnt.torproject.org"\'r\nRSA host key for fsngnt.torproject.org has changed and you have requested strict checking.\'r\nHost key verification failed.\'r\n
[...]
it seems the SSH configuration is hosed and the master can't talk with the new node.