Puppet error when applying ssh_authorized_key resources on apt.lizard: "Could not evaluate: Path is nil"

Symptom

While tackling #18184, i removed @groente's SSH key from hiera and manually ran puppet agent -t on apt.lizard, which resulted in errors for (all?) ssh_authorized_key resources:

root@apt:~# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for apt.lizard
Info: Applying configuration version '1740075967'
Notice: /Stage[main]/Tails::Profile::Reprepro::Snapshots::Tagged/Package[hardlink]/ensure: created (corrective)
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_lizard-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_lizard-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-boyska-reprepro-time-based-snapshots]/Ssh_authorized_key[boyska_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-boyska-reprepro-time-based-snapshots]/Ssh_authorized_key[boyska_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_backups-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_backups-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_lizard-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_lizard-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 17.73 seconds

Investigation

When i add -d, the debug log says something that is untrue:

Debug: The required user is not yet present on the system

The reprepro-time-based-snapshots does exist (and has always existed) in the system:

root@apt:~# id reprepro-time-based-snapshots
uid=998(reprepro-time-based-snapshots) gid=998(reprepro-time-based-snapshots) groups=998(reprepro-time-based-snapshots)

I added debugging lines to the ssh_authorized_key type definition:

diff --git a/lib/puppet/type/ssh_authorized_key.rb b/lib/puppet/type/ssh_authorized_key.rb
index 701dfe5..df35036 100644
--- a/lib/puppet/type/ssh_authorized_key.rb
+++ b/lib/puppet/type/ssh_authorized_key.rb
@@ -113,8 +113,10 @@ module Puppet
 
         begin
           File.expand_path("~#{resource[:user]}/.ssh/authorized_keys")
-        rescue
+        rescue => e
           Puppet.debug 'The required user is not yet present on the system'
+          Puppet.debug "Rescued: #{e.inspect}"
+          Puppet.debug "Path: ~#{resource[:user]}/.ssh/authorized_keys"
           nil
         end
       end

We can see that the actual error is #<RuntimeError: can't set length of shared string> when running the File.expand_path(...) line:

Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]/ensure: created (corrective)
Debug: The required user is not yet present on the system
Debug: Rescued: #<RuntimeError: can't set length of shared string>
Debug: Path: ~reprepro-time-based-snapshots/.ssh/authorized_keys
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]: Could not evaluate: Path is nil

Also, the path is correct and actually does exists (and has always existed):

root@apt:~# ls -alh ~reprepro-time-based-snapshots/.ssh/authorized_keys 
-rw------- 1 reprepro-time-based-snapshots reprepro-time-based-snapshots 5.7K Nov 13  2023 /srv/apt-snapshots/time-based/.ssh/authorized_keys

/cc @anarcat @groente @lavamind @lelutin, do any of you have a clue?

added Tails label

changed the description

mentioned in issue #18193 (closed)

I found that the issue only manifests itself when I execute puppet manually on a shell, and not when running it with systemd restart puppet-run. Trying to use the full line from the systemd service doesn't fix (/usr/bin/puppet agent --config /etc/puppet/puppet.conf --onetime --no-daemonize --detailed-exitcode --no-usecacheonfailure).

I took a quick look at the systemd service (systemctl show puppet-run.service) and saw no environment variables set. The only thing that comes to mind is that maybe there are capabilities set that influence how memory management in ruby works. But i won't spend more time on this now.

On another front, i found that the key is not automatically removed from the authorized_keys file, i filed #18193 (closed).

I'm sending this to the Icebox for now.

added Icebox label

do you have any way of reproducing this error? i tried manually adding a key to reprepro-time-based-snapshots's authorized_keys and that was purged without a problem...

If Puppet is run by the systemd puppet-run.service, all works fine.

The problem happens when you run run puppet agent -t manually in apt.lizard.

The issue is that, when Puppet is run manually, ssh_authorized_key fails to apply with an error.

To reproduce, just login to apt.lizard and run puppet agent -t as root.

ehrr.. no?

root@apt:~# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for apt.lizard
Info: Applying configuration version '1741901492'
Notice: /Stage[main]/Tails::Profile::Reprepro::Snapshots::Tagged/Package[hardlink]/ensure: created (corrective)
Notice: Applied catalog in 17.25 seconds
root@apt:~# 

This is intriguing!

When logging in as zen and using sudo, the error doesn't happen:

$ ssh zen@apt.lizard sudo puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for apt.lizard
Info: Applying configuration version '1743162649'
Notice: /Stage[main]/Tails::Profile::Reprepro::Snapshots::Tagged/Package[hardlink]/ensure: created (corrective)
Notice: Applied catalog in 16.48 seconds

But when logging in as root directly, the error does happen:

$ ssh root@apt.lizard puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for apt.lizard
Info: Applying configuration version '1743162789'
Notice: /Stage[main]/Tails::Profile::Reprepro::Snapshots::Tagged/Package[hardlink]/ensure: created (corrective)
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_lizard-reprepro-time-based-snapshots]/ensure: created (corrective)
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_lizard-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-anonym-reprepro-time-based-snapshots]/Ssh_authorized_key[anonym_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-boyska-reprepro-time-based-snapshots]/Ssh_authorized_key[boyska_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_backups-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-boyska-reprepro-time-based-snapshots]/Ssh_authorized_key[boyska_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_backups-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_lizard-reprepro-time-based-snapshots]/ensure: created (corrective)
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_personal-reprepro-time-based-snapshots]/ensure: created (corrective)
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_lizard-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_personal-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]/ensure: created (corrective)
Info: Stage[main]: Unscheduling all events on Stage[main]
Error: /Stage[main]/Tails::Profile::Reprepro/Rbac::Ssh[foundations-team-members-to-reprepro-time-based-snapshots]/Rbac::Sshto[foundations-team-members-intrigeri-reprepro-time-based-snapshots]/Ssh_authorized_key[intrigeri_iguana-reprepro-time-based-snapshots]: Could not evaluate: Path is nil
Notice: Applied catalog in 16.29 seconds

@groente, can you please check if you're able to reproduce it like this?

hm, nope, still can't reproduce:

groente@lizard:~$ ssh root@apt.lizard puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for apt.lizard
Info: Applying configuration version '1743166718'
Notice: /Stage[main]/Tails::Profile::Reprepro::Snapshots::Tagged/Package[hardlink]/ensure: created (corrective)
Notice: Applied catalog in 18.60 seconds
groente@lizard:~$