A new alert related to this is being created.

refs: #42050

howto/puppet.md

@@ -1280,6 +1280,35 @@ monitoring tool.
 
  [Puppet reporter]: https://github.com/voxpupuli/puppet-prometheus_reporter
 
+### Agent running on non-production environment for too long
+
+When we're working on changes that we want to test on a limited number of hosts,
+we can change the environment that the puppet agent is using. We usually do this
+for short periods of time and it is highly desirable to move the host back to
+the `production` environment once our tests are done.
+
+In some circumstances, it's possible that a host was forgotten on an environment
+other than production. This has the undesirable effect that that host might miss
+out on important changes like access revocation, policy changes and the like.
+
+If a host has been left away from production for too long, first check out which
+environment it is running on:
+
+    # grep environment /etc/puppet/puppet.conf 
+    environment = alertmanager_template_tests
+
+Check out with TPA members to see if someone is currently actively working on
+that branch and if the host should still be left on that environment. If so,
+create a silence for the alert, but for a maximum of 2 weeks at a time.
+
+If the host is not supposed to stay away from production, then check out if
+bringing it back will cause any undesirable changes:
+
+    patn --environment production
+
+If all seems well, run the same command as above but with `pat` instead of
+`patn`.
+
 ### Problems pushing to the Puppet server
 
 If you get this error when pushing commits to the Puppet server: