puppetdb is failing puppet runs since buster/bullseye point upgrade
puppet catalog runs are failing intermittently with an error like:
root@gitlab-02:~# pat
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=b88fe540833780418991630b46806cf81256b973&version=9&certname=gitlab-02.torproject.org&command=replace_catalog&producer-timestamp=2022-03-29T15:57:42.071Z' on at least 1 of the following 'server_urls': https://puppet.torproject.org:8081
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
it's fairly harmless: puppet skips the run and moves on. eventually it succeeds, which is why things are flapping like mad in nagios.
this also makes the problem pretty hard to reproduce. because of this, diagnostics are spotty. but it seems this is the error we get from java:
Mar 29 16:00:19 pauli/pauli java[6063]: WARNING: An illegal reflective access operation has occurred
Mar 29 16:00:19 pauli/pauli java[6063]: WARNING: Illegal reflective access by clojure.lang.InjectedInvoker/0x0000000840062040 (file:/usr/share/java/clojure.jar) to meth
od sun.nio.ch.ChannelInputStream.close()
Mar 29 16:00:19 pauli/pauli java[6063]: WARNING: Please consider reporting this to the maintainers of clojure.lang.InjectedInvoker/0x0000000840062040
Mar 29 16:00:19 pauli/pauli java[6063]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 29 16:00:19 pauli/pauli java[6063]: WARNING: All illegal access operations will be denied in a future release
Edited by anarcat