GitLab logfile production_json.log is broken

For a while now, GitLab's production_json.log has been broken. This is makes troubleshooting errors in GitLab with production.log quite difficult, not the least because it doesn't record timestamps (!).

One will find the current log file empty, while rotated files contain some variation of this stack trace. I'm reproducing below only the first line of this beatiful specimen:

Could not log "process_action.action_controller" event. JSON::GeneratorError: "\xA3" from ASCII-8BIT to UTF-8 ["/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/json-2.11.3/lib/json/common.rb:916:in `generate'"

Of note, the unrecognized character is not always the same! Sometimes it's "\xA0", "\xA1" and so forth.

I've searched through the upstream issue tracker, to no avail.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information