Fix + Logs to pinpoint #249 bug
Based on the log messages, it seems most likely that rdsys is failing when two processes overlap. These are when: propagateUpdate
in pkg/core/backend_resources.go
tries to write to a channel and getResourceStreamHandler
has closed that channel. This only happens when these two processes occur at the same time, but due to rdsys changing in the past year or so to have distributors on other servers that are registering and unregistering channels at intervals that are not synchronized with the kraken ticker, this occurs much more often than it did in the past. This MR adds a signalling channel to backend_resources so that when a channel is closed, propagateUpdate
will not attempt to write to it. We are leaving in the logs for now to confirm that this has indeed fixed the bug.