Skip to content

Turn frequent broker log messages into useful metrics

I took a look at the broker logs recently and noticed they are extremely noisy. Particularly with three different messages:

Feb 26 19:37:48 snowflake-broker-40349 broker[278439]: 2025/02/26 19:37:48 Client: Timed out.
Feb 26 19:37:48 snowflake-broker-40349 broker[278439]: 2025/02/26 19:37:48 Warning: matching with snowflake client failed
Feb 26 19:37:48 snowflake-broker-40349 broker[278439]: 2025/02/26 19:37:48 bad request: rejected relay pattern from proxy = bad request

These feel like the kind of events we would be interested in seeing counts of, but I don't think the log messages are useful to us at their current rate of occurrence. We can make prometheus metrics for these events pretty easily. We might also want collector metrics to have a public archive.