Skip to content

Reduce or remove use of mutexes for Snowflake broker metrics

After some profiling the Snowflake broker, we found that goroutines do sometimes block on obtaining the mutex for metrics updates. In particular, (*IPC).ProxyPolls and (*IPC).ClientOffers are called very frequently and each of these request the i.ctx.metrics.lock mutex multiple times.

  • We should be able to update our metrics more efficiently by using sync/atomic.
  • The prometheus metrics library we are using is thread safe, so many calls to functions/metrics from there don't need to surrounded by mutex access:

    All exported functions and methods are safe to be used concurrently unless specified otherwise.

  • For our own prometheus metrics, we use atomic integers, which means they are also thread safe
Edited by Cecylia Bocovich