long term prometheus metrics

data retention on the primary prometheus has been expanded to 30 days, which is nice, but that's not enough. create another (a third, technically, but a second in this cluster) prometheus server that would scrape all metrics off the first server, but at a different sampling rate so we can keep metrics for a longer, possibly multi-year timeline.

Review the storage requirements math in legacy/trac#29388 (moved) and compare with reality.

This, obviously, is a followup to the general prometheus setup ticket in legacy/trac#29389 (moved).