reconsider xz -9

hi!

i wonder if we could reconsider the use of xz -9 in the archival mechanisms here:

https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/blob/fcedbefd546de822d8f9cfe6cd22e3769708a7b0/src/main/resources/create-tarballs.sh#L125

there we do:

   tar -I "xz -9e" -c -h -f ${TARBALLS[$i]}.tar.xz ${TARBALLS[$i]}

the -9 there means that the compressor might take up to 600MB of memory to run, if i read the manpage right...

is there space in the pipeline to consider changing this? @lavamind suggested maybe using zstd, but could we also consider a lower compression ratio?

maybe some experiments with a few tarballs could help clarifying this? 😄

this is part of the improvements we're looking at for the collector pipeline ( tpo/tpa/team#40650 (closed))