Remove old descriptor files from out/ after archiving

Unless I'm mistaken (or misconfigured) -- which is entirely possible -- collector will accumulate uncompressed data in out/ indefinitely, long after it's been archived in archive/ and will no longer be modified.

This takes up a lot of disk space and it'd be nice to

a) get confirmation I can remove data from out/ than is older than N months (2? 3?) b) have it deleted automagically (or at least with a config setting)

added component::metrics/collector in Legacy / Trac metrics-2018 in Legacy / Trac owner::metrics-team in Legacy / Trac parent::20518 in Legacy / Trac priority::medium in Legacy / Trac severity::normal in Legacy / Trac status::assigned in Legacy / Trac type::enhancement in Legacy / Trac labels

You're right. It's safe to delete descriptors that are at least N = 2 months old. In fact, on the main CollecTor instance I usually delete descriptors that are at least N = 1.5 months old. But I do this manually, because of https://xkcd.com/1205/ and https://xkcd.com/1319/, and more importantly because I'm afraid of messing up with scripts and accidentally deleting all descriptors.

I admit that these reasons don't fully apply anymore with three CollecTor instances running. Would you submit a patch to delete descriptors older than N = 2 months, possibly even with a configurable N?

https://gitweb.torproject.org/collector.git/tree/src/main/resources/create-tarballs.sh

And if you have ideas for making that script even less copy-and-pasty, please feel free to tweak it!

Replying to karsten:

You're right. It's safe to delete descriptors that are at least N = 2 months old. In fact, on the main CollecTor instance I usually delete descriptors that are at least N = 1.5 months old. But I do this manually, because of https://xkcd.com/1205/ and https://xkcd.com/1319/, and more importantly because I'm afraid of messing up with scripts and accidentally deleting all descriptors.

I think you accidentally pasted the wrong link :-) The description is here.

I admit that these reasons don't fully apply anymore with three CollecTor instances running. Would you submit a patch to delete descriptors older than N = 2 months, possibly even with a configurable N?

https://gitweb.torproject.org/collector.git/tree/src/main/resources/create-tarballs.sh

And if you have ideas for making that script even less copy-and-pasty, please feel free to tweak it!

One of the planned improvements is to integrate all scripted maintenance into java and get rid of scripting. We're planning and working on these steps already, see legacy/trac#20518 (moved), legacy/trac#20546 (moved). So, please check the back first to avoid duplicate work. Setting parent to legacy/trac#20546 (moved).

Trac:
Parent: N/A to legacy/trac#20546 (moved)
Owner: metrics-team to iwakeh
Status: new to accepted

legacy/trac#20518 (moved) is the correct parent number. (copy and paste is hard sometimes ;-)

Trac:
Parent: legacy/trac#20546 (moved) to legacy/trac#20518 (moved)

Tweak summary.

Trac:
Summary: collector should rm data from out/ after archiving to Remove old descriptor files from out/ after archiving

Trac:
Keywords: N/A deleted, metrics-2018 added

Move to metrics-team as these are not worked on by me during the next week.

Trac:
Status: accepted to assigned
Owner: iwakeh to metrics-team

moved from legacy/trac#21219 (moved)

added Feature label and removed 1 deleted label

removed 1 deleted label

assigned to @karsten

added Doing label

I'll pick this up, also to avoid future occurrences of #40001 (closed).

I have a patch that I'm currently running on corsicum a.k.a. collector2.tp.o. (Even if it breaks files in the out/ directory there, that's not a big deal, because we only care about recent descriptors on that instance anyway.)

Fixed in https://gitweb.torproject.org/collector.git/commit/?id=66ddc4d7d996ad2877aea44ea03982f14f069545. Deployed on both CollecTor hosts. Closing.

closed

mentioned in issue #28003 (closed)