Replace create-tarball.sh shell script with Java module

This script's should be transferred to java.

The new createtars module should:

• provide at least the functionality of the script
• be configurable as other CollecTor modules
• not impede other modules

Please collect more features and functionality that the script can't/doesn't provide, but which should be part of this module in the comments below.

changed milestone to %CollecTor 2.0.0 in legacy/trac

added component::metrics/collector in Legacy / Trac metrics-2018 in Legacy / Trac milestone::CollecTor 2.0.0 in Legacy / Trac owner::metrics-team in Legacy / Trac parent::20518 in Legacy / Trac priority::medium in Legacy / Trac severity::normal in Legacy / Trac status::assigned in Legacy / Trac type::enhancement in Legacy / Trac labels

Trac:
Parent: N/A to legacy/trac#20518 (moved)

This module should also remove files from 'out' once they are archived.

Trac:
Description: This script's should be transferred to java.

The new createtars module should:

• provide at least the functinality of the script
• be configurable as other CollecTor modules
• not impede other modules

Please collect more features and functionality that the script can't/doesn't provide, but which should be part of this module in the comments below.

to

This script's should be transferred to java.

The new createtars module should:

• provide at least the functionality of the script
• be configurable as other CollecTor modules
• not impede other modules

Please collect more features and functionality that the script can't/doesn't provide, but which should be part of this module in the comments below.
Milestone: N/A to CollecTor 2.0.0

Capitalize some words in the summary.

Trac:
Summary: replace create-tarball.sh shell script with java module to Replace create-tarball.sh shell script with Java module

Trac:
Keywords: N/A deleted, metrics-2018 added

Trac:
Owner: N/A to metrics-team
Status: new to assigned

Trac:
Reviewer: N/A to iwakeh

Trac:
Reviewer: iwakeh to N/A
Status: assigned to accepted
Owner: metrics-team to iwakeh

Adding metrics-team to cc

Trac:
Cc: N/A to metrics-team

Trac:
Status: accepted to assigned
Owner: iwakeh to metrics-team

This create-tar code should call index-creation whenever reasonable (see patch in ticket legacy/trac#20351 (moved)).

The implementation should take the situation of legacy/trac#26193 (moved) into account and provide a solution, too.

Looks like we had two tickets for this issue. Here's what I wrote in legacy/trac#31866 (moved) which I'm now closing as duplicate:

We're currently generating tarballs using the tar and xz command-line tools triggered by a cronjob. While this is very fast, it doesn't integrate that well with the rest of our code. For example, it would be much easier to extract descriptor types, publication times, and file digests for legacy/trac#31204 (moved) if tarball generation happened in Java.

One possible issue might be that generated tarballs are larger or that compression takes longer. This is something I wanted to figure out early, which is why I ran some tests today:

|= Compression preset level =||= xz =||= XZ for Java 1.6 =| |----------------------------||--------||-------------------| | 1 | 269M | 1m27.522s | 269M | 1m22.905s | | 3 | 77M | 1m6.590s | 77M | 1m15.03s | | 6 | 30M | 3m8.426s | 30M | 4m54.837s | | 9 | 18M | 2m56.801s | 18M | 5m6.998s | | 9e | 16M | 7m2.364s || NA |

We're currently using xz -9e, but I can't find this option in XZ for Java. The closest is compression preset level 9. That means that our tarballs would be 18M/16M = 12.5% larger and be created in 306.998s/422.364s = 73% of the current time.

Is it a blocker that our tarballs would be 12.5% larger? If so, we might try harder to configure XZ for Java in the same way as xz -9e operates, even though that would very likely increase tarball generation time.

Not working on this at the moment, just leaving my thoughts here for discussion and for picking this up as time permits.

mentioned in issue legacy/trac#20351 (moved)

mentioned in issue legacy/trac#31866 (moved)

moved from legacy/trac#20350 (moved)

added Feature label and removed 1 deleted label

removed 1 deleted label

This is not a small change, but it's worth doing in the long term. Adding to backlog.

added Backlog label

added Roadmap::Future label and removed Backlog label

added Icebox label and removed Roadmap::Future label