Tarballs are not compressed in a run following an aborted run
View options
Today I wondered why some of our 2018-05 tarballs are over 3 days old even though we're creating new tarballs every 2 days. And I wondered even more after finding a seemingly completed run from yesterday. Here's what happened:
- The May 19 run succeeded without issues.
- The May 21 run was interrupted, probably due to a host reboot. Apparently, it finished compressing tarballs except for bridge extra-info descriptors, but it did not move any of them in place.
- The May 23 run went through, but it did not compress any tarballs except for bridge extra-info descriptors.
I think this is related to line 102 in the script:
for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
echo `date` "Creating" ${TARBALLS[$i]}'.tar'
tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then # <- this one
echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
xz -9e ${TARBALLS[$i]}.tar
fi
doneExplanation: there were still compressed files in the working directory from May 21, and as a result we did not attempt to compress new tarballs from May 23.
Suggested fix: remove that if statement and compress the tarball regardless of whether a previous compressed tarball exists in the working directory using xz -9e -f:
diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources/create-tarballs.sh
index cd16b2dc..d247c520 100755
--- a/src/main/resources/create-tarballs.sh
+++ b/src/main/resources/create-tarballs.sh
@@ -99,10 +99,8 @@ done
for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
echo `date` "Creating" ${TARBALLS[$i]}'.tar'
tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
- if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then
- echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
- xz -9e ${TARBALLS[$i]}.tar
- fi
+ echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
+ xz -9e -f ${TARBALLS[$i]}.tar
done
cd $OUTDIR/webstats/