Tarballs are not compressed in a run following an aborted run

Today I wondered why some of our 2018-05 tarballs are over 3 days old even though we're creating new tarballs every 2 days. And I wondered even more after finding a seemingly completed run from yesterday. Here's what happened:

  • The May 19 run succeeded without issues.
  • The May 21 run was interrupted, probably due to a host reboot. Apparently, it finished compressing tarballs except for bridge extra-info descriptors, but it did not move any of them in place.
  • The May 23 run went through, but it did not compress any tarballs except for bridge extra-info descriptors.

I think this is related to line 102 in the script:

for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
  echo `date` "Creating" ${TARBALLS[$i]}'.tar'
  tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then           # <- this one
    echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
    xz -9e ${TARBALLS[$i]}.tar
  fi
done

Explanation: there were still compressed files in the working directory from May 21, and as a result we did not attempt to compress new tarballs from May 23.

Suggested fix: remove that if statement and compress the tarball regardless of whether a previous compressed tarball exists in the working directory using xz -9e -f:

diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources/create-tarballs.sh
index cd16b2dc..d247c520 100755
--- a/src/main/resources/create-tarballs.sh
+++ b/src/main/resources/create-tarballs.sh
@@ -99,10 +99,8 @@ done
 for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
   echo `date` "Creating" ${TARBALLS[$i]}'.tar'
   tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
-  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then
-    echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
-    xz -9e ${TARBALLS[$i]}.tar
-  fi
+  echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
+  xz -9e -f ${TARBALLS[$i]}.tar
 done
 
 cd $OUTDIR/webstats/
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information