Skip to content
Snippets Groups Projects

Test running tasks in create-traballs.sh script in parallel, 4 at a time

Merged Test running tasks in create-traballs.sh script in parallel, 4 at a time
use-less-resources into master
1 unresolved thread
Merged Hiro requested to merge use-less-resources into master
1 unresolved thread

I am running at test on colchicifolium where I execute task in the create-tarballs script in parallel, 4 at a time.

I am testing how much time we can save with the overall execution and if we will use give or take the same resources (monitoring: https://grafana1.torproject.org/d/Z7T7Cfemz/node-exporter-full?orgId=1&var-job=node&var-node=colchicifolium.torproject.org&var-port=9100)

@lavamind @anarcat please let me know what you think.

Edited by Hiro

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
src/main/resources/create-tarballs.sh
114 echo `date` "Creating symlink for" ${TARBALLS[$i]}
114 echo `date` "Creating symlink for" ${TARBALLS[$i]}
115 115 ln -s ${DIRECTORIES[$i]} ${TARBALLS[$i]}
116 116 else
117 117 # This is a workaround for the "tar u" bug in GNU tar 1.20
118 echo `date` "Touching symlink and directories for" ${TARBALLS[$i]}
118 echo `date` "Touching symlink and directories for" ${TARBALLS[$i]}
119 119 find -L ${TARBALLS[$i]} -type d | xargs touch
120 120 fi
121 121 done
122 122
123 N=4
123 124 for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
124 echo `date` "Creating" ${TARBALLS[$i]}'.tar.xz'
125 tar -I "xz -9e" -c -h -f ${TARBALLS[$i]}.tar.xz ${TARBALLS[$i]}
125 ((j=j%N)); ((j++==0)) && wait
  • anarcat
    anarcat @anarcat

    i think i understand what this does, but a comment would certainly not hurt here.

    also: have you considered just rewriting this entire thing in a sane programming language as opposed to a scripting language like bash? :smile: i have this mental boundary that when i start doing arithmetic, parallel processing, or when i cross 100 lines, i rewrite my program in python (used to be perl, now it's python, but you might prefer golang or rust, whatever...)

  • Hiro
    Hiro @hiro
    Author Owner

    Yes python would certainly be better than bash. This was just a test though to see if we could at least finish this job in something like less than 24 hours. So it just runs 4 compressions at the same time.

  • Please register or sign in to reply
  • Hiro added 1 commit

    added 1 commit

    Compare with previous version

  • Hiro marked this merge request as ready

    marked this merge request as ready

  • Hiro
    Hiro @hiro
    Author Owner

    I have switched this to 2 parallel runs. If it is ok for you (CC: @anarcat @lavamind) I'll just merge it so I can track it in collector repository.

  • Hiro merged

    merged

  • Hiro mentioned in commit a285fbca

    mentioned in commit a285fbca

  • Jérôme Charaoui mentioned in issue tpo/tpa/team#40650 (closed)

    mentioned in issue tpo/tpa/team#40650 (closed)

    • Please register or sign in to reply