Mainly Tech projects on Python and Electronic Design Automation.

Saturday, March 21, 2009

Batch Process Runner in bash shell - Forgotten Enhancements

A comment on an href="http://paddy3118.blogspot.com/2007/12/batch-process-runner-in-bash-shell.html">earlier
post caused me to dig out this enhancement to my original
batch process runner.



What you do is create a file of one-liner commands that you would enter
at a command prompt, for example, this is file style="font-weight: bold;">process_list.txt :


1  color="#0000ff"># (Comments have '#' at the left margin)
color="#804040">2 sleep 4; echo slept for 4 at line 2; csdf_sdf_sd; exit 19
color="#804040">3 sleep 1; cause-an-error; echo slept for 1 at line 3
color="#804040">4 sleep 3; echo color="#ff00ff">'slept for 3 at line 4'
color="#804040">5 sleep 7; echo color="#ff00ff">"slept for 7 at line 5"
color="#804040">6 # Whee!
color="#804040">7 sleep 9; another-erro; echo slept for 9 at line 7
color="#804040">8 sleep 5; echo color="#ff00ff">'slept "for 5" at line 8'


A bit of warning, bash returns the exit code of this laast command it
executes, earlier exit codes won't be seen.



Give my script the number of processes to run in parallel, N, followed
by the above file, and it will create N '.job'  files out of
the non-comment lines of process_list.txt and execute them as
background jobs, with their output sent to '.n' files.:


bash$ ./process_list_runner.sh 2 process_list.txt

## STARTING 6 Processes from file: process_list.txt, 2 at a time with id plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP

# 2 Jobs in background. 6/6 started. 2009-03-21-05:36:36

## FINISHED, (stats in plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.csv)

bash$ style="font-weight: bold;">ls -1 plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.*
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.1
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.2
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.3
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.4
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.5
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.6
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.csv
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.job_0
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.job_1
bash$




The script does what it can to create a csv file of run stats, which,
with a bit of tidying up in calc produces the following:




frame="void" rules="none">
width="340"> width="110"> width="115"> width="73">























































































































width="1409">##
STARTING 6 Processes from file: process_list.txt
size="4">2 at a time with id
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP
























align="center" bgcolor="#ccffff" height="123"
valign="middle">JOB
align="right" bgcolor="#ccffff" valign="middle">LINE align="center" bgcolor="#ccffff" valign="middle">Command
being timed
align="center" bgcolor="#ccffff" valign="middle">User
time (seconds)
align="center" bgcolor="#ccffff" valign="middle">System
time (seconds)
align="center" bgcolor="#ccffff" valign="middle">Percent
of CPU this job got
align="center" bgcolor="#ccffff" valign="middle">Elapsed
(wall clock) time (h:mm:ss or m:ss)
align="center" bgcolor="#ccffff" valign="middle">Maximum
resident set size (kbytes)
align="center" bgcolor="#ccffff" valign="middle">Major
(requiring I/O) page faults
align="center" bgcolor="#ccffff" valign="middle">Signals
delivered
align="center" bgcolor="#ccffff" valign="middle">Page
size (bytes)
align="center" bgcolor="#ccffff" valign="middle">Exit
status
sdnum="2057;" align="center" bgcolor="#ffffcc"
height="18">1
sdnum="2057;" align="center" bgcolor="#ffffcc">2 bgcolor="#ffffcc">sleep 4; echo slept for 4 at line 2;
csdf_sdf_sd; exit 19
sdnum="2057;" align="center" bgcolor="#ffffcc">0.13 sdnum="2057;" align="center" bgcolor="#ffffcc">0.15 sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">6.00% sdval="0.0000503472222222222" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ffffcc">00:04.35
sdval="760832" sdnum="2057;" align="center"
bgcolor="#ffffcc">760832
sdnum="2057;" align="center" bgcolor="#ffffcc">3277 sdnum="2057;" align="center" bgcolor="#ffffcc">6 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ffffcc">65536
sdnum="2057;" align="center" bgcolor="#ffffcc">19
sdnum="2057;" align="center" bgcolor="#ccccff"
height="17">2
sdnum="2057;" align="center" bgcolor="#ccccff">3 bgcolor="#ccccff">sleep 1; cause-an-error; echo slept for 1
at line 3
sdnum="2057;" align="center" bgcolor="#ccccff">0.12 sdnum="2057;" align="center" bgcolor="#ccccff">0.09 sdnum="2057;0;0.00%" align="center" bgcolor="#ccccff">16.00% sdval="0.0000152777777777778" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ccccff">00:01.32
sdval="761088" sdnum="2057;" align="center"
bgcolor="#ccccff">761088
sdnum="2057;" align="center" bgcolor="#ccccff">3273 sdnum="2057;" align="center" bgcolor="#ccccff">6 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ccccff">65536
sdnum="2057;" align="center" bgcolor="#ccccff">0
sdnum="2057;" align="center" bgcolor="#ffffcc"
height="17">3
sdnum="2057;" align="center" bgcolor="#ffffcc">4 bgcolor="#ffffcc">sleep 3; echo 'slept for 3 at line 4' sdnum="2057;" align="center" bgcolor="#ffffcc">0.07 sdnum="2057;" align="center" bgcolor="#ffffcc">0.06 sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">4.00% sdval="0.0000377314814814815" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ffffcc">00:03.26
sdval="617216" sdnum="2057;" align="center"
bgcolor="#ffffcc">617216
sdnum="2057;" align="center" bgcolor="#ffffcc">2639 sdnum="2057;" align="center" bgcolor="#ffffcc">3 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ffffcc">65536
sdnum="2057;" align="center" bgcolor="#ffffcc">0
sdnum="2057;" align="center" bgcolor="#ccccff"
height="17">4
sdnum="2057;" align="center" bgcolor="#ccccff">5 bgcolor="#ccccff">sleep 7; echo "slept for 7 at line 5" sdnum="2057;" align="center" bgcolor="#ccccff">0.07 sdnum="2057;" align="center" bgcolor="#ccccff">0.07 sdnum="2057;0;0.00%" align="center" bgcolor="#ccccff">2.00% sdval="0.000084837962962963" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ccccff">00:07.33
sdval="617216" sdnum="2057;" align="center"
bgcolor="#ccccff">617216
sdnum="2057;" align="center" bgcolor="#ccccff">2644 sdnum="2057;" align="center" bgcolor="#ccccff">3 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ccccff">65536
sdnum="2057;" align="center" bgcolor="#ccccff">0
sdnum="2057;" align="center" bgcolor="#ffffcc"
height="18">5
sdnum="2057;" align="center" bgcolor="#ffffcc">7 bgcolor="#ffffcc">sleep 9; another-erro; echo slept for 9 at
line 7
sdnum="2057;" align="center" bgcolor="#ffffcc">0.09 sdnum="2057;" align="center" bgcolor="#ffffcc">0.13 sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">2.00% sdval="0.000106597222222222" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ffffcc">00:09.21
sdval="760064" sdnum="2057;" align="center"
bgcolor="#ffffcc">760064
sdnum="2057;" align="center" bgcolor="#ffffcc">3274 sdnum="2057;" align="center" bgcolor="#ffffcc">6 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ffffcc">65536
sdnum="2057;" align="center" bgcolor="#ffffcc">0
sdnum="2057;" align="center" bgcolor="#ccccff"
height="17">6
sdnum="2057;" align="center" bgcolor="#ccccff">8 bgcolor="#ccccff">sleep 5; echo 'slept "for 5" at line 8' sdnum="2057;" align="center" bgcolor="#ccccff">0.1 sdnum="2057;" align="center" bgcolor="#ccccff">0.06 sdnum="2057;" align="center" bgcolor="#ccccff">0.03 sdval="0.0000622685185185185" sdnum="2057;0;MM:SS.00"
align="center" bgcolor="#ccccff">00:05.38
sdval="617216" sdnum="2057;" align="center"
bgcolor="#ccccff">617216
sdnum="2057;" align="center" bgcolor="#ccccff">2639 sdnum="2057;" align="center" bgcolor="#ccccff">3 sdval="65536" sdnum="2057;" align="center"
bgcolor="#ccccff">65536
sdnum="2057;" align="center" bgcolor="#ccccff">0




The script itself is:


 1  color="#0000ff">#!/bin/bash 
color="#804040"> 2 #!/opt/TWWfsw/bin/bash -
color="#804040"> 3
color="#804040"> 4 set color="#008080">-u
color="#804040"> 5
color="#804040"> 6 ##
color="#804040"> 7 ## process_runner.sh <concurrent> <total procs>
color="#804040"> 8 ##
color="#804040"> 9 ## Example script given the maximum number of processes to
color="#804040"> 10 ## run concurrently, c, and the total number of processes, n
color="#804040"> 11 ## runs at most c, processes in the background until all n
color="#804040"> 12 ## Have been run.
color="#804040"> 13 ##
color="#804040"> 14 ## Author Donald 'Paddy' McCarthy Dec. 17 2007
color="#804040"> 15 ##
color="#804040"> 16
color="#804040"> 17 # how many processes to run in parallel
color="#804040"> 18 concurrent= color="#a020f0">$1
color="#804040"> 19 # File of commands
color="#804040"> 20 proclist= color="#804040">" color="#a020f0">$2"
color="#804040"> 21
color="#804040"> 22 ##
color="#804040"> 23 ##
color="#804040"> 24
color="#804040"> 25 # main loop wait time between checking background procs.
color="#804040"> 26 tick= color="#ff00ff">1
color="#804040"> 27 # Unique_id for process files of this run
color="#804040"> 28 unique_id=plr_ color="#6a5acd">`date +${ color="#a020f0">USER// color="#804040">/ color="#a020f0">}_%F-%T_ color="#a020f0">${HOSTNAME color="#804040">// color="#804040">/ color="#a020f0">}`
color="#804040"> 29 unique_id= color="#a020f0">${unique_id color="#804040">//: color="#804040">/_ color="#a020f0">}
color="#804040"> 30
color="#804040"> 31 function read_proclistfile color="#6a5acd">{
color="#804040"> 32 # read processes from file ignoring comment lines
color="#804040"> 33 maxprocs= color="#ff00ff">0
color="#804040"> 34 local -i color="#008080">linenumber= color="#ff00ff">0
color="#804040"> 35 while color="#804040"> color="#804040">read color="#804040">; color="#804040"> color="#804040">do
color="#804040"> 36 (( color="#008080">linenumber+= color="#ff00ff">1))
color="#804040"> 37 # echo [ "${REPLY:0:1}" == "#" ] $maxprocs $REPLY
color="#804040"> 38 if color="#804040">[ color="#804040">" color="#a020f0">${REPLY color="#804040">: color="#ff00ff">0: color="#ff00ff">1} color="#804040">" color="#804040">!= color="#804040">" color="#ff00ff">#" color="#804040">] color="#804040">; color="#804040">then
color="#804040"> 39 ((maxprocs+ color="#804040">= color="#ff00ff">1))
color="#804040"> 40 allprocs color="#804040">[ color="#a020f0">$maxprocs color="#804040">] color="#804040">= color="#804040">" color="#a020f0">$REPLY"
color="#804040"> 41 allprocline color="#804040">[ color="#a020f0">$maxprocs color="#804040">] color="#804040">= color="#a020f0">$linenumber
color="#804040"> 42 # echo $linenumber $maxprocs $REPLY
color="#804040"> 43 fi
color="#804040"> 44 done color="#804040">< color="#a020f0">$proclist
color="#804040"> 45 }
color="#804040"> 46
color="#804040"> 47 read_proclistfile
color="#804040"> 48
color="#804040"> 49 printf color="#804040">" color="#6a5acd">\n## STARTING %i Processes from file: %s, %i at a time with id %s color="#6a5acd">\n\n" \
color="#804040"> 50 $maxprocs color="#a020f0">$proclist color="#a020f0">$concurrent color="#a020f0">$unique_id
color="#804040"> 51
color="#804040"> 52
color="#804040"> 53
color="#804040"> 54 function assemble_job color="#6a5acd">{
color="#804040"> 55 #
color="#804040"> 56 local color="#008080">plr_unique_id= color="#804040">" color="#a020f0">$1"; color="#804040">local color="#008080">plr_ran=" color="#a020f0">$2"; color="#804040">local color="#008080">plr_proc= color="#804040">" color="#a020f0">$3"; color="#804040">local color="#008080">plr_procline= color="#804040">" color="#a020f0">$4"; color="#804040">local color="#008080">plr_proclist= color="#804040">" color="#a020f0">$5"
color="#804040"> 57 cat color="#804040"><<!
color="#804040"> 58
color="#804040"> 59 # color="#a020f0">$plr_unique_id color="#ff00ff"> $plr_ran color="#ff00ff">
color="#804040"> 60 ` color="#804040">local color="#6a5acd">`
color="#804040"> 61
color="#804040"> 62 trap 'error= color="#6a5acd">\$?;printf " color="#6a5acd">\n\n##STATISTICS For Job %i color="#6a5acd">\n" color="#a020f0">$plr_ran; printf "# Line %i of file %s color="#6a5acd">\n\n" color="#a020f0">$plr_procline color="#ff00ff"> $plr_proclist color="#ff00ff">; exit \$ color="#ff00ff">error' EXIT
color="#804040"> 63
color="#804040"> 64
color="#804040"> 65 $plr_proc
color="#804040"> 66
color="#804040"> 67 !
color="#804040"> 68 }
color="#804040"> 69
color="#804040"> 70 function print_runstats color="#6a5acd">{
color="#804040"> 71 printf color="#804040">' color="#ff00ff"># %i Jobs in background. %i/%i started. %s\r color="#804040">' \
color="#804040"> 72 ` color="#804040">jobs color="#6a5acd"> -r color="#804040">| color="#6a5acd">wc -l` $ran color="#a020f0">$maxprocs color="#804040">" color="#6a5acd">`date +%F-%T` color="#804040">"
color="#804040"> 73 }
color="#804040"> 74
color="#804040"> 75 # Bash array running keeps track of the background process numbers
color="#804040"> 76 # Start with nothing running (sentinel value will not be a process number
color="#804040"> 77 for color="#6a5acd">((i= color="#ff00ff">0; i color="#804040">< color="#a020f0">$concurrent color="#804040">; i+ color="#804040">= color="#ff00ff">1 )) color="#804040">; color="#804040">do running color="#804040">[ color="#a020f0">$i] color="#804040">= color="#ff00ff">123456789 color="#804040">; color="#804040">done
color="#804040"> 78
color="#804040"> 79 ran= color="#ff00ff">0
color="#804040"> 80 until
color="#804040"> 81 while color="#804040">[ color="#a020f0">$ran -lt color="#a020f0">$maxprocs color="#804040">] color="#804040">; color="#804040">do
color="#804040"> 82 for color="#6a5acd">((p= color="#ff00ff">0; p color="#804040">< color="#a020f0">$concurrent color="#804040">; p+ color="#804040">= color="#ff00ff">1 )) color="#804040">; color="#804040">do
color="#804040"> 83 proc= color="#a020f0">${running color="#a020f0">[$p color="#a020f0">]}
color="#804040"> 84 # Over all running processes...
color="#804040"> 85 # $proc still running?
color="#804040"> 86 ps -p color="#a020f0">$proc | color="#804040">fgrep color="#a020f0">$proc >/dev/null
color="#804040"> 87 if color="#804040">[ color="#a020f0">$? -ne color="#804040">' color="#ff00ff">0' color="#804040">] color="#804040">; color="#804040">then
color="#804040"> 88 # Not found i.e. its finished
color="#804040"> 89 # So start the next
color="#804040"> 90 ((ran+ color="#804040">= color="#ff00ff">1))
color="#804040"> 91 #(assemble_job ) >$unique_id.job_$p
color="#804040"> 92 ( assemble_job color="#804040">" color="#a020f0">$unique_id color="#804040">" color="#804040">" color="#a020f0">$ran" color="#804040">" color="#a020f0">${allprocs color="#a020f0">[$ran color="#a020f0">]} color="#804040">" \
color="#804040"> 93 ${ color="#a020f0">allprocline color="#a020f0">[$ran color="#a020f0">]} color="#a020f0">$proclist color="#804040">) color="#804040">> color="#a020f0">$unique_id.job_ color="#a020f0">$p
color="#804040"> 94 chmod +x color="#a020f0">$unique_id.job_ color="#a020f0">$p
color="#804040"> 95 # run one job in background collecting run stats
color="#804040"> 96 (/bin/ color="#804040">time -v -a -o color="#a020f0">$unique_id. color="#a020f0">$ran < /dev/null color="#a020f0">$unique_id.job_ color="#a020f0">$p 2>&1 color="#804040">) color="#804040">> color="#a020f0">$unique_id. color="#a020f0">$ran &
color="#804040"> 97 running color="#804040">[ color="#a020f0">$p] color="#804040">= color="#a020f0">$!
color="#804040"> 98 runningprocnum color="#804040">[ color="#a020f0">$p] color="#804040">= color="#a020f0">$ran
color="#804040"> 99 if color="#804040">[ color="#a020f0">$ran -ge color="#a020f0">$maxprocs color="#804040">] color="#804040">; color="#804040">then color="#804040">break color="#ff00ff">1; color="#804040">fi
color="#804040">100 #exit
color="#804040">101 fi
color="#804040">102 done
color="#804040">103 sleep color="#a020f0">$tick
color="#804040">104 # Status
color="#804040">105 print_runstats
color="#804040">106 done
color="#804040">107
color="#804040">108 ## break 1 gets us here!
color="#804040">109
color="#804040">110 # Keep on printing status while there are background processes
color="#804040">111 # even though there are no more to start.
color="#804040">112 sleep color="#a020f0">$tick
color="#804040">113 # Status
color="#804040">114 print_runstats
color="#804040">115 do color="#804040">[ color="#6a5acd">`jobs color="#6a5acd"> -r color="#804040">| color="#6a5acd">wc -l` -eq color="#ff00ff">0 ]
color="#804040">116 done
color="#804040">117 wait
color="#804040">118
color="#804040">119 function stats2csv color="#6a5acd">{
color="#804040">120 # Gather overall stats in csv file
color="#804040">121
color="#804040">122 # CSV Heading
color="#804040">123 printf color="#804040">" color="#6a5acd">\n## STARTING %i Processes from file: %s color="#6a5acd">\n%i at a time with id %s color="#6a5acd">\n\n" \
color="#804040">124 $maxprocs color="#a020f0">$proclist color="#a020f0">$concurrent color="#a020f0">$unique_id color="#804040">> color="#a020f0">$unique_id.csv
color="#804040">125 # CSV Column labels
color="#804040">126 gawk color="#804040">' color="#ff00ff">/^##STATISTICS For Job/{s++}
color="#804040">127 s&&/^# Line .*of file/{printf "JOB,LINE";next}
color="#804040">128 s&&/: /{l=$0;sub(/: .*/,"");$1=$1;printf",%s",$0}
color="#804040">129 s&&l~/Exit status:/{s=0;print"";l="";exit}
color="#804040">130 color="#804040">' color="#a020f0">$unique_id. color="#804040">[ color="#ff00ff">0-9]* color="#804040">>> color="#a020f0">$unique_id.csv
color="#804040">131
color="#804040">132 # Stick allprocs into env for later gawk substitution of command
color="#804040">133 for i color="#804040">in color="#a020f0">${!allprocs color="#a020f0">[@] color="#a020f0">}; color="#804040">do
color="#804040">134 declare color="#008080">-x _ap color="#a020f0">$i= color="#804040">" color="#a020f0">${allprocs color="#a020f0">[$i color="#a020f0">]} color="#804040">"
color="#804040">135 done
color="#804040">136 #env|grep '_ap.=' ; debug
color="#804040">137 # CSV Rows
color="#804040">138 gawk color="#804040">' color="#ff00ff">/^##STATISTICS For Job/{s++;j=$NF}
color="#804040">139 s&&/^# Line .*of file/{printf "%s,%s",j,$3;next}
color="#804040">140 s&&/Command [^:]*: /{printf",%s",ENVIRON["_ap" j]; next}
color="#804040">141 s&&/: /{l=$0;sub(/.*: /,"");printf",%s",$0}
color="#804040">142 s&&l~/Exit status:/{s=0;print"";l="";nextfile}
color="#804040">143 color="#804040">' color="#a020f0">$unique_id. color="#804040">[ color="#ff00ff">0-9]* color="#804040">>> color="#a020f0">$unique_id.csv
color="#804040">144 }
color="#804040">145
color="#804040">146 stats2csv
color="#804040">147
color="#804040">148 printf color="#804040">" color="#6a5acd">\n## FINISHED, (stats in %s) color="#6a5acd">\n\n" color="#a020f0">$unique_id.csv
color="#804040">149
color="#804040">150 exit color="#ff00ff">0
color="#804040">151
color="#804040">152
color="#804040">153
color="#804040">154


I remember writing this with the help of the advanced bash tutorial
which is great, and looking back at this program, I would need the
tutorials help if I needed to extend it!



Why not in Python?


 Of course I thought of doing it in Python, but bash's job
control features were used to flesh out the initial idea and it grew to
become the finished article. Looking back at this program as it's over
a year old, I find that I like bash's job control, and gawk one-liners
 (but not the readability) :-)



- Paddy.



href="http://kompozer.net/"> alt="Document made with KompoZer"
src="http://kompozer.sourceforge.net/images/kompozer_80x15.png"
border="0">

3 comments:

  1. wow this is a very clever script. very useful for all types of batch processes.

    two small changes got me running:
    my time was in /usr/bin/time
    I needed to add ./ to my PATH

    ReplyDelete
  2. Congratulations Jerm.

    What do you use it for?

    ReplyDelete
  3. It's not a very big program, but the techniques you use make it interesting: you do a lot in little code.

    If you're still into shell scripting, you might take a look at PPSS: code.google.com/p/ppss --> It is not pretty, but it seems to work for me at least.

    ReplyDelete

Followers

Subscribe Now: google

Add to Google Reader or Homepage

Go deh too!

whos.amung.us