Batch Process Runner in bash shell - Forgotten Enhancements

A comment on an href="http://paddy3118.blogspot.com/2007/12/batch-process-runner-in-bash-shell.html">earlier
post caused me to dig out this enhancement to my original
batch process runner.



What you do is create a file of one-liner commands that you would enter
at a command prompt, for example, this is file style="font-weight: bold;">process_list.txt :

1  color="#0000ff"># (Comments have '#' at the left margin)
 color="#804040">2 sleep 4; echo slept for 4 at line 2; csdf_sdf_sd; exit 19
 color="#804040">3 sleep 1; cause-an-error; echo slept for 1 at line 3
 color="#804040">4 sleep 3; echo  color="#ff00ff">'slept for 3 at line 4'
 color="#804040">5 sleep 7; echo  color="#ff00ff">"slept for 7 at line 5"
 color="#804040">6 # Whee!
 color="#804040">7 sleep 9; another-erro; echo slept for 9 at line 7
 color="#804040">8 sleep 5; echo  color="#ff00ff">'slept "for 5" at line 8'

A bit of warning, bash returns the exit code of this laast command it
executes, earlier exit codes won't be seen.



Give my script the number of processes to run in parallel, N, followed
by the above file, and it will create N '.job'  files out of
the non-comment lines of process_list.txt and execute them as
background jobs, with their output sent to '.n' files.:

bash$ ./process_list_runner.sh 2 process_list.txt

## STARTING 6 Processes from file: process_list.txt, 2 at a time with id plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP

# 2 Jobs in background. 6/6 started. 2009-03-21-05:36:36

## FINISHED, (stats in plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.csv)

bash$  style="font-weight: bold;">ls -1 plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.*
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.1
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.2
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.3
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.4
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.5
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.6
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.csv
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.job_0
plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP.job_1
bash$ 

The script does what it can to create a csv file of run stats, which,
with a bit of tidying up in calc produces the following:

frame="void" rules="none">
width="340"> width="110"> width="115"> width="73">

width="1409">## STARTING 6 Processes from file: process_list.txt
size="4">2 at a time with id plr_HPDV8025EA_2009-03-21-05_36_19_PADDYS-HPLAPTOP
align="center" bgcolor="#ccffff" height="123" valign="middle">JOB	align="right" bgcolor="#ccffff" valign="middle">LINE	align="center" bgcolor="#ccffff" valign="middle">Command being timed	align="center" bgcolor="#ccffff" valign="middle">User time (seconds)	align="center" bgcolor="#ccffff" valign="middle">System time (seconds)	align="center" bgcolor="#ccffff" valign="middle">Percent of CPU this job got	align="center" bgcolor="#ccffff" valign="middle">Elapsed (wall clock) time (h:mm:ss or m:ss)	align="center" bgcolor="#ccffff" valign="middle">Maximum resident set size (kbytes)	align="center" bgcolor="#ccffff" valign="middle">Major (requiring I/O) page faults	align="center" bgcolor="#ccffff" valign="middle">Signals delivered	align="center" bgcolor="#ccffff" valign="middle">Page size (bytes)	align="center" bgcolor="#ccffff" valign="middle">Exit status
sdnum="2057;" align="center" bgcolor="#ffffcc" height="18">1	sdnum="2057;" align="center" bgcolor="#ffffcc">2	bgcolor="#ffffcc">sleep 4; echo slept for 4 at line 2; csdf_sdf_sd; exit 19	sdnum="2057;" align="center" bgcolor="#ffffcc">0.13	sdnum="2057;" align="center" bgcolor="#ffffcc">0.15	sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">6.00%	sdval="0.0000503472222222222" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ffffcc">00:04.35	sdval="760832" sdnum="2057;" align="center" bgcolor="#ffffcc">760832	sdnum="2057;" align="center" bgcolor="#ffffcc">3277	sdnum="2057;" align="center" bgcolor="#ffffcc">6	sdval="65536" sdnum="2057;" align="center" bgcolor="#ffffcc">65536	sdnum="2057;" align="center" bgcolor="#ffffcc">19
sdnum="2057;" align="center" bgcolor="#ccccff" height="17">2	sdnum="2057;" align="center" bgcolor="#ccccff">3	bgcolor="#ccccff">sleep 1; cause-an-error; echo slept for 1 at line 3	sdnum="2057;" align="center" bgcolor="#ccccff">0.12	sdnum="2057;" align="center" bgcolor="#ccccff">0.09	sdnum="2057;0;0.00%" align="center" bgcolor="#ccccff">16.00%	sdval="0.0000152777777777778" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ccccff">00:01.32	sdval="761088" sdnum="2057;" align="center" bgcolor="#ccccff">761088	sdnum="2057;" align="center" bgcolor="#ccccff">3273	sdnum="2057;" align="center" bgcolor="#ccccff">6	sdval="65536" sdnum="2057;" align="center" bgcolor="#ccccff">65536	sdnum="2057;" align="center" bgcolor="#ccccff">0
sdnum="2057;" align="center" bgcolor="#ffffcc" height="17">3	sdnum="2057;" align="center" bgcolor="#ffffcc">4	bgcolor="#ffffcc">sleep 3; echo 'slept for 3 at line 4'	sdnum="2057;" align="center" bgcolor="#ffffcc">0.07	sdnum="2057;" align="center" bgcolor="#ffffcc">0.06	sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">4.00%	sdval="0.0000377314814814815" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ffffcc">00:03.26	sdval="617216" sdnum="2057;" align="center" bgcolor="#ffffcc">617216	sdnum="2057;" align="center" bgcolor="#ffffcc">2639	sdnum="2057;" align="center" bgcolor="#ffffcc">3	sdval="65536" sdnum="2057;" align="center" bgcolor="#ffffcc">65536	sdnum="2057;" align="center" bgcolor="#ffffcc">0
sdnum="2057;" align="center" bgcolor="#ccccff" height="17">4	sdnum="2057;" align="center" bgcolor="#ccccff">5	bgcolor="#ccccff">sleep 7; echo "slept for 7 at line 5"	sdnum="2057;" align="center" bgcolor="#ccccff">0.07	sdnum="2057;" align="center" bgcolor="#ccccff">0.07	sdnum="2057;0;0.00%" align="center" bgcolor="#ccccff">2.00%	sdval="0.000084837962962963" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ccccff">00:07.33	sdval="617216" sdnum="2057;" align="center" bgcolor="#ccccff">617216	sdnum="2057;" align="center" bgcolor="#ccccff">2644	sdnum="2057;" align="center" bgcolor="#ccccff">3	sdval="65536" sdnum="2057;" align="center" bgcolor="#ccccff">65536	sdnum="2057;" align="center" bgcolor="#ccccff">0
sdnum="2057;" align="center" bgcolor="#ffffcc" height="18">5	sdnum="2057;" align="center" bgcolor="#ffffcc">7	bgcolor="#ffffcc">sleep 9; another-erro; echo slept for 9 at line 7	sdnum="2057;" align="center" bgcolor="#ffffcc">0.09	sdnum="2057;" align="center" bgcolor="#ffffcc">0.13	sdnum="2057;0;0.00%" align="center" bgcolor="#ffffcc">2.00%	sdval="0.000106597222222222" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ffffcc">00:09.21	sdval="760064" sdnum="2057;" align="center" bgcolor="#ffffcc">760064	sdnum="2057;" align="center" bgcolor="#ffffcc">3274	sdnum="2057;" align="center" bgcolor="#ffffcc">6	sdval="65536" sdnum="2057;" align="center" bgcolor="#ffffcc">65536	sdnum="2057;" align="center" bgcolor="#ffffcc">0
sdnum="2057;" align="center" bgcolor="#ccccff" height="17">6	sdnum="2057;" align="center" bgcolor="#ccccff">8	bgcolor="#ccccff">sleep 5; echo 'slept "for 5" at line 8'	sdnum="2057;" align="center" bgcolor="#ccccff">0.1	sdnum="2057;" align="center" bgcolor="#ccccff">0.06	sdnum="2057;" align="center" bgcolor="#ccccff">0.03	sdval="0.0000622685185185185" sdnum="2057;0;MM:SS.00" align="center" bgcolor="#ccccff">00:05.38	sdval="617216" sdnum="2057;" align="center" bgcolor="#ccccff">617216	sdnum="2057;" align="center" bgcolor="#ccccff">2639	sdnum="2057;" align="center" bgcolor="#ccccff">3	sdval="65536" sdnum="2057;" align="center" bgcolor="#ccccff">65536	sdnum="2057;" align="center" bgcolor="#ccccff">0

The script itself is:

 1  color="#0000ff">#!/bin/bash 
 color="#804040"> 2 #!/opt/TWWfsw/bin/bash -
 color="#804040"> 3 
 color="#804040"> 4 set  color="#008080">-u
 color="#804040"> 5 
 color="#804040"> 6 ##
 color="#804040"> 7 ## process_runner.sh <concurrent> <total procs>
 color="#804040"> 8 ##
 color="#804040"> 9 ## Example script given the maximum number of processes to
 color="#804040"> 10 ## run concurrently, c, and the total number of processes, n
 color="#804040"> 11 ## runs at most c, processes in the background until all n
 color="#804040"> 12 ## Have been run.
 color="#804040"> 13 ##
 color="#804040"> 14 ## Author Donald 'Paddy' McCarthy Dec. 17 2007
 color="#804040"> 15 ##
 color="#804040"> 16 
 color="#804040"> 17 # how many processes to run in parallel
 color="#804040"> 18 concurrent= color="#a020f0">$1
 color="#804040"> 19 # File of commands
 color="#804040"> 20 proclist= color="#804040">" color="#a020f0">$2"
 color="#804040"> 21 
 color="#804040"> 22 ##
 color="#804040"> 23 ##
 color="#804040"> 24 
 color="#804040"> 25 # main loop wait time between checking background procs.
 color="#804040"> 26 tick= color="#ff00ff">1
 color="#804040"> 27 # Unique_id for process files of this run
 color="#804040"> 28 unique_id=plr_ color="#6a5acd">`date +${ color="#a020f0">USER//  color="#804040">/ color="#a020f0">}_%F-%T_ color="#a020f0">${HOSTNAME color="#804040">//  color="#804040">/ color="#a020f0">}`
 color="#804040"> 29 unique_id= color="#a020f0">${unique_id color="#804040">//: color="#804040">/_ color="#a020f0">}
 color="#804040"> 30 
 color="#804040"> 31 function read_proclistfile  color="#6a5acd">{
 color="#804040"> 32  # read processes from file ignoring comment lines
 color="#804040"> 33  maxprocs= color="#ff00ff">0
 color="#804040"> 34  local -i  color="#008080">linenumber= color="#ff00ff">0
 color="#804040"> 35  while color="#804040">  color="#804040">read color="#804040">; color="#804040">  color="#804040">do
 color="#804040"> 36  (( color="#008080">linenumber+= color="#ff00ff">1))
 color="#804040"> 37  # echo [ "${REPLY:0:1}" == "#" ] $maxprocs $REPLY
 color="#804040"> 38  if  color="#804040">[  color="#804040">" color="#a020f0">${REPLY color="#804040">: color="#ff00ff">0: color="#ff00ff">1} color="#804040">"  color="#804040">!=  color="#804040">" color="#ff00ff">#"  color="#804040">]  color="#804040">;  color="#804040">then
 color="#804040"> 39  ((maxprocs+ color="#804040">= color="#ff00ff">1))
 color="#804040"> 40  allprocs color="#804040">[ color="#a020f0">$maxprocs color="#804040">] color="#804040">= color="#804040">" color="#a020f0">$REPLY"
 color="#804040"> 41  allprocline color="#804040">[ color="#a020f0">$maxprocs color="#804040">] color="#804040">= color="#a020f0">$linenumber
 color="#804040"> 42  # echo $linenumber $maxprocs $REPLY
 color="#804040"> 43  fi
 color="#804040"> 44  done  color="#804040"><  color="#a020f0">$proclist
 color="#804040"> 45 }
 color="#804040"> 46 
 color="#804040"> 47 read_proclistfile
 color="#804040"> 48 
 color="#804040"> 49 printf  color="#804040">" color="#6a5acd">\n## STARTING %i Processes from file: %s, %i at a time with id %s color="#6a5acd">\n\n" \
 color="#804040"> 50  $maxprocs  color="#a020f0">$proclist  color="#a020f0">$concurrent  color="#a020f0">$unique_id
 color="#804040"> 51 
 color="#804040"> 52 
 color="#804040"> 53 
 color="#804040"> 54 function assemble_job  color="#6a5acd">{
 color="#804040"> 55  #
 color="#804040"> 56  local  color="#008080">plr_unique_id= color="#804040">" color="#a020f0">$1"; color="#804040">local  color="#008080">plr_ran=" color="#a020f0">$2"; color="#804040">local  color="#008080">plr_proc= color="#804040">" color="#a020f0">$3"; color="#804040">local  color="#008080">plr_procline= color="#804040">" color="#a020f0">$4"; color="#804040">local  color="#008080">plr_proclist= color="#804040">" color="#a020f0">$5"
 color="#804040"> 57  cat  color="#804040"><<!
 color="#804040"> 58 
 color="#804040"> 59 #  color="#a020f0">$plr_unique_id color="#ff00ff"> $plr_ran color="#ff00ff"> 
 color="#804040"> 60 ` color="#804040">local color="#6a5acd">`
 color="#804040"> 61 
 color="#804040"> 62 trap 'error= color="#6a5acd">\$?;printf " color="#6a5acd">\n\n##STATISTICS For Job %i color="#6a5acd">\n"  color="#a020f0">$plr_ran; printf "# Line %i of file %s color="#6a5acd">\n\n"  color="#a020f0">$plr_procline color="#ff00ff"> $plr_proclist color="#ff00ff">; exit \$ color="#ff00ff">error' EXIT
 color="#804040"> 63 
 color="#804040"> 64 
 color="#804040"> 65 $plr_proc
 color="#804040"> 66 
 color="#804040"> 67 !
 color="#804040"> 68 }
 color="#804040"> 69 
 color="#804040"> 70 function print_runstats  color="#6a5acd">{
 color="#804040"> 71  printf  color="#804040">' color="#ff00ff"># %i Jobs in background. %i/%i started. %s\r color="#804040">' \
 color="#804040"> 72  ` color="#804040">jobs color="#6a5acd"> -r color="#804040">| color="#6a5acd">wc -l` $ran  color="#a020f0">$maxprocs  color="#804040">" color="#6a5acd">`date +%F-%T` color="#804040">"
 color="#804040"> 73 }
 color="#804040"> 74 
 color="#804040"> 75 # Bash array running keeps track of the background process numbers
 color="#804040"> 76 # Start with nothing running (sentinel value will not be a process number
 color="#804040"> 77 for  color="#6a5acd">((i= color="#ff00ff">0; i color="#804040">< color="#a020f0">$concurrent color="#804040">; i+ color="#804040">= color="#ff00ff">1 )) color="#804040">;  color="#804040">do running color="#804040">[ color="#a020f0">$i] color="#804040">= color="#ff00ff">123456789 color="#804040">;  color="#804040">done
 color="#804040"> 78 
 color="#804040"> 79 ran= color="#ff00ff">0
 color="#804040"> 80 until
 color="#804040"> 81  while  color="#804040">[  color="#a020f0">$ran -lt  color="#a020f0">$maxprocs  color="#804040">] color="#804040">;  color="#804040">do
 color="#804040"> 82  for  color="#6a5acd">((p= color="#ff00ff">0; p color="#804040">< color="#a020f0">$concurrent color="#804040">; p+ color="#804040">= color="#ff00ff">1 )) color="#804040">;  color="#804040">do
 color="#804040"> 83  proc= color="#a020f0">${running color="#a020f0">[$p color="#a020f0">]}
 color="#804040"> 84  # Over all running processes...
 color="#804040"> 85  # $proc still running?
 color="#804040"> 86  ps -p  color="#a020f0">$proc |  color="#804040">fgrep  color="#a020f0">$proc >/dev/null
 color="#804040"> 87  if  color="#804040">[  color="#a020f0">$? -ne  color="#804040">' color="#ff00ff">0'  color="#804040">]  color="#804040">;  color="#804040">then
 color="#804040"> 88  # Not found i.e. its finished
 color="#804040"> 89  # So start the next
 color="#804040"> 90  ((ran+ color="#804040">= color="#ff00ff">1))
 color="#804040"> 91  #(assemble_job ) >$unique_id.job_$p
 color="#804040"> 92  ( assemble_job  color="#804040">" color="#a020f0">$unique_id color="#804040">"  color="#804040">" color="#a020f0">$ran"  color="#804040">" color="#a020f0">${allprocs color="#a020f0">[$ran color="#a020f0">]} color="#804040">" \
 color="#804040"> 93  ${ color="#a020f0">allprocline color="#a020f0">[$ran color="#a020f0">]}  color="#a020f0">$proclist  color="#804040">)  color="#804040">>  color="#a020f0">$unique_id.job_ color="#a020f0">$p
 color="#804040"> 94  chmod +x  color="#a020f0">$unique_id.job_ color="#a020f0">$p
 color="#804040"> 95  # run one job in background collecting run stats
 color="#804040"> 96  (/bin/ color="#804040">time -v -a -o  color="#a020f0">$unique_id. color="#a020f0">$ran < /dev/null  color="#a020f0">$unique_id.job_ color="#a020f0">$p 2>&1  color="#804040">)  color="#804040">>  color="#a020f0">$unique_id. color="#a020f0">$ran &
 color="#804040"> 97  running color="#804040">[ color="#a020f0">$p] color="#804040">= color="#a020f0">$!
 color="#804040"> 98  runningprocnum color="#804040">[ color="#a020f0">$p] color="#804040">= color="#a020f0">$ran
 color="#804040"> 99  if  color="#804040">[  color="#a020f0">$ran -ge  color="#a020f0">$maxprocs  color="#804040">] color="#804040">;  color="#804040">then  color="#804040">break  color="#ff00ff">1;  color="#804040">fi
 color="#804040">100  #exit
 color="#804040">101  fi
 color="#804040">102  done
 color="#804040">103  sleep  color="#a020f0">$tick
 color="#804040">104  # Status
 color="#804040">105  print_runstats
 color="#804040">106  done
 color="#804040">107 
 color="#804040">108  ## break 1 gets us here!
 color="#804040">109 
 color="#804040">110  # Keep on printing status while there are background processes 
 color="#804040">111  # even though there are no more to start.
 color="#804040">112  sleep  color="#a020f0">$tick
 color="#804040">113  # Status
 color="#804040">114  print_runstats
 color="#804040">115 do  color="#804040">[  color="#6a5acd">`jobs color="#6a5acd"> -r color="#804040">| color="#6a5acd">wc -l` -eq  color="#ff00ff">0 ]
 color="#804040">116 done
 color="#804040">117 wait
 color="#804040">118 
 color="#804040">119 function stats2csv  color="#6a5acd">{
 color="#804040">120  # Gather overall stats in csv file
 color="#804040">121 
 color="#804040">122  # CSV Heading
 color="#804040">123  printf  color="#804040">" color="#6a5acd">\n## STARTING %i Processes from file: %s color="#6a5acd">\n%i at a time with id %s color="#6a5acd">\n\n" \
 color="#804040">124  $maxprocs  color="#a020f0">$proclist  color="#a020f0">$concurrent  color="#a020f0">$unique_id  color="#804040">>  color="#a020f0">$unique_id.csv
 color="#804040">125  # CSV Column labels 
 color="#804040">126  gawk  color="#804040">' color="#ff00ff">/^##STATISTICS For Job/{s++}
 color="#804040">127  s&&/^# Line .*of file/{printf "JOB,LINE";next}
 color="#804040">128  s&&/: /{l=$0;sub(/: .*/,"");$1=$1;printf",%s",$0}
 color="#804040">129  s&&l~/Exit status:/{s=0;print"";l="";exit}
 color="#804040">130   color="#804040">'  color="#a020f0">$unique_id. color="#804040">[ color="#ff00ff">0-9]*  color="#804040">>>  color="#a020f0">$unique_id.csv
 color="#804040">131 
 color="#804040">132  # Stick allprocs into env for later gawk substitution of command
 color="#804040">133  for i  color="#804040">in  color="#a020f0">${!allprocs color="#a020f0">[@] color="#a020f0">};  color="#804040">do
 color="#804040">134  declare  color="#008080">-x _ap color="#a020f0">$i= color="#804040">" color="#a020f0">${allprocs color="#a020f0">[$i color="#a020f0">]} color="#804040">"
 color="#804040">135  done
 color="#804040">136  #env|grep '_ap.=' ; debug
 color="#804040">137  # CSV Rows
 color="#804040">138  gawk  color="#804040">' color="#ff00ff">/^##STATISTICS For Job/{s++;j=$NF}
 color="#804040">139  s&&/^# Line .*of file/{printf "%s,%s",j,$3;next}
 color="#804040">140  s&&/Command [^:]*: /{printf",%s",ENVIRON["_ap" j]; next}
 color="#804040">141  s&&/: /{l=$0;sub(/.*: /,"");printf",%s",$0}
 color="#804040">142  s&&l~/Exit status:/{s=0;print"";l="";nextfile}
 color="#804040">143   color="#804040">'  color="#a020f0">$unique_id. color="#804040">[ color="#ff00ff">0-9]*  color="#804040">>>  color="#a020f0">$unique_id.csv
 color="#804040">144 }
 color="#804040">145 
 color="#804040">146 stats2csv
 color="#804040">147 
 color="#804040">148 printf  color="#804040">" color="#6a5acd">\n## FINISHED, (stats in %s) color="#6a5acd">\n\n"  color="#a020f0">$unique_id.csv
 color="#804040">149 
 color="#804040">150 exit  color="#ff00ff">0
 color="#804040">151 
 color="#804040">152 
 color="#804040">153 
 color="#804040">154 

I remember writing this with the help of the advanced bash tutorial
which is great, and looking back at this program, I would need the
tutorials help if I needed to extend it!

Why not in Python?

 Of course I thought of doing it in Python, but bash's job
control features were used to flesh out the initial idea and it grew to
become the finished article. Looking back at this program as it's over
a year old, I find that I like bash's job control, and gawk one-liners
 (but not the readability) :-)



- Paddy.



href="http://kompozer.net/"> alt="Document made with KompoZer"
src="http://kompozer.sourceforge.net/images/kompozer_80x15.png"
border="0">