How to run n batch processes but only m at a time?
It seems that resource constraints allow every four batch processes to share one license - so long as all four batch processes are started from the same terminal on the same machine. Users may have tens of processes to run and currently they are run individually on our compute farm.
I tend to code whilst having a cold for some reason? And needed to use more advanced bash scripting - I decided against Python for the task as I would like my colleague to take over further developments of the code and I can't rely on him knowing Python, (and I don't want to code in Perl today).
I came up with this little demonstrator which runs processes that go to sleep for a random amount of time. The script runs as many as you specify but only a maximum number of proc's at a time.
The script is followed by sample output and was tested on cygwin using bash version 3.2.9(11)-release (i686-pc-cygwin):
1 #!/bin/bash
2
3 ##
4 ## process_runner.sh <concurrent> <total procs>
5 ##
6 ## Example script given the maximum number of processes to
7 ## run concurrently, c, and the total number of processes, n
8 ## runs at most c, processes in the background until all n
9 ## Have been run.
10 ##
11 ## Author Donald 'Paddy' McCarthy Dec. 17 2007
12 ##
13
14 # how many processes to run in parallel
15 concurrent=$1
16 # total processes to run
17 maxprocs=$2
18
19 printf "\n## STARTING %i Processes, %i at a time\n\n" \
20 $maxprocs $concurrent
21
22
23 # main loop wait time between checking background procs.
24 tick=1
25
26
27 # dummy processes sleep for a random time
28 function runproc {
29 local -i n=$1
30 local -i j
31 (( j = 5+$RANDOM*10/32767 ))
32 #(date; printf "#>>>JOB %i : Sleeping for %i\n" $n $j)
33 printf "OUT JOB ,%03i, Sleep for ,%2i, , @,%s\n" $n $j "`date`"
34 sleep $j
35 returned=$?
36 printf "IN JOB ,%03i, Slept for ,%2i, returned ,%i, @,%s\n" \
37 $n $j $returned "`date`"
38 #(date; printf "#<<<JOB %i : Slept for %i\n" $n $j)
39 }
40
41 function print_runstats {
42 printf '# %i Jobs in background. %i/%i started\n\n' \
43 `jobs -r|wc -l` $ran $maxprocs
44 }
45
46 # Bash array running keeps track of the background process numbers
47 # Start with nothing running (sentinel value will not be a process number
48 for ((i=0; i<$concurrent; i+=1 )); do running[$i]=123456789; done
49
50 ran=0
51 until
52 while [ $ran -lt $maxprocs ]; do
53 for ((p=0; p<$concurrent; p+=1 )); do
54 proc=${running[$p]}
55 # Over all running processes...
56 # $proc still running?
57 ps -p $proc | fgrep $proc >/dev/null
58 if [ $? -ne '0' ] ; then
59 # Not found i.e. finished
60 # Run another in background and store the proc number
61 runproc $ran &
62 running[$p]=$!
63 ((ran+=1))
64 if [ $ran -ge $maxprocs ]; then break 1; fi
65 fi
66 done
67 sleep $tick
68 # Status
69 print_runstats
70 done
71
72 sleep $tick
73 # Status
74 print_runstats
75 do [ `jobs -r|wc -l` -eq 0 ]
76 done
77 wait
78 printf "\n## FINISHED\n"
79
80 exit 0
81
82 sample_output=<<!
83 bash$ ./process_runner.sh 2 5
84
85 ## STARTING 5 Processes, 2 at a time
86
87 OUT JOB ,000, Sleep for ,10, , @,Tue Dec 18 09:26:00 GMTST 2007
88 OUT JOB ,001, Sleep for , 8, , @,Tue Dec 18 09:26:00 GMTST 2007
89 # 2 Jobs in background. 2/5 started
90
91 # 2 Jobs in background. 2/5 started
92
93 # 2 Jobs in background. 2/5 started
94
95 # 2 Jobs in background. 2/5 started
96
97 # 2 Jobs in background. 2/5 started
98
99 # 2 Jobs in background. 2/5 started
100
101 IN JOB ,001, Slept for , 8, returned ,0, @,Tue Dec 18 09:26:09 GMTST 2007
102 # 1 Jobs in background. 2/5 started
103
104 OUT JOB ,002, Sleep for ,11, , @,Tue Dec 18 09:26:09 GMTST 2007
105 IN JOB ,000, Slept for ,10, returned ,0, @,Tue Dec 18 09:26:10 GMTST 2007
106 # 2 Jobs in background. 3/5 started
107
108 OUT JOB ,003, Sleep for , 7, , @,Tue Dec 18 09:26:11 GMTST 2007
109 # 2 Jobs in background. 4/5 started
110
111 # 2 Jobs in background. 4/5 started
112
113 # 2 Jobs in background. 4/5 started
114
115 # 2 Jobs in background. 4/5 started
116
117 # 2 Jobs in background. 4/5 started
118
119 IN JOB ,003, Slept for , 7, returned ,0, @,Tue Dec 18 09:26:18 GMTST 2007
120 # 1 Jobs in background. 4/5 started
121
122 OUT JOB ,004, Sleep for , 6, , @,Tue Dec 18 09:26:19 GMTST 2007
123 # 2 Jobs in background. 5/5 started
124
125 IN JOB ,002, Slept for ,11, returned ,0, @,Tue Dec 18 09:26:20 GMTST 2007
126 # 1 Jobs in background. 5/5 started
127
128 IN JOB ,004, Slept for , 6, returned ,0, @,Tue Dec 18 09:26:25 GMTST 2007
129
130 ## FINISHED
131 !
132
133
134
135
136
