tag:blogger.com,1999:blog-11149365.post5941293135324746230..comments2024-03-23T04:34:59.089+00:00Comments on Go deh!: Easy Command-line ParallelismPaddy3118http://www.blogger.com/profile/06899509753521482267noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-11149365.post-12220287724328249922010-02-02T07:38:22.876+00:002010-02-02T07:38:22.876+00:00Thanks Ole for your comment. Being "old schoo...Thanks Ole for your comment. Being "old school" Unix, I am used to only creating Unix friendly file names, but I take your point. <br /><br />I'll take a look at Parallel.Paddy3118https://www.blogger.com/profile/06899509753521482267noreply@blogger.comtag:blogger.com,1999:blog-11149365.post-9427408027868689592010-01-28T08:13:40.121+00:002010-01-28T08:13:40.121+00:00Both xargs and vxargs deal badly with filenames co...Both xargs and vxargs deal badly with filenames containing special characters. To see the problem try this:<br /><br />touch important_file<br />touch 'not important_file'<br />ls not* | xargs rm<br /><br />You may consider Parallel https://savannah.nongnu.org/projects/parallel/ instead.Ole Tangehttp://ole.tange.dknoreply@blogger.comtag:blogger.com,1999:blog-11149365.post-25035437051919997632009-09-15T05:52:19.183+01:002009-09-15T05:52:19.183+01:00Hi jblanca,
I took a look at psubprocess. Although...Hi jblanca,<br />I took a look at psubprocess. Although it doesn't fit my needs, I wish you well with psubprocess. <br /><br />I do have my process_runner script. We have LSF available, and now the use of xargs -P. I feel i am filling rapildy diminishing holes with parallelism solutions :-)<br /><br />P.S.<br />Reddit just pointed me in the direction of http://vxargs.sourceforge.net/ which is aimed at multi-machine parallelism of arbitrary commands.Paddy3118https://www.blogger.com/profile/06899509753521482267noreply@blogger.comtag:blogger.com,1999:blog-11149365.post-9340203587636675062009-09-14T07:33:51.195+01:002009-09-14T07:33:51.195+01:00Hi:
I've got a cluster with multicore nodes an...Hi:<br />I've got a cluster with multicore nodes and I had the need for easy paralellism too. I've develop a little tool in python that is capable of splitting a job in pieces, running multiple subjobs and joining the outputs. I've not released yet, but maybe it could be of some interest to you. You can take it a look at http://bioinf.comav.upv.es/svn/psubprocess/trunk/src/<br />jblanca _at__ btc dot upv dot esAnonymousnoreply@blogger.com