Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash

[Brian Harring <ferringb@gmail.com>]

On Sat, Jun 02, 2012 at 03:50:06PM -0700, Zac Medico wrote:
> On 06/02/2012 02:31 PM, Micha?? G??rny wrote:
> > On Sat, 2 Jun 2012 15:54:03 -0400
> > Mike Frysinger <vapier@gentoo.org> wrote:
> > 
> >> # @FUNCTION: redirect_alloc_fd
> >> # @USAGE: <var> <file> [redirection]
> >> # @DESCRIPTION:
> > 
> > (...and a lot of code)
> > 
> > I may be wrong but wouldn't it be simpler to just stick with a named
> > pipe here? Well, at first glance you wouldn't be able to read exactly
> > one result at a time but is it actually useful?
> 
> I'm pretty sure that the pipe has remain constantly open in read mode
> (which can only be done by assigning it a file descriptor). Otherwise,
> there's a race condition that can occur, where a write is lost because
> it's written just before the reader closes the pipe.

There isn't a race; write side, it'll block once it exceeds pipe buf 
size; read side, bash's read functionality is explicitly byte by byte 
reads to avoid consuming data it doesn't need.

That said, Mgorny's suggestion ignores that the the code already is 
pointed at a fifo.  Presume he's suggesting "Just open it everytime 
you need to fuck with it"... which, sure, 'cept that complicates the 
read side (either having to find a free fd, open to it, then close 
it), or abuse cat or $(<) to pull the results and make the reclaim 
code handle multiple results in a single shot.

Frankly, don't see the point in doing that.  The code isn't that 
complex frankly, and we *need* the overhead of this to be minimal- 
the hand off/reclaim is effectively the bottleneck for scaling.

If the jobs you've backgrounded are a second a piece, it matters less; 
if they're quick little bursts of activity, the scaling *will* be 
limited by how fast we can blast off/reclaim jobs.  Keep in mind that 
the main process has to go find more work to queue up between the 
reclaims, thus this matters more than you'd think.


Either way, that limit varies dependent on time required for each job 
vs # of cores; that said, you run code like this on a 48 core and you 
see it start becoming an actual bottleneck (which is why I came up 
with this hacky bash semaphore).

~harring