[gentoo-dev] Distribuited build

[gentoo-dev] Distribuited build

[Marko Mikulicic]

Hello,

Do you all use only a single machine to build ?
Does anyone use distributed build tools, like distcc or
other tricks.
  While building a sigle package can be "upgraded" to
distcc simply using the parallel make features and
redirecting the compiler, it should be useful if entire ebuilds
could be done in parallel.
  For example, when a package has two dependents they could
be built in parallel. This would be specially useful for
this packages wich doesn't build correctly with -j>1.

I have a lot of unused ghz in my lan which could do lots of
background jobs if they could be well distributed.
  I'm not thinking jet of a way to implement this (it should not
be very difficult) but I'm interested to know what
are the need, habits, testing methodology of the developers.


Marko

PS: for those who doesn't know distcc: http://ftp.samba.org/ftp/distcc/.
It preprocess locally and compiles on remote hosts.

If you share the filesystem with the sources, and have the same
set of libraries/headers on all machines then "makeitdisturbed"
offer better performance. 
http://pcm4036.die.supsi.ch/projects/makeitdisturbed

Re: [gentoo-dev] Distribuited build

[Paul de Vrieze]

On Sunday 09 June 2002 22:31, Marko Mikulicic wrote:
> Hello,
>
> Do you all use only a single machine to build ?
> Does anyone use distributed build tools, like distcc or
> other tricks.
>   While building a sigle package can be "upgraded" to
> distcc simply using the parallel make features and
> redirecting the compiler, it should be useful if entire ebuilds
> could be done in parallel.
>   For example, when a package has two dependents they could
> be built in parallel. This would be specially useful for
> this packages wich doesn't build correctly with -j>1.
>
> I have a lot of unused ghz in my lan which could do lots of
> background jobs if they could be well distributed.
>   I'm not thinking jet of a way to implement this (it should not
> be very difficult) but I'm interested to know what
> are the need, habits, testing methodology of the developers.
>

While I didn't test it yet, openmosix could be a solution for better 
utilization of your computing power. Parallel dependencies are sometimes 
difficult (cross dependencies), also it would require tweaks to emerge

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

Re: [gentoo-dev] Distribuited build

[Marko Mikulicic]

Paul de Vrieze wrote:
> On Sunday 09 June 2002 22:31, Marko Mikulicic wrote
>
> 
> While I didn't test it yet, openmosix could be a solution for better 
> utilization of your computing power. Parallel dependencies are sometimes 
> difficult (cross dependencies), also it would require tweaks to emerge
> 

Well, I delibertely left out openmosix because it is automatically 
supported with make -j n. However there is a big issue with it:
  Openmosix process migration can distribute the load of a single system
through a cluster but if you install n packages they are build in 
sequence so that only compiles from a single build get eventually
distributed. And it's even worst when some packages have parallel build 
disabled.
  Moreover openmosix is not applicable in a etherogeneous network, whith
freebsd, linux, win32, tru64, linux-alpha machines, while distcc can
execute cross-compilers on every architecture (windows too).
  and openmosix requires all to run a special kernel, administrative 
issues,.... I think that something lighter is needed, for it to be
really useful. I have access to a LAN with 16 PIV 1.5Gh but with any 
kindof linux distros, versions, libraries, users. The only 
administrative power I have is to run niced jobs and a good amount of
ram.

  However the problem is not openmosix vs. distcc, while both are 
completely transparent with portage and both have the same problem:
  sequential builds.

As for cross dependencies, how does portage handle the now? There could
a queue for each pending build where dependent build are added.
RDEPEND does not conflict with paralell builds.
  I agree this would require tweaks to emerge. I think it's not
urgent in portage1, but it could go in portage2, which with it's added
performance could resolve this kind of dependence in acceptable time.
  What do you think ?

Marko

Re: [gentoo-dev] Distribuited build

[Nick Jones]

OpenMosix is not a viable option for compiling with "make -j".
The threads of a source file building do not exist long enough
to be migrated. I tired it... Didn't do very well.

> Paul de Vrieze wrote:
> > On Sunday 09 June 2002 22:31, Marko Mikulicic wrote
> >
> > 
> > While I didn't test it yet, openmosix could be a solution for better 
> > utilization of your computing power. Parallel dependencies are sometimes 
> > difficult (cross dependencies), also it would require tweaks to emerge
> > 
> 
> Well, I delibertely left out openmosix because it is automatically 
> supported with make -j n. However there is a big issue with it:
>   Openmosix process migration can distribute the load of a single system
> through a cluster but if you install n packages they are build in 
> sequence so that only compiles from a single build get eventually
> distributed. And it's even worst when some packages have parallel build 
> disabled.
>   Moreover openmosix is not applicable in a etherogeneous network, whith
> freebsd, linux, win32, tru64, linux-alpha machines, while distcc can
> execute cross-compilers on every architecture (windows too).
>   and openmosix requires all to run a special kernel, administrative 
> issues,.... I think that something lighter is needed, for it to be
> really useful. I have access to a LAN with 16 PIV 1.5Gh but with any 
> kindof linux distros, versions, libraries, users. The only 
> administrative power I have is to run niced jobs and a good amount of
> ram.
> 
>   However the problem is not openmosix vs. distcc, while both are 
> completely transparent with portage and both have the same problem:
>   sequential builds.
> 
> As for cross dependencies, how does portage handle the now? There could
> a queue for each pending build where dependent build are added.
> RDEPEND does not conflict with paralell builds.
>   I agree this would require tweaks to emerge. I think it's not
> urgent in portage1, but it could go in portage2, which with it's added
> performance could resolve this kind of dependence in acceptable time.
>   What do you think ?
> 
> Marko