[gentoo-user] distcc and crossdev, anyone?

[gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

Hello list,

I've bought (against my better judgement) an Atom N270 box to be a LAN 
server, but it's a bit slow compared with the other boxes on the 
network. A big bit, actually - 69 minutes to compile a kernel compared 
with less than 9 minutes on this workstation.

I thought I'd give distcc a go, but after reading the Gentoo distcc and 
crossdev guides and doing what they say I get no result. I might just as 
well not have made the effort. The Atom box just labours with the emerge 
without trying to send anything to the server box I've set up for the 
purpose.

Are the Gentoo guides up to date?

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] distcc and crossdev, anyone?

[Jacob Todd]

[-- Attachment #1: Type: text/plain, Size: 41 bytes --]

Could you post your distcc config files?

[-- Attachment #2: Type: text/html, Size: 48 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Friday 17 December 2010 23:23:10 Jacob Todd wrote:
> Could you post your distcc config files?

$ extract /etc/conf.d/distccd
DISTCCD_OPTS=""
DISTCCD_EXEC="/usr/bin/distccd"
DISTCCD_PIDFILE="/var/run/distccd/distccd.pid"
DISTCCD_OPTS="${DISTCCD_OPTS} --port 3632"
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.2.0/24"
DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.2.2"
DISTCCD_OPTS="${DISTCCD_OPTS} -N 15"

(Extract is just a mini-script to cut out comments.)

$ cat /etc/distcc/hosts
ostn.ethnet

Ostn is the box that's supposed to do the compilation, but the Atom 
client box just doesn't bother trying distcc. If it had and I had an 
error in my config I'd have got an error message.

$ grep distcc /etc/make.conf
DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
FEATURES="buildpkg ccache distcc fixpackages parallel-fetch userfetch"

Maybe another of those features is incompatible with distcc. I'd also 
have expected an error message in that case, but I just get a bog-
standard emerge process running locally.

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] distcc and crossdev, anyone?

[Al]

On Friday 17 December 2010 22:56:29 Peter Humphrey wrote:
> Hello list,
> 
> I've bought (against my better judgement) an Atom N270 box to be a LAN
> server, but it's a bit slow compared with the other boxes on the
> network. A big bit, actually - 69 minutes to compile a kernel compared
> with less than 9 minutes on this workstation.
> 
> I thought I'd give distcc a go, but after reading the Gentoo distcc and
> crossdev guides and doing what they say I get no result. I might just as
> well not have made the effort. The Atom box just labours with the emerge
> without trying to send anything to the server box I've set up for the
> purpose.
> 
> Are the Gentoo guides up to date?
Hi I have a N270 netbook and use crossdev and distcc.
It is definately usefull but for doing a kernel compile I havn't tried it.
Do you see any improvement when using for emerges.
Whats your setup like

Al

Re: [gentoo-user] distcc and crossdev, anyone?

[Stroller]

On 17/12/2010, at 10:56pm, Peter Humphrey wrote:
> ... an Atom N270 box ... server, but it's a bit slow compared with the other boxes on the 
> network. A big bit, actually - 69 minutes to compile a kernel compared 
> with less than 9 minutes on this workstation.

9 minutes!?!? I'm flabbergasted. The machines I have around here, I consider 1 hour to compile a kernel pretty good. Actually I'm in the process of migrating to newer hardware, but I haven't tested kernel compilation times.

Nevertheless: it's a server. Open a `tmux` session, start it compiling, go watch a movie.

Stroller.

Re: [gentoo-user] distcc and crossdev, anyone?

[Paul Hartman]

On Sat, Dec 18, 2010 at 2:44 AM, Stroller
<stroller@stellar.eclipse.co.uk> wrote:
>
> On 17/12/2010, at 10:56pm, Peter Humphrey wrote:
>> ... an Atom N270 box ... server, but it's a bit slow compared with the other boxes on the
>> network. A big bit, actually - 69 minutes to compile a kernel compared
>> with less than 9 minutes on this workstation.
>
> 9 minutes!?!? I'm flabbergasted. The machines I have around here, I consider 1 hour to
> compile a kernel pretty good. Actually I'm in the process of migrating to newer hardware,
> but I haven't tested kernel compilation times.

[brag]
real    1m46.250s
user    11m54.140s
sys     0m57.290s
[/brag]

Less than 2 minutes here ;) That is for "make -j9 all" on Core i7 920
(OC'ed to 3.5GHz)

To be more on topic, I've never been able to figure out distcc to the
point where I feel comfortable that I've done it correctly. I have a
laptop where emerging a new release of KDE takes more than 1 day, and
the above mentioned workstation where it takes an hour. Followed the
wiki and I could see compilation happening on the remote machine, but
it was few and far between. It usually seemed like using it was slower
than not using it at all. I tried to set it to just not use the local
machine for anything but was never able to get that to work. (I'm not
sure if it's even possible?)

I probably did something wrong or misunderstood some fundamental part
of it, but I gave up on it long ago.

Re: [gentoo-user] distcc and crossdev, anyone?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Fri, 17 Dec 2010 22:56:29 +0000, Peter Humphrey wrote:

> I've bought (against my better judgement) an Atom N270 box to be a LAN 
> server, but it's a bit slow compared with the other boxes on the 
> network. A big bit, actually - 69 minutes to compile a kernel compared 
> with less than 9 minutes on this workstation.
> 
> I thought I'd give distcc a go, but after reading the Gentoo distcc and 
> crossdev guides and doing what they say I get no result. I might just
> as well not have made the effort. The Atom box just labours with the
> emerge without trying to send anything to the server box I've set up
> for the purpose.

I've found there's just too much overhead with distcc, plus much of the
work is still done locally. I have a couple of Atom boxes, a server and a
netbook, and I've set up a chroot for each on my workstation. In the
chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to
both computers, then I emerge -k on the Atom box.


-- 
Neil Bothwick

Top Oxymorons Number 1: Microsoft Works

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:

> I've found there's just too much overhead with distcc, plus much of
> the work is still done locally.

I expected that but I wanted to try it to see.

> I have a couple of Atom boxes, a server and a netbook, and I've set up
> a chroot for each on my workstation. In the chroot I have
> FEATURES=buildpkg, using an NFS mounted PKGDIR available to both
> computers, then I emerge -k on the Atom box.

Maybe I'll go this way instead. Thanks for the idea, which is similar to 
one from YoYo Siska three days ago.

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] distcc and crossdev, anyone?

[Bill Longman]

On 12/18/2010 07:15 AM, Peter Humphrey wrote:
> On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:
> 
>> I've found there's just too much overhead with distcc, plus much of
>> the work is still done locally.
> 
> I expected that but I wanted to try it to see.
> 
>> I have a couple of Atom boxes, a server and a netbook, and I've set up
>> a chroot for each on my workstation. In the chroot I have
>> FEATURES=buildpkg, using an NFS mounted PKGDIR available to both
>> computers, then I emerge -k on the Atom box.
> 
> Maybe I'll go this way instead. Thanks for the idea, which is similar to 
> one from YoYo Siska three days ago.

I had my Atom 330 running as a distcc client for a long time. I have
several other speedy CPUs alongside it so it could spray plenty of
compilation requests out its gigabit NIC to various much beefier
machines. But as Neil stated, lots of the processing still occurs
locally, so as you increase nodes, you need to decrease the amount of
compilation done locally. With such a disparity between CPU, it takes
less time overall to just do it the way Neil describes - make a chroot
and then just build it with the intention that the slow CPUs will use
the binary build.

You still need lots of CPU to compile, so a slow machine will still
compile slowly. If your client is a pokey 1.6GHz Atom and you're sending
jobs to two quad core 3GHz CPUs on your subnet, you'll soon see that the
Atom's load goes up toward 8 as it tries to bring those remote jobs
back. So, the four threads on my 330 get completely filled up and it's
dog slow. And it's even more painful when you use the preprocessor
because the client must zip the compile "construction" before it ships
it out, so you have even less CPU available for compilation (although
you get some of that back).

All said and done, my back-of-the-napkin and seat-of-the-pants
calculation tells me that I still get a _minimum_ 25% reduction in
overall compile times with distcc. That's my experience after using
distcc for almost ten years with various configurations of network and CPUs.

Re: [gentoo-user] distcc and crossdev, anyone?

[Petri Rosenström]

[-- Attachment #1: Type: text/plain, Size: 2822 bytes --]

On Tue, Dec 21, 2010 at 1:56 AM, Bill Longman <bill.longman@gmail.com>wrote:

> On 12/18/2010 07:15 AM, Peter Humphrey wrote:
> > On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:
> >
> >> I've found there's just too much overhead with distcc, plus much of
> >> the work is still done locally.
> >
> > I expected that but I wanted to try it to see.
> >
> >> I have a couple of Atom boxes, a server and a netbook, and I've set up
> >> a chroot for each on my workstation. In the chroot I have
> >> FEATURES=buildpkg, using an NFS mounted PKGDIR available to both
> >> computers, then I emerge -k on the Atom box.
> >
> > Maybe I'll go this way instead. Thanks for the idea, which is similar to
> > one from YoYo Siska three days ago.
>
> I had my Atom 330 running as a distcc client for a long time. I have
> several other speedy CPUs alongside it so it could spray plenty of
> compilation requests out its gigabit NIC to various much beefier
> machines. But as Neil stated, lots of the processing still occurs
> locally, so as you increase nodes, you need to decrease the amount of
> compilation done locally. With such a disparity between CPU, it takes
> less time overall to just do it the way Neil describes - make a chroot
> and then just build it with the intention that the slow CPUs will use
> the binary build.
>
> You still need lots of CPU to compile, so a slow machine will still
> compile slowly. If your client is a pokey 1.6GHz Atom and you're sending
> jobs to two quad core 3GHz CPUs on your subnet, you'll soon see that the
> Atom's load goes up toward 8 as it tries to bring those remote jobs
> back. So, the four threads on my 330 get completely filled up and it's
> dog slow. And it's even more painful when you use the preprocessor
> because the client must zip the compile "construction" before it ships
> it out, so you have even less CPU available for compilation (although
> you get some of that back).
>
> All said and done, my back-of-the-napkin and seat-of-the-pants
> calculation tells me that I still get a _minimum_ 25% reduction in
> overall compile times with distcc. That's my experience after using
> distcc for almost ten years with various configurations of network and
> CPUs.
>
>
I have set up my system as Neil described chroots for different systems on a
fast computer. I use this setup for my gentoo boxes I have and it has made
my compilations fast(er). I tried to use distcc with one U2300 celeron and
some amd 4x cpu and the amd didn't really compile, because the U2300 was a
bottleneck, so I decided to chroot it and been happy ever since.

I have been thinking about a tool that could automagically start the emerge
on the remote system. I thought about just ssh in with a script. But I am on
so many flaky Internet connections that it isn't reliable enough.

Petri

[-- Attachment #2: Type: text/html, Size: 3408 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:

> I've found there's just too much overhead with distcc, plus much of
> the work is still done locally. I have a couple of Atom boxes, a
> server and a netbook, and I've set up a chroot for each on my
> workstation. In the chroot I have FEATURES=buildpkg, using an NFS
> mounted PKGDIR available to both computers, then I emerge -k on the
> Atom box.

I'd like to try this, but I haven't yet found the right set of 
parameters: either I'm not exporting the PKGDIR properly or my fstab 
isn't right. I've followed this guide: http://en.gentoo-
wiki.com/wiki/NFS.

I get this on the workstation when trying to nfs-mount the exported 
PKGDIR:

# mount /mnt/nfs
mount.nfs: an incorrect mount option was specified

Is there a secret incantation?

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Thursday 23 December 2010 14:44:25 I wrote:

> I get this on the workstation when trying to nfs-mount the exported
> PKGDIR:
> 
> # mount /mnt/nfs
> mount.nfs: an incorrect mount option was specified

The system log on vt12 says "bad mount option value specified: vers=4". 
Ah-hah! I thought. All I have to do is add "vers=3" to the fstab options 
and it should work.

It did. Sorry about the noise.

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:
> On Fri, 17 Dec 2010 22:56:29 +0000, Peter Humphrey wrote:
> > I've bought (against my better judgement) an Atom N270 box to be a LAN
> > server, but it's a bit slow compared with the other boxes on the
> > network. A big bit, actually - 69 minutes to compile a kernel compared
> > with less than 9 minutes on this workstation.
> > 
> > I thought I'd give distcc a go, but after reading the Gentoo distcc and
> > crossdev guides and doing what they say I get no result. I might just
> > as well not have made the effort. The Atom box just labours with the
> > emerge without trying to send anything to the server box I've set up
> > for the purpose.
> 
> I've found there's just too much overhead with distcc, plus much of the
> work is still done locally. I have a couple of Atom boxes, a server and a
> netbook, and I've set up a chroot for each on my workstation. In the
> chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to
> both computers, then I emerge -k on the Atom box.

I've been experimenting with nfs-mounting the whole Atom file system to /target 
in a chroot on my workstation, then setting --root=/target and --config-
root=/target on every portage command. I can't recommend it.

Numerous packages require to be installed into both the chroot and the target. I 
suppose that's not too onerous, even though I haven't found a way to predict 
which packages will be affected, but I've found that, when I go back to the Atom 
box and emerge -pkuv world, a lot of the packages that should already have been 
upgraded haven't been, and I have to emerge them on the Atom box directly.

The states of the target and the native chroot are neither consistent nor 
independent - it's a mess.

It was a nice idea to enable portage to work in this way, but it's still full of 
holes. Maybe all packages need some extra configuring; I don't know. A lot more 
work is definitely needed by someone, at any rate.

I've decided to revert to Neil's method (once I've shaken this infection off).

-- 
Rgds
Peter

Re: [gentoo-user] distcc and crossdev, anyone?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

On Fri, 1 Apr 2011 12:43:44 +0100, Peter Humphrey wrote:

> I've decided to revert to Neil's method (once I've shaken this
> infection off).

Could you please confirm that the infection is in no way linked to me or
my "method" :-O 


-- 
Neil Bothwick

Always proofread carefully to see if you any words out.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Dale]

Neil Bothwick wrote:
> On Fri, 1 Apr 2011 12:43:44 +0100, Peter Humphrey wrote:
>
>    
>> I've decided to revert to Neil's method (once I've shaken this
>> infection off).
>>      
> Could you please confirm that the infection is in no way linked to me or
> my "method" :-O
>
>
>    

I hope you get rid of the infection but don't send it this way.  I don't 
need a trip to the hospital again.  Having folks check on you is nice 
but I would prefer hotel room service to a hospital nurse and a pricey 
Dr.  o_O

Of course, I would rather spend the money on building a computer too.  
More fun than wondering if you are about to meet your maker.

Dale

:-)  :-)

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 02 April 2011 09:57:57 Neil Bothwick wrote:

> Could you please confirm that the infection is in no way linked to me or
> my "method" :-O

Gladly. Not sure what it's linked to, nor even what it is, but it doesn't half 
sap the energy.

-- 
Rgds
Peter

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 18 December 2010 10:18:43 Neil Bothwick wrote:

> I've found there's just too much overhead with distcc, plus much of the
> work is still done locally. I have a couple of Atom boxes, a server and a
> netbook, and I've set up a chroot for each on my workstation. In the
> chroot I have FEATURES=buildpkg, using an NFS mounted PKGDIR available to
> both computers, then I emerge -k on the Atom box.

Just to confirm, and to save me having to think more deeply than I'm able pro 
tem, does each chroot have identical make.conf and package.use to those on its 
target? And do you nfs-mount only the PKGDIR, or the whole of /usr/portage/ ?

-- 
Rgds
Peter

Re: [gentoo-user] distcc and crossdev, anyone?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

On Sat, 2 Apr 2011 16:19:45 +0100, Peter Humphrey wrote:

> > I've found there's just too much overhead with distcc, plus much of
> > the work is still done locally. I have a couple of Atom boxes, a
> > server and a netbook, and I've set up a chroot for each on my
> > workstation. In the chroot I have FEATURES=buildpkg, using an NFS
> > mounted PKGDIR available to both computers, then I emerge -k on the
> > Atom box.  
> 
> Just to confirm, and to save me having to think more deeply than I'm
> able pro tem, does each chroot have identical make.conf and package.use

Yes. the script that I use to start up and enter the chroot for each
system not only does the usual mounting of /dev/ and /proc in the
chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with the
real target. Make.conf has to be maintained manually, because there are
settings in the two that need to be different, although I suppose I could
split out the common settings, USE, CHOST etc, into a separate file and
source that.

> to those on its target? And do you nfs-mount only the PKGDIR, or the
> whole of /usr/portage/ ?

Just PKGDIR and DISTDIR, I have an NFS exported directory that contains a
global DISTDIR and individual PKGDIRS, as well as my and layman's
overlays.


-- 
Neil Bothwick

We can sympathize with a child who is afraid of the dark, but the
tragedy of life is that most people are afraid of the light.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Saturday 02 April 2011 23:47:42 Neil Bothwick wrote:

> Yes. the script that I use to start up and enter the chroot for each
> system not only does the usual mounting of /dev/ and /proc in the
> chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with the
> real target. Make.conf has to be maintained manually, because there are
> settings in the two that need to be different, although I suppose I could
> split out the common settings, USE, CHOST etc, into a separate file and
> source that.

In my case the chroot is identical in structure to the real target, apart from 
the number of cores, so I can copy make.conf into the chroot without risk.

> > to those on its target? And do you nfs-mount only the PKGDIR, or the
> > whole of /usr/portage/ ?
> 
> Just PKGDIR and DISTDIR, I have an NFS exported directory that contains a
> global DISTDIR and individual PKGDIRS, as well as my and layman's
> overlays.

I'm hoping not to have to use any overlays here, mostly because the target box 
is going to be a LAN server, so shouldn't need any cutting-edge versions of 
anything.

My setup mounts /usr/portage over nfs from the target; it's going to contain the 
latest tree for rsync'ing clients from, so it's the master version.

Interesting - many thanks. It's all getting quite involved.

-- 
Rgds
Peter

Re: [gentoo-user] distcc and crossdev, anyone?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On Sun, 3 Apr 2011 14:55:39 +0100, Peter Humphrey wrote:

> > Yes. the script that I use to start up and enter the chroot for each
> > system not only does the usual mounting of /dev/ and /proc in the
> > chroot, it also rsyncs /etc/portage and /var/lib/portage/world* with
> > the real target. Make.conf has to be maintained manually, because
> > there are settings in the two that need to be different, although I
> > suppose I could split out the common settings, USE, CHOST etc, into a
> > separate file and source that.  
> 
> In my case the chroot is identical in structure to the real target,
> apart from the number of cores, so I can copy make.conf into the chroot
> without risk.

You probably don't want EMERGE_DEFAULT_OPTS="--usepkg" in the chroot's
make.conf.

I also turn off the ELOG* functions in the chroot, as the emails it sends
contain the wrong hostname, leading to much confusion.


-- 
Neil Bothwick

/ For security reasons, all text in this mail
  is double-rot13 encrypted. /

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Sunday 03 April 2011 18:08:25 Neil Bothwick wrote:

> You probably don't want EMERGE_DEFAULT_OPTS="--usepkg" in the chroot's
> make.conf.

In fact I don't have it in either of them; so far I've been issuing manual 
parameters. When I've settled the process down I'll encapsulate it in scripts.

> I also turn off the ELOG* functions in the chroot, as the emails it sends
> contain the wrong hostname, leading to much confusion.

Good idea. Logging isn't working for me yet either, but with any luck it will 
be.

Thanks again

-- 
Rgds
Peter

Re: [gentoo-user] distcc and crossdev, anyone?

[Peter Humphrey]

On Sunday 03 April 2011 18:24:51 Peter Humphrey wrote:

> Logging isn't working for me yet either,

I should have said that e-mailing of logs isn't working.

-- 
Rgds
Peter