[gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Walter Dnes]

  I ran into a couple of problems with distcc cross-compiling on a
64-bit host for a 32-bit host.  One was with ffmpeg, and the other
one was seamonkey (built-from-source).  There's a thread at
http://www.gossamer-threads.com/lists/gentoo/user/298324 which mentions
ffmpeg with symptoms identical to mine.

  ffmpeg is no problem doing locally.  But seamonkey built-from-source
is a 14 hour marathon on the ancient 32-bit Atom.  It's the reason I got
into distcc in the first place.  Fortunately, seamonkey-bin exists, and
builds properly.  Basic Youtube videos (End Of The Line; Travelling
Wilburys; HTML5; H264; 360-line-resolution) peg the double-core Atom
with a cpu load of 2.75.  That's not Seamonkey's fault; what do you
expect from an Atom, driving an un-accelerated framebuffer?  I'm happy
that the devs went to the trouble of reverse-engineering the Poulsbo
(bleagh) chip.

  The thread listed above mentions that distcc without "pump" can
sometimes solve crossdev build problems.  Can it be used to build
seamonkey from source on the host and manually push it to the client?

  While we're on the topic of distcc, it's update time.  The client
shows...

aa1 ~ # gcc-config -l
 [1] i686-pc-linux-gnu-4.8.4 *

...and the host shows...

[d531][waltdnes][~] gcc-config -l
 [1] i686-pc-linux-gnu-4.8.3 *

 [2] x86_64-pc-linux-gnu-4.8.4 *

According to the crossdev "--help" listing...
    -S, --stable             Use latest stable versions as default
    -C, --clean target       Uninstall specified target

  This implies that on the host, I should do the following...

crossdev -C -t i686-pc-linux-gnu
crossdev -S -t i686-pc-linux-gnu

  Before I take the plunge, can anybody with experience confirm that
this is the correct way to do it?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Fernando Rodriguez]

On Saturday, May 02, 2015 9:10:01 PM Walter Dnes wrote:
>   I ran into a couple of problems with distcc cross-compiling on a
> 64-bit host for a 32-bit host.  One was with ffmpeg, and the other
> one was seamonkey (built-from-source).  There's a thread at
> http://www.gossamer-threads.com/lists/gentoo/user/298324 which mentions
> ffmpeg with symptoms identical to mine.
> 
>   ffmpeg is no problem doing locally.  But seamonkey built-from-source
> is a 14 hour marathon on the ancient 32-bit Atom.  It's the reason I got
> into distcc in the first place.  Fortunately, seamonkey-bin exists, and
> builds properly.  Basic Youtube videos (End Of The Line; Travelling
> Wilburys; HTML5; H264; 360-line-resolution) peg the double-core Atom
> with a cpu load of 2.75.  That's not Seamonkey's fault; what do you
> expect from an Atom, driving an un-accelerated framebuffer?  I'm happy
> that the devs went to the trouble of reverse-engineering the Poulsbo
> (bleagh) chip.
> 
>   The thread listed above mentions that distcc without "pump" can
> sometimes solve crossdev build problems.  Can it be used to build
> seamonkey from source on the host and manually push it to the client?
> 
>   While we're on the topic of distcc, it's update time.  The client
> shows...
> 
> aa1 ~ # gcc-config -l
>  [1] i686-pc-linux-gnu-4.8.4 *
> 
> ...and the host shows...
> 
> [d531][waltdnes][~] gcc-config -l
>  [1] i686-pc-linux-gnu-4.8.3 *
> 
>  [2] x86_64-pc-linux-gnu-4.8.4 *
> 
> According to the crossdev "--help" listing...
>     -S, --stable             Use latest stable versions as default
>     -C, --clean target       Uninstall specified target
> 
>   This implies that on the host, I should do the following...
> 
> crossdev -C -t i686-pc-linux-gnu
> crossdev -S -t i686-pc-linux-gnu
> 
>   Before I take the plunge, can anybody with experience confirm that
> this is the correct way to do it?
> 
> 

The difference is that without pump mode the source files are run through the 
preprocessor locally and a single preprocessed file is sent to the host for 
each compilation unit. With pump mode the actual source is sent to the host 
and it request the headers from the client as needed and the host caches them 
so they're only sent once. So pump mode uses less network traffic but it's not 
necessarily  faster, in fact it's much slower in some cases.

Some packages do custom preprocessing and other weird things during the build 
process that cause problems with pump mode since it caches copies of the 
unmodified headers. If you're lucky it just fails (and usually falls back on 
compiling locally), if you're not then it may succeed and you'll get runtime 
bugs. I haven't find a package yet that fails without pump mode as long as your 
CFLAGS are set properly.

As for updating, you likely don't need to. According to the distcc FAQ the 
minor version number is not as important. If you still want to, I *think* you 
can just mask the other versions using an atom like cross-i686-pc-mingw32/gcc 
to ensure that you get the version you want but I'm not sure about this and 
there may be a better way.

-- 
Fernando Rodriguez

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Walter Dnes]

On Sun, May 03, 2015 at 02:57:46PM -0400, Fernando Rodriguez wrote

> Some packages do custom preprocessing and other weird things during
> the build process that cause problems with pump mode since it caches
> copies of the unmodified headers. If you're lucky it just fails (and
> usually falls back on compiling locally), if you're not then it may
> succeed and you'll get runtime bugs. I haven't find a package yet
> that fails without pump mode as long as your CFLAGS are set properly.

  Seamonkey fails during the build process.  Two tries, and the build
log was 74,046 bytes each time.  I have an Intel x86_64 as the host, and
an Atom i686 (32-bit only) as the client.  Given your description, I may
drop "pump" altogether from my "xmerge" script.  I'll unmerge
seamonkey-bin, and try distcc-building seamonkey from source, without
"pump", Monday when I have more time.  Here are a few lines from the
failed build log, using "pump"...

Executing: gcc -o nsinstall_real -march=atom -mtune=atom -fstack-protector -pipe -mno-avx -DXP_UNIX -MD -MP -MF .deps/nsinstall_real.pp -O2 -DUNICODE -D_UNICODE -Wl,-O1 -Wl,--as-needed host_nsinstall.o host_pathsub.o

/usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: i386:x86-64 architecture of input file `host_nsinstall.o' is incompatible with i386 output

/usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: i386:x86-64 architecture of input file `host_pathsub.o' is incompatible with i386 output

/usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: host_nsinstall.o: file class ELFCLASS64 incompatible with ELFCLASS32

/usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: final link failed: File in wrong format

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Fernando Rodriguez]

On Sunday, May 03, 2015 11:59:08 PM Walter Dnes wrote:
> On Sun, May 03, 2015 at 02:57:46PM -0400, Fernando Rodriguez wrote
> 
> > Some packages do custom preprocessing and other weird things during
> > the build process that cause problems with pump mode since it caches
> > copies of the unmodified headers. If you're lucky it just fails (and
> > usually falls back on compiling locally), if you're not then it may
> > succeed and you'll get runtime bugs. I haven't find a package yet
> > that fails without pump mode as long as your CFLAGS are set properly.
> 
>   Seamonkey fails during the build process.  Two tries, and the build
> log was 74,046 bytes each time.  I have an Intel x86_64 as the host, and
> an Atom i686 (32-bit only) as the client.  Given your description, I may
> drop "pump" altogether from my "xmerge" script.  I'll unmerge
> seamonkey-bin, and try distcc-building seamonkey from source, without
> "pump", Monday when I have more time.  Here are a few lines from the
> failed build log, using "pump"...
> 
> Executing: gcc -o nsinstall_real -march=atom -mtune=atom -fstack-protector -
pipe -mno-avx -DXP_UNIX -MD -MP -MF .deps/nsinstall_real.pp -O2 -DUNICODE -
D_UNICODE -Wl,-O1 -Wl,--as-needed host_nsinstall.o host_pathsub.o
> 
> /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
i386:x86-64 architecture of input file `host_nsinstall.o' is incompatible with 
i386 output
> 
> /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
i386:x86-64 architecture of input file `host_pathsub.o' is incompatible with 
i386 output
> 
> /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
host_nsinstall.o: file class ELFCLASS64 incompatible with ELFCLASS32
> 
> /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
final link failed: File in wrong format
> 
> 

The error on the link that you posted looks like cause by pump mode, this one 
I'm not so sure.

It looks like you're not using the cross compiler on the host as it would not 
generate 64 bit code. Did you add -m32 to your CFLAGS on the client box? Also 
you may need to set the custom-cflags use flag. Can you verify that it is using 
the cross compiler on the host?

I'm not sure exactly what the gentoo recommended distcc/cross compile setup 
but if you do it like I suggested on your other thread (using the host 64bit 
compiler) it should work. Look at the links you got under /usr/lib/distcc/bin. 
All you need to do is create scripts on the host with the exact same names and 
have them execute the compiler that you want with the options you want (I just 
have it execute the 64bit compiler with -m32). Then make sure that distccd (on 
host) finds them before the actual compiler by putting it in the PATH 
environment variable before anything else. For that you may need to modify the 
init script or unit file if using systemd or just start distccd manually.

A simpler hack is to just delete the c++, cc, gcc, and g++ symlinks from the 
/usr/lib/distcc/bin directory. That will force distcc to only trap the 
compiler invocations that use the full compiler name and end up using the 
cross compiler in the host, but if you do this you may end up compiling more 
stuff locally.


-- 
Fernando Rodriguez

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Fernando Rodriguez]

On Monday, May 04, 2015 5:29:34 AM Fernando Rodriguez wrote:
> On Sunday, May 03, 2015 11:59:08 PM Walter Dnes wrote:
> > On Sun, May 03, 2015 at 02:57:46PM -0400, Fernando Rodriguez wrote
> > 
> > > Some packages do custom preprocessing and other weird things during
> > > the build process that cause problems with pump mode since it caches
> > > copies of the unmodified headers. If you're lucky it just fails (and
> > > usually falls back on compiling locally), if you're not then it may
> > > succeed and you'll get runtime bugs. I haven't find a package yet
> > > that fails without pump mode as long as your CFLAGS are set properly.
> > 
> >   Seamonkey fails during the build process.  Two tries, and the build
> > log was 74,046 bytes each time.  I have an Intel x86_64 as the host, and
> > an Atom i686 (32-bit only) as the client.  Given your description, I may
> > drop "pump" altogether from my "xmerge" script.  I'll unmerge
> > seamonkey-bin, and try distcc-building seamonkey from source, without
> > "pump", Monday when I have more time.  Here are a few lines from the
> > failed build log, using "pump"...
> > 
> > Executing: gcc -o nsinstall_real -march=atom -mtune=atom -fstack-protector 
-
> pipe -mno-avx -DXP_UNIX -MD -MP -MF .deps/nsinstall_real.pp -O2 -DUNICODE -
> D_UNICODE -Wl,-O1 -Wl,--as-needed host_nsinstall.o host_pathsub.o
> > 
> > /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
> i386:x86-64 architecture of input file `host_nsinstall.o' is incompatible 
with 
> i386 output
> > 
> > /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
> i386:x86-64 architecture of input file `host_pathsub.o' is incompatible with 
> i386 output
> > 
> > /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
> host_nsinstall.o: file class ELFCLASS64 incompatible with ELFCLASS32
> > 
> > /usr/lib/gcc/i686-pc-linux-gnu/4.8.4/../../../../i686-pc-linux-gnu/bin/ld: 
> final link failed: File in wrong format
> > 
> > 
> 
> The error on the link that you posted looks like cause by pump mode, this 
one 
> I'm not so sure.
> 
> It looks like you're not using the cross compiler on the host as it would 
not 
> generate 64 bit code. Did you add -m32 to your CFLAGS on the client box? 
Also 
> you may need to set the custom-cflags use flag. Can you verify that it is 
using 
> the cross compiler on the host?
> 
> I'm not sure exactly what the gentoo recommended distcc/cross compile setup 
> but if you do it like I suggested on your other thread (using the host 64bit 
> compiler) it should work. Look at the links you got under 
/usr/lib/distcc/bin. 
> All you need to do is create scripts on the host with the exact same names 
and 
> have them execute the compiler that you want with the options you want (I 
just 
> have it execute the 64bit compiler with -m32). Then make sure that distccd 
(on 
> host) finds them before the actual compiler by putting it in the PATH 
> environment variable before anything else. For that you may need to modify 
the 
> init script or unit file if using systemd or just start distccd manually.
> 
> A simpler hack is to just delete the c++, cc, gcc, and g++ symlinks from the 
> /usr/lib/distcc/bin directory. That will force distcc to only trap the 
> compiler invocations that use the full compiler name and end up using the 
> cross compiler in the host, but if you do this you may end up compiling more 
> stuff locally.

Or you just replace them (c++, cc, gcc, and g++) with a wrapper to make sure 
it invokes the full compiler name. That's what they recommend here: 
https://wiki.gentoo.org/wiki/Raspberry_Pi_Cross_building

-- 
Fernando Rodriguez

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Walter Dnes]

On Mon, May 04, 2015 at 05:29:34AM -0400, Fernando Rodriguez wrote

> The error on the link that you posted looks like cause by pump mode,
> this one I'm not so sure.
> 
> It looks like you're not using the cross compiler on the host as
> it would not generate 64 bit code. Did you add -m32 to your CFLAGS
> on the client box? Also you may need to set the custom-cflags use
> flag. Can you verify that it is using the cross compiler on the host?

  I did a large world update this past week.  If all the libs and
programs were going as 64-bit code, to my 32-bit-only machine, Gentoo
should be badly broken by now, to the point of being unbootable.  The
problem appears to be isolated to seamonkey.  I'll first try adding
"-m32".  If that doesn't work, I'll drop the "pump" option.  I'll let
you know how things turn out.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Walter Dnes]

On Mon, May 04, 2015 at 05:29:34AM -0400, Fernando Rodriguez wrote

> All you need to do is create scripts on the host with the exact
> same names and have them execute the compiler that you want with
> the options you want (I just have it execute the 64bit compiler with
> -m32). Then make sure that distccd (on host) finds them before the
> actual compiler by putting it in the PATH environment variable before
> anything else.

  Much to my surprise, adding "-m32" to the client's CFLAGS (and
therefore also CXXFLAGS) results in seamonkey building properly.  I
tried it out on the same video, and cpu load "only" climbs to 2.5 versus
2.75 with seamonkey-bin.  The build took 1hr and 43 minutes on the Core
2 Duo host, versus 14 hours doing it on the Atom.

  Why is seamonkey the only program (so far for me) that needs "-m32"?
Would it need "-m64" if it was being cross-compiled on a 32-bit host
system for 64-bit client?  Is there a wiki that we can contribute this
info to?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Fernando Rodriguez]

On Monday, May 04, 2015 3:41:54 PM Walter Dnes wrote:
>   Why is seamonkey the only program (so far for me) that needs "-m32"?
> Would it need "-m64" if it was being cross-compiled on a 32-bit host
> system for 64-bit client?  Is there a wiki that we can contribute this
> info to?

It has to do with my last post. Basicly the makefiles are invoking the full 
compiler name for the files that are meant to run on the target but it invokes 
just gcc for the files that are meant to run on the host. This is in the 
context of cross-compiling, not distcc, so the file that's failing is meant to 
run locally (it's a custom build tool). If you where compiling locally (or in 
two machines with the same compiler) it would not matter cause the host and 
target compiler are the same, but when distcc comes in it builds those files 
with the host (system) compiler on the host.

Changing the c++, cc, gcc, and g++ symlinks to a wrapper script that invokes 
the compiler by it's full name as show in the RaspberryPi wiki page *should* 
fix it.

-- 
Fernando Rodriguez

Re: [gentoo-user] Difference between "normal distcc" and "distcc with pump"?

[Fernando Rodriguez]

On Monday, May 04, 2015 4:36:08 PM Fernando Rodriguez wrote:
> On Monday, May 04, 2015 3:41:54 PM Walter Dnes wrote:
> >   Why is seamonkey the only program (so far for me) that needs "-m32"?
> > Would it need "-m64" if it was being cross-compiled on a 32-bit host
> > system for 64-bit client?  Is there a wiki that we can contribute this
> > info to?
> 
> It has to do with my last post. Basicly the makefiles are invoking the full 
> compiler name for the files that are meant to run on the target but it 
invokes 
> just gcc for the files that are meant to run on the host. This is in the 
> context of cross-compiling, not distcc, so the file that's failing is meant 
to 
> run locally (it's a custom build tool). If you where compiling locally (or 
in 
> two machines with the same compiler) it would not matter cause the host and 
> target compiler are the same, but when distcc comes in it builds those files 
> with the host (system) compiler on the host.
> 
> Changing the c++, cc, gcc, and g++ symlinks to a wrapper script that invokes 
> the compiler by it's full name as show in the RaspberryPi wiki page *should* 
> fix it.
> 

That sounds confusing cause I use host to mean distcc host at one point and 
host compiler on another. I will use "server" to refer to distcc host on this 
post. When you run a GNU standard configure script you can specify two 
compilers, the host and target compiler. When compiling locally they're both 
the same. But when cross-compiling the host is the system compiler and is used 
for compiling things that will be executed as part of the build process (most 
packages don't do this). On gentoo this is set from the CHOST variable on 
make.conf, but either it's not usually passed to configure scripts by portage 
or some scripts just ignore it and invoke the host compiler as cc, c++, g++, 
or gcc.

When cross-compiling the target compiler is the one for the target 
architecture where the package will be deployed to. This is always invoked by 
the full name (on GNU compliant packages).

Distcc just traps the compiler invocations on the client and perform the same 
invokations on the server. In your case the seamonkey is trying to compile 
something with the host compiler, distcc is trapping it and compiling it with 
the host compiler on the server. Since the host compiler in the server is not 
the same as the host compiler on the server things go bad.

So you don't need -m32 unless you want to use the host compiler on the server. 
Since you want to use a cross-compiler on the server that was an ugly hack 
because you're actually using both compilers on the server. If the version of 
the cross-compiler gets out of sync with the host compiler things can go bad 
easily. So the proper fix in your scenario is to get rid of the -m32 and make 
the host compiler links a wapper script so that everything is compiled with 
the cross-compiler on the server.

-- 
Fernando Rodriguez