[gentoo-amd64] -fPIC - Toolchain broken?

[gentoo-amd64] -fPIC - Toolchain broken?

[Sebastian Redl]

Hi,

While trying to compile OpenOffice.org, I was blocked by the inability
to compile icu-3.4.1.
The compilation fails on linking the third library it tries to build,
libicui18n.so, with this message:

/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
warning: creating a DT_TEXTREL in object.
/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
ucol_wgt.o: relocation R_X86_64_PC32 against `compareRanges' can not be
used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
final link failed: Bad value
collect2: ld returned 1 exit status

Normally, that would simply indicate that a source file was compiled
without -fPIC, which is wrong. However, that's not the case here: manual
checking of the compiler command lines of all previous sources shows
that they were all, in fact, compiled with -fPIC and -DPIC.

I even manually removed the previously built libraries (libicudata.so
and libicuuc.so) and relinked them with -fPIC, although I don't even
know whether the flag has any influence on linking. Same error.

I have the same problem when manually trying to compile any
Mozilla-based application. However, most programs and libraries build
fine - I just finished merging KOffice without any problems.

So, is my toolchain screwed up? Am I doing something wrong? Is the icu
ebuild simply broken?

Here's my emerge --info:

Portage 2.1.1-r1 (default-linux/amd64/2006.1/desktop, gcc-4.1.1,
glibc-2.4-r3, 2
.6.14-gentoo-r5 x86_64)
=================================================================
System uname: 2.6.14-gentoo-r5 x86_64 AMD Athlon(tm) 64 Processor 3200+
Gentoo Base System version 1.12.5
Last Sync: Tue, 17 Oct 2006 19:59:01 +0000
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: 1.3.7, 2.0.30
dev-lang/python:     2.3.5-r2, 2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     [Not Present]
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.59-r7
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.13-r4
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O0 -pipe -march=athlon64"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf
/etc/java-config/vms/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-O0 -pipe -march=athlon64"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks metadata-transfer parallel-fetch sandbox
sfperms strict userfetch"
GENTOO_MIRRORS="http://www.distfiles.local http://distfiles.gentoo.org
http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LANG="en_US.utf8"
LC_ALL="en_US.utf8"
LINGUAS="en en_GB de de_AT"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times
--compress --force --whole-file --delete --delete-after --stats
--timeout=180 --exclude='/distfiles' --exclude='/local'
--exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/overlays/personal
/usr/local/overlays/gentoo-java-experimental /usr/local/overlays/mozilla"
SYNC="rsync://192.168.1.1/gentoo-portage"
USE="amd64 X acl acpi alsa apache2 avi bash-completion berkdb
bitmap-fonts bzip2 cairo cdparanoia cdr cli cracklib crypt css cups dbus
dlloader dri dvd dvdr dvdread elibc_glibc emboss encode fam firefox flac
foomatic fortran gdbm gif gpm gstreamer gtk2 hal imap
input_devices_evdev input_devices_joystick input_devices_keyboard
input_devices_mouse ipv6 isdnlog jack java jpeg jpeg2k kde
kdeenablefinal kdehiddenvisibility kernel_linux libg++ linguas_de
linguas_de_AT linguas_en linguas_en_GB logitech-mouse mad mikmod mng
mono mozilla mp3 mpeg mysql mysqli ncurses nls nptl nptlonly offensive
ogg oggvorbis openal opengl pam pcre perl png ppds pppd python qt qt3
qt4 quicktime readline reflection samba sdl session sndfile soundtouch
spell spl sqlite ssl svg tcpd theora tiff truetype truetype-fonts
type1-fonts udev unicode usb userland_GNU video_cards_fglrx
video_cards_radeon vorbis wmf xine xinerama xml xorg xosd xpm xprint
xscreensaver xv xvid zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS,
PORTAGE_RSYNC_EXTRA_OPTS

The -O0 was a temporary thing that I need to get rid of; most of my
system is -O3 -fomit-frame-pointer.

Thanks,
Sebastian

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] -fPIC - Toolchain broken?

[Hemmann, Volker Armin]

On Wednesday 18 October 2006 17:08, Sebastian Redl wrote:


>
> So, is my toolchain screwed up? Am I doing something wrong? Is the icu
> ebuild simply broken?
>

> CFLAGS="-O0 -pipe -march=athlon64"

maybe that is part of your problem? -O0? NO optimiziation? that is pretty 
untested, you know... 
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] -fPIC - Toolchain broken?

[Simon Stelling]

Sebastian Redl wrote:
> Hi,
	>
> While trying to compile OpenOffice.org, I was blocked by the inability
> to compile icu-3.4.1.
> The compilation fails on linking the third library it tries to build,
> libicui18n.so, with this message:
> 
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> warning: creating a DT_TEXTREL in object.
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> ucol_wgt.o: relocation R_X86_64_PC32 against `compareRanges' can not be
> used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> final link failed: Bad value
> collect2: ld returned 1 exit status

It worked fine on my working system (CFLAGS="-O2 -march=opteron -pipe"), 
but failed the same way in my development chroot (CFLAGS="-ggdb 
-march=k8 -pipe").

I didn't look at the code yet, but I guess it has some #ifndef 
__OPTIMIZED__. __OPTIMIZED__ is set by gcc with -O2 and above. I guess 
if you switch back to your -O3 CFLAGS it will compile.

> I even manually removed the previously built libraries (libicudata.so
> and libicuuc.so) and relinked them with -fPIC, although I don't even
> know whether the flag has any influence on linking. Same error.

-fPIC only affects compilation

> I have the same problem when manually trying to compile any
> Mozilla-based application. However, most programs and libraries build
> fine - I just finished merging KOffice without any problems.

> 
> So, is my toolchain screwed up? Am I doing something wrong? Is the icu
> ebuild simply broken?

I have exactly the same versions of binutils in both working and 
development system and never had problems, so I'd say all is fine on 
your side. We'll have to fix the ebuild I guess.

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] -fPIC - Toolchain broken?

[Simon Stelling]

Simon Stelling wrote:
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld: 
>>
>> warning: creating a DT_TEXTREL in object.
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld: 
>>
>> ucol_wgt.o: relocation R_X86_64_PC32 against `compareRanges' can not be
>> used when making a shared object; recompile with -fPIC
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld: 
>>
>> final link failed: Bad value
>> collect2: ld returned 1 exit status
> 
> I didn't look at the code yet, but I guess it has some #ifndef 
> __OPTIMIZED__. __OPTIMIZED__ is set by gcc with -O2 and above. I guess 
> if you switch back to your -O3 CFLAGS it will compile.

Actually, it would be __OPTIMIZE__, but that wasn't the culprit. The 
function compareRanges is declared as follows:

static U_INLINE int32_t U_CALLCONV
compareRanges(const void *context, const void *left, const void *right);

And in another function used as an argument to a third function, i.e. 
pointed at with a pointer, but as it's inline you can't point at it.

Now the only thing left is to find out why the heck it fails with 
-O{0,1} but not >= -02...

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] -fPIC - Toolchain broken?

[Sebastian Redl]

Simon Stelling wrote:
> I didn't look at the code yet, but I guess it has some #ifndef
> __OPTIMIZED__. __OPTIMIZED__ is set by gcc with -O2 and above. I guess
> if you switch back to your -O3 CFLAGS it will compile.
Worked with -O3.

You know, they say optimization can cause problems ... ;-)

Anyway, just so I understand: the function is inline/static (don't know
which of these causes the problem) which causes compareRanges to be
exported, but not compiled position-independently. The linker, seeing
that, fails. But with -O>1, it somehow doesn't actually export the
function or something like that, and thus it works?

Sebsatian Redl
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] -fPIC - Toolchain broken?

[Simon Stelling]

Sebastian Redl wrote:
> Anyway, just so I understand: the function is inline/static (don't know
> which of these causes the problem) which causes compareRanges to be
> exported, but not compiled position-independently. The linker, seeing
> that, fails. But with -O>1, it somehow doesn't actually export the
> function or something like that, and thus it works?

I didn't dig any deeper then what I have written already for icu as we 
have another test case provided in bug 151832. Basically gcc shouldn't 
inline the function or at least provide a callable symbol because its 
address is used later on (in the same file, even), which is what we 
found in the gcc documentation but not in the code ;)

So yes, it IS a toolchain-bug, but one the gcc-4.1 users share (gcc 3.4 
works correctly apparently) :D

[1] http://bugs.gentoo.org/show_bug.cgi?id=151832

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: -fPIC - Toolchain broken?

[Duncan]

Sebastian Redl <sebastian.redl@getdesigned.at> posted
45364363.3010601@getdesigned.at, excerpted below, on  Wed, 18 Oct 2006
17:08:19 +0200:

> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> warning: creating a DT_TEXTREL in object.
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> ucol_wgt.o: relocation R_X86_64_PC32 against `compareRanges' can not be
> used when making a shared object; recompile with -fPIC
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.1/../../../../x86_64-pc-linux-gnu/bin/ld:
> final link failed: Bad value
> collect2: ld returned 1 exit status
> 
> Normally, that would simply indicate that a source file was compiled
> without -fPIC, which is wrong. However, that's not the case here: manual
> checking of the compiler command lines of all previous sources shows that
> they were all, in fact, compiled with -fPIC and -DPIC.
> 
> I even manually removed the previously built libraries (libicudata.so and
> libicuuc.so) and relinked them with -fPIC, although I don't even know
> whether the flag has any influence on linking. Same error.
> 
> I have the same problem when manually trying to compile any Mozilla-based
> application. However, most programs and libraries build fine - I just
> finished merging KOffice without any problems.

I see the specifics of this are being investigated, but here's some
general comments, which hopefully prove enlightening.  =8^)  I think I
wrote about this a few days ago, but for those that may have missed it...

-fPIC is an interesting beast.  It's required on amd64 for all shared
objects (*.so*), but isn't recommended (except for hardened) for
application binaries, both due to occasional unexpected complications and
because it's slower.  Thus, putting it in CFLAGS is NOT a good thing,
unless it's for a specific library package only, to get it to merge while
you are waiting for the bug you filed on it (right?) to get
researched/fixed/tested.

HOWEVER, and a big HOWEVER this can be, just because the error says -fPIC
doesn't NECESSARILY mean that's got anything to do with the real problem. 
The situation is often the following.  During the configure step, some of
the various tests involve compiling tiny stub programs and seeing if they
compile correctly with the option being tested.  The problem is that
sometimes what is being tested doesn't always return a gcc error, only a
gcc warning.  If that's the case, the test often simply checks for a
warning, not WHAT the warning was.  Apparently, the -fPIC test is one such
test-for-warning-only test, so if gcc throws a warning of some kind for
ANY reason during that test, it'll often cause the configure script to
believe that -fPIC doesn't work on the platform, when in fact on our
platform it's required, for libraries/shared-objects anyway!  Obviously,
this WILL cause problems.

One of the most common causes here is warning triggering CFLAGS, such as
those that might have been configured for a different version of gcc, that
the current version doesn't recognize.  In fact, this is /so/ common a
problem, that the Gentoo amd64 team came up with a normally workable if
not 100% solution -- use the profile bashrc script to test for
unrecognized flags and eliminate them automatically.  This apparently
reduced the false bug reports due to stale CFLAGS substantially, but the
script is just that, and doesn't test all possible cases as doing so
would be "complex", and besides, there are other ways to trigger a warning
as well.

So... anytime I see this error, the first thing I try is flipping some
CFLAGS on and off, and see if there's a reasonable combination that works.
 Only if that fails do I try -fPIC and bug it as necessary.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: -fPIC - Toolchain broken?

[Sebastian Redl]

Duncan wrote:
> So... anytime I see this error, the first thing I try is flipping some
> CFLAGS on and off, and see if there's a reasonable combination that works.
>  Only if that fails do I try -fPIC and bug it as necessary.
>   
But wouldn't that apply only to errors that occur during the
configuration step? Or have you observed a situation where a compilation
change triggered by a failing non-essential test caused an error later?
> -fPIC is an interesting beast.  It's required on amd64 for all shared
> objects (*.so*), but isn't recommended (except for hardened) for
> application binaries, both due to occasional unexpected complications and
> because it's slower.  Thus, putting it in CFLAGS is NOT a good thing,
> unless it's for a specific library package only, to get it to merge while
> you are waiting for the bug you filed on it (right?) to get
> researched/fixed/tested.
>   
Speaking of which, is there any way to configure flags per package?
Something like /etc/portage/package.flags, where I can configure CFLAGS,
CXXFLAGS and LDFLAGS?

Sebastian Redl
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: -fPIC - Toolchain broken?

[Conway S. Smith]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Thu, 19 Oct 2006 11:54:47 +0200
Sebastian Redl <sebastian.redl@getdesigned.at> wrote:
<snip>
> Speaking of which, is there any way to configure flags per package?
> Something like /etc/portage/package.flags, where I can configure
> CFLAGS, CXXFLAGS and LDFLAGS?
> 

Yes, you can use /etc/portage/env/<category>/<package>[-<version>]; for
example, I have a /etc/portage/env/app-admin/logrotate-3.7.2 file,
without -combine in the C[XX]FLAGS, as it won't compile with -combine &
I have -combine in make.conf.  I could instead have
/etc/portage/env/app-admin/logrotate, but if a future version of
logrotate worked w/ -combine, it wouldn't use it unless I removed or
edited /etc/portage/env/app-admin/logrotate.  Also note that any CFLAGS
or CXXFLAGS set in these files completely replace those in make.conf,
they are not added to those in make.conf.


Conway S. Smith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-amd64] Re: -fPIC - Toolchain broken?

[Simon Stelling]

Conway S. Smith wrote:
> Also note that any CFLAGS
> or CXXFLAGS set in these files completely replace those in make.conf,
> they are not added to those in make.conf.

If you want to add instead of replace, CFLAGS="${CFLAGS} -fadded" will 
do the job. Removing is done with CFLAGS=" ${CFLAGS}" ; 
CFLAGS=${CFLAGS//-fremoved}

So you can do effectively anything, given you know the bash tricks ;)

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: -fPIC - Toolchain broken?

[Duncan]

Simon Stelling <blubb@gentoo.org> posted 45379CDC.1010504@gentoo.org,
excerpted below, on  Thu, 19 Oct 2006 17:42:20 +0200:

> If you want to add instead of replace, CFLAGS="${CFLAGS} -fadded" will do
> the job. Removing is done with CFLAGS=" ${CFLAGS}" ;
> CFLAGS=${CFLAGS//-fremoved}
> 
> So you can do effectively anything, given you know the bash tricks ;)

Hey, thanks!  The add was a no-brainer here, but I hadn't thought about the
remove trick yet.  Very nice! =8^)

Will those tricks work in make.conf directly, say to add or subtract from
CFLAGS when assigning CXXFLAGS?  I had at first thought not, but thinking
again, perhaps so, if python takes it literally but never uses it except
by passing it to bash which immediately expands it as appropriate.

I suppose I could test, but it'd be after some sleep and if you know in
the mean time...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: -fPIC - Toolchain broken?

[Simon Stelling]

Duncan wrote:
> Will those tricks work in make.conf directly, say to add or subtract from
> CFLAGS when assigning CXXFLAGS?  I had at first thought not, but thinking
> again, perhaps so, if python takes it literally but never uses it except
> by passing it to bash which immediately expands it as appropriate.

Don't think it will work, but I never tried it. Portage uses the 
completely strange python shlex to parse make.conf which looks like sh 
but that's about it ;)

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: -fPIC - Toolchain broken?

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

On Thu, 19 Oct 2006 20:43:28 +0000 (UTC)
"Duncan" <1i5t5.duncan@cox.net> wrote:

> Simon Stelling <blubb@gentoo.org> posted 45379CDC.1010504@gentoo.org,
> excerpted below, on  Thu, 19 Oct 2006 17:42:20 +0200:
> 
> > If you want to add instead of replace, CFLAGS="${CFLAGS} -fadded"
> > will do the job. Removing is done with CFLAGS=" ${CFLAGS}" ;
> > CFLAGS=${CFLAGS//-fremoved}

FWIW I think Simon meant:

CFLAGS=" ${CFLAGS}" ; CFLAGS=${CFLAGS// -fremoved}

assuming that adding the space was done to catch flags that match part
of longer flags.

> > So you can do effectively anything, given you know the bash
> > tricks ;)
> 
> Hey, thanks!  The add was a no-brainer here, but I hadn't thought
> about the remove trick yet.  Very nice! =8^)

Watch also for flags that start with the same string; for example:

  CFLAGS=${CFLAGS// -finline-functions}

would end up with "-called-once" in CFLAGS, if
"-finline-functions-called-once" was set.

Another easy one is doing:

  CFLAGS=${CFLAGS// -g}

when CFLAGS has "-ggdb2" or similar set...

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-amd64] Re: -fPIC - Toolchain broken?

[Duncan]

"Kevin F. Quinn" <kevquinn@gentoo.org> posted
20061023162050.48b4e70f@c1358217.kevquinn.com, excerpted below, on  Mon,
23 Oct 2006 16:20:50 +0200:

> FWIW I think Simon meant:
> 
> CFLAGS=" ${CFLAGS}" ; CFLAGS=${CFLAGS// -fremoved}
> 
> assuming that adding the space was done to catch flags that match part
> of longer flags.

What about adding a space at each end, then testing for a space at each
end of the removed,, but then you have to be sure and add one space back
in so as not to concatenate two flags into one with the removal, so...

CFLAGS=" ${CFLAGS} " ; CFLAGS=${CFLAGS// -fremoved/ /}

If I'm not mistaken, that should eliminate the partial flag matches you
mentioned without causing other issues.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: -fPIC - Toolchain broken?

[Duncan]

Sebastian Redl <sebastian.redl@getdesigned.at> posted
45374B67.6010901@getdesigned.at, excerpted below, on  Thu, 19 Oct 2006
11:54:47 +0200:

> Duncan wrote:
>> So... anytime I see this error, the first thing I try is flipping some
>> CFLAGS on and off, and see if there's a reasonable combination that
>> works.
>>  Only if that fails do I try -fPIC and bug it as necessary.
>>   
> But wouldn't that apply only to errors that occur during the configuration
> step? Or have you observed a situation where a compilation change
> triggered by a failing non-essential test caused an error later?

Yes.  Case in point is the very -fPIC we are discussing.  If the configure
tests -fPIC and due to a warning decides it can't be used, it doesn't
simply abort, but continues the process thru the rest of the config and
into the compile and linking.  Only later in the linking, and gets an error
similar to that of this thread because shared objects require -fPIC on
this platform, does it realize things went wrong.  Then the error it spits
out says to compile with -fPIC, when that's what it would have been doing
if the config hadn't misinterpreted an unrelated warning on the -fPIC test
as an indication that it shouldn't be used.

(The other half of your post addressed in the other subthread.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list