* [gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
@ 2012-10-06 2:50 Gregory M. Turner
2012-10-06 7:47 ` [gentoo-dev] " Gregory M. Turner
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-06 2:50 UTC (permalink / raw
To: gentoo-dev
----[ "eclass/flag-o-matic.eclass" ]----->8----->
--- PORTAGE/eclass/flag-o-matic.eclass
+++ OVERLAY/eclass/flag-o-matic.eclass
@@ -117,6 +117,42 @@
return 0
}
+# @FUNCTION: prepend-ldpath
+# @USAGE: <path>
+# @DESCRIPTION:
+# Place the specified ldpath into LDFLAGS before any options which could
+# add additional paths to ld's search path. Specifically, it will place
+# the new <path> "/foo" into LDFLAGS as "-L/foo" just before the first
+# occurance matching any of the globs: '-L*', '-T', '--library-path*',
+# and '--script*', but not matching any of the globs: '-Tbss=*',
+# '-Tdata=*', '-Ttext=*', and '-Ttext-segment=*'. If no such match is
+# found, then this is equivalent to "append-ldflags -L<path>".
+prepend-ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ else
+ new+=( "-L${1}" "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L${1}" )
+ export LDFLAGS="${new[*]}"
+}
+
# @FUNCTION: filter-lfs-flags
# @DESCRIPTION:
# Remove flags that enable Large File Support.
<-----8<-----
I think my code is probably fine, or if it's buggy, so be it. A prior
question is: does this have sufficient utility?
I "need" (not exactly... keep reading) something like this for python.
Currently python*.ebuild contains:
append-ldflags "-L."
which is required in order to pull in the newly-rebuilt
libpython${VERSION}.so at ebuild-time, instead of the system version
(perhaps it's only so on obscure platforms?).
The problem is, LDFLAGS may already have some library paths coming in
from the environment. In this case, we end up with something like:
LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."
and python goes ahead links everything against, i.e.:
/random/prefix/usr/lib/libpython3.2.so,
or whatever is the version in question, so we achieve precisely nothing,
effecting the outcome we are trying to prevent (the problem manifests in
subtle, hard-to-diagnose ways, unfortunately).
In general, when we say append-ldflags -Lsomething, we may or may not
mean "always use something/libfoo, not $(libdir)/libfoo to link." In
the case where we do mean "always", append-ldflags is a broken approach.
An alternative would be to just replace
append-ldflags "-L."
with
export LDFLAGS="-L. ${LDFLAGS}".
however, there are problems with this:
First, it's aesthetically annoying that this spreads out the "-L*"
arguments into two different "zones" in LDFLAGS. No big deal, but it
definitely doesn't make visually scanning the logs any easier.
Second, although I'm not aware of any ld arguments which act as
"modifiers" to "-L" arguments (except the other -L arguments and
ldscript arguments, which the above patch scans for), ld does use
order-dependent patterns for other arguments.
For example: "-lfoo -( -lbar -lbaz -)" is not the same thing as "-(
-lfoo -lbar -) -lbaz". So order-dependencies might emerge in some
future binutils... in that case we'd still probably need to extend the
case statement above to include the new order-affecting arguments, but
at least we'd have a place in which to do so.
Third, although the meaning of -L* options may not be affected by other
arguments in an order-dependant manner, the reverse is not so. For
example, see --mri-script, --script (which both can affect and be
affected by the command-line library search path in an order-dependent
manner!), -rpath-link (SunOS only), etc...
One could certainly argue that, in practice, points two and three amount
to hopelessly obscure nitpicking and that nobody in their right mind
should be relying on this type of stuff in their LDFLAGS.
I'm not sure I'd state it quite so strongly as that, (after all these
might come just from profile.bashrc and be targeted to a particular
ebuild) but justification is clearly on the "thin" side.
Indeed, if we're going to worry about side effects, it's not entirely
clear that what what my patch does is safe. For example, if the
environment supplied "-L/foo/bar --script baz", and python for some
reason had a "baz" file in "${S}", then some kind of breakage would
surely happen when we did
prepend-ldpath "."
Also, we might also legitimately worry that the presence of this
function in flag-o-matic will somehow "encourage" people to do this
more, and, I'd have to agree that it's a pretty shitty approach to
getting gcc to find your generated libraries (perhaps I could answer
this concern by means of some kind of "don't use unless you must"
disclaimer, or by grafting this code into python.eclass during configure()?)
Gross as it may be, it's the approach we've adopted in dev-lang/python*,
a critically important, mandatory package, and it's broken. It's
definitely not just an academic problem, at least not for my
cygwin-overlay where it makes it impossible to toggle USE=threads for
this package. Presumably the same goes for BSD and any other platforms
relying on
So we either need to fix the tactics or the strategy -- personally I'm a
bit reluctant to mess with the latter, which, I guess, is how I end up
with the above, despite some misgivings.
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-06 2:50 [gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath Gregory M. Turner
@ 2012-10-06 7:47 ` Gregory M. Turner
2012-10-06 8:31 ` Fabian Groffen
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-06 7:47 UTC (permalink / raw
To: gentoo-dev
My god, I am a horrible self-editor. Sorry. Please ignore the magnum
opus above and allow me to try again.
In dev-lang/python*, we use
append-ldflags '-L.'
to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir). But, this doesn't work if
LDFLAGS contains "-L$(libdir)".
We could try to fix this like:
export LDFLAGS="-L. ${LDFLAGS}"
or so. That would cover 99.9% of the cases out there. But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.
The flag-o-matic patch in my previous post analyses LDPATH and finds the
first statement that might possibly cause the ld command-line
library-search-path to be impacted, and inserts the "-L." just before
that statement, hopefully achieving the desired result with the least
possible side-effects.
I wonder: do people feel this distinction is meaningful enough to
justify actually making it? Or am I just being an anal-retentive nut-job?
If you tried to read my first post, thanks for not offing yourself like
the guys who get stuck sitting next to Ted Striker in _Airplane!_
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-06 7:47 ` [gentoo-dev] " Gregory M. Turner
@ 2012-10-06 8:31 ` Fabian Groffen
2012-10-10 22:47 ` Gregory M. Turner
2012-10-06 15:13 ` Duncan
2012-10-09 21:26 ` Mike Frysinger
2 siblings, 1 reply; 20+ messages in thread
From: Fabian Groffen @ 2012-10-06 8:31 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]
On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
> My god, I am a horrible self-editor. Sorry. Please ignore the magnum
> opus above and allow me to try again.
>
> In dev-lang/python*, we use
>
> append-ldflags '-L.'
>
> to ensure linking is performed against the built libpython.so in-tree,
> rather than than in the one in $(libdir). But, this doesn't work if
> LDFLAGS contains "-L$(libdir)".
>
> We could try to fix this like:
>
> export LDFLAGS="-L. ${LDFLAGS}"
>
> or so. That would cover 99.9% of the cases out there. But very rarely,
> indiscriminately placing our '-L.' before every other clause in LDFLAGS
> might cause an unanticipated side-effect.
>
> The flag-o-matic patch in my previous post analyses LDPATH and finds the
> first statement that might possibly cause the ld command-line
> library-search-path to be impacted, and inserts the "-L." just before
> that statement, hopefully achieving the desired result with the least
> possible side-effects.
I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it. I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.
--
Fabian Groffen
Gentoo on a different level
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-06 8:31 ` Fabian Groffen
@ 2012-10-10 22:47 ` Gregory M. Turner
2012-10-11 3:28 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-10 22:47 UTC (permalink / raw
To: gentoo-dev
On 10/6/2012 1:31 AM, Fabian Groffen wrote:
> On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
>> In dev-lang/python*, we use
>>
>> append-ldflags '-L.'
>>
>> to ensure linking is performed against the built libpython.so in-tree,
>> rather than than in the one in $(libdir). But, this doesn't work if
>> LDFLAGS contains "-L$(libdir)".
>>
>> We could try to fix this like:
>>
>> export LDFLAGS="-L. ${LDFLAGS}"
>>
>> or so. That would cover 99.9% of the cases out there. But very rarely,
>> indiscriminately placing our '-L.' before every other clause in LDFLAGS
>> might cause an unanticipated side-effect.
> I think it would make more sense in this case to just add one more patch
> to Python, such that the build-system just inserts it. I recall it's
> necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
> more why it works/worked on Linux fine.
I was thinking along these lines as well. As it turned out, however, I
was not able to find an automagical, PMS-compliant, non-crazy way to do
it that resolved my test-case (more on that below), without effectively
hooking econf().
Literally hooking a core API seems like a Pandora's box better left
unopened, so this is what I came up with (note: the fpectl stuff was
also in my overlay and the patches got entangled -- rather than editing
the hunk by hand, I just left it. It's orthogonal to this issue,
however, and for present purposes can be ignored):
------>8----->
--- PORTAGE/dev-lang/python/python-2.7.3-r2.ebuild
+++ OVERLAY/dev-lang/python/python-2.7.3-r2.ebuild
@@ -218,13 +255,6 @@
# http://bugs.python.org/issue15506
export ac_cv_path_PKG_CONFIG=$(tc-getPKG_CONFIG)
- # Set LDFLAGS so we link modules with -lpython2.7 correctly.
- # Needed on FreeBSD unless Python 2.7 is already installed.
- # Please query BSD team before removing this!
- # On AIX this is not needed, but would record '.' as runpath.
- [[ ${CHOST} == *-aix* ]] ||
- append-ldflags "-L."
-
local dbmliborder
if use gdbm; then
dbmliborder+="${dbmliborder:+:}gdbm"
@@ -248,11 +278,18 @@
&& myconf="${myconf} --enable-framework=${EPREFIX}/usr/lib" \
|| myconf="${myconf} --enable-shared"
+ if [[ ${CHOST} == *-cygwin* ]] ; then
+ fpeconfig="--without-fpectl"
+ myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"
+ else
+ fpeconfig="--with-fpectl"
+ fi
+
# note: for a framework build we need to use ucs2 because OSX
# uses that internally too:
# http://bugs.python.org/issue763708
- OPT="" econf \
- --with-fpectl \
+ OPT="" cpython_econf \
+ ${fpeconfig} \
$(use_enable ipv6) \
$(use_with threads) \
$( (use wide-unicode && use !aqua) && echo "--enable-unicode=ucs4"
|| echo "--enable-unicode=ucs2") \
--- PORTAGE/eclass/python.eclass
+++ OVERLAY/eclass/python.eclass
@@ -401,6 +401,64 @@
fi
}
+# see http://thread.gmane.org/gmane.linux.gentoo.devel/80633/focus=80635
+_python_prepend_cwd_ldpath() {
+ local new=()
+ local f
+ local done=no
+ for f in ${LDFLAGS} ; do
+ case "${f}" in
+ -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+ new+=( "${f}" )
+ ;;
+ -L*|-T*|--library-path*|--script*)
+ if [[ ${done} == yes ]] ; then
+ new+=( "${f}" )
+ elif [[ "${f}" == "-L." ]] ; then
+ # if it's somehow already there, don't duplicate it
+ new+=( "-L." )
+ done=yes
+ else
+ new+=( "-L." "${f}" )
+ done=yes
+ fi
+ ;;
+ *)
+ new+=( "${f}" )
+ ;;
+ esac
+ done
+ [[ ${done} == no ]] && new+=( "-L." )
+ export LDFLAGS="${new[*]}"
+}
+
+# @FUNCTION: cpython_econf
+# @DESCRIPTION:
+# econf() substitute for use in dev-lang/python ebuilds
+#
+# On some platforms, it is neccesary to prepend "-L." to ldpath before
+# proceeding with python's configure process. Using cpython_econf()
+# instead of econf() will ensure that this is taken care of correctly
+# before python's configure step can proceed. This is not needed for
+# any python ebuilds other than dev-lang/python; any other ebuild
+# calling this function will receive an error.
+cpython_econf() {
+ if [[ "${CATEGORY}/${PN}" != "dev-lang/python" ]] ; then
+ die "cpython_econf can only be used by dev-lang/python ebuilds"
+ fi
+ # econf will enforce ${EBUILD_PHASE} requirements, so we don't bother.
+
+ # Set LDFLAGS so we link modules with -lpython2.7 correctly.
+ # Needed on FreeBSD unless Python 2.7 is already installed.
+ # Even if python is already installed, linking against the old
+ # python will cause problems, i.e., when toggling USE="threads".
+ # Also needed on cygwin. Please query BSD team before removing this!
+ # On AIX this is not needed, and would record '.' as runpath.
+ [[ ${CHOST} == *-aix* ]] || _python_prepend_cwd_ldpath
+
+ econf "$@"
+}
+
# @FUNCTION: python_pkg_setup
# @DESCRIPTION:
# Perform sanity checks and initialize environment.
<-----8<-----
Some notes about the above:
Of course we would presumably want to make a similar change in all of
the dev-lang/python ebuilds to the one above.
We could do this 100% automagically in python_pkg_setup, which was my
initial plan. But my test-case involves LDPATH being "scrubbed" in
every phase by profile.bashrc. After python_pkg_setup() changed LDPATH,
it just broke again in subsequent phases (at least, I think that's what
would have happened -- I never tested it). Yes, that's a GIGO problem;
however, it's debatable whether Gentoo or profile.bashrc is the one
putting the "G)arbage I)n". The most robust route is to insert our
LDFLAG just before econf runs -- that's how the current implementation
does it, after all, and we potentially create a regression if we move
this logic to pkg_setup.
Also, I decided to leave my internal API as "_python_foo," although
today, I'd say "__python_foo" is more correct. python.eclass is chock
full of "_python_foo" API's, so I decided it was probably cleaner to
follow the existing patterns for now, rather than change one lone API
while the rest remain unchanged.
As for cpython_econf -- is it OK that it doesn't begin with "python"?
Or should it perhaps be "python_cpython_econf?"
Thanks for your input,
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-06 7:47 ` [gentoo-dev] " Gregory M. Turner
2012-10-06 8:31 ` Fabian Groffen
@ 2012-10-06 15:13 ` Duncan
2012-10-09 21:26 ` Mike Frysinger
2 siblings, 0 replies; 20+ messages in thread
From: Duncan @ 2012-10-06 15:13 UTC (permalink / raw
To: gentoo-dev
Gregory M. Turner posted on Sat, 06 Oct 2012 00:47:57 -0700 as excerpted:
> If you tried to read my first post, thanks for not offing yourself like
> the guys who get stuck sitting next to Ted Striker in _Airplane!_
It was worth it just for this:
>> LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."
=:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-06 7:47 ` [gentoo-dev] " Gregory M. Turner
2012-10-06 8:31 ` Fabian Groffen
2012-10-06 15:13 ` Duncan
@ 2012-10-09 21:26 ` Mike Frysinger
2012-10-11 3:37 ` Gregory M. Turner
2 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2012-10-09 21:26 UTC (permalink / raw
To: gentoo-dev; +Cc: Gregory M. Turner
[-- Attachment #1: Type: Text/Plain, Size: 1362 bytes --]
On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
> My god, I am a horrible self-editor. Sorry. Please ignore the magnum
> opus above and allow me to try again.
>
> In dev-lang/python*, we use
>
> append-ldflags '-L.'
>
> to ensure linking is performed against the built libpython.so in-tree,
> rather than than in the one in $(libdir). But, this doesn't work if
> LDFLAGS contains "-L$(libdir)".
well, some notes here ...
for users: putting -L<some-system-path> is wrong. it changes the meaning of
"system paths" when it comes to searching for linking by elevating it to "user
specified path". imo, you should be fixing the source rather than the symptom.
for packagers: using -L$(libdir) is almost always wrong and pointless. the
toolchain itself knows the proper system path to search for libraries to link
against, and when you cross-compile, the --libdir you specify to configure is
not going to be valid when dereferenced on the build system.
> We could try to fix this like:
>
> export LDFLAGS="-L. ${LDFLAGS}"
>
> or so. That would cover 99.9% of the cases out there. But very rarely,
> indiscriminately placing our '-L.' before every other clause in LDFLAGS
> might cause an unanticipated side-effect.
i would go with this if you want to support these people doing it wrong
-mike
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-09 21:26 ` Mike Frysinger
@ 2012-10-11 3:37 ` Gregory M. Turner
2012-10-11 3:50 ` Diego Elio Pettenò
2012-10-11 4:14 ` Mike Frysinger
0 siblings, 2 replies; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-11 3:37 UTC (permalink / raw
To: gentoo-dev
On 10/9/2012 2:26 PM, Mike Frysinger wrote:
> On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
>> My god, I am a horrible self-editor. Sorry. Please ignore the magnum
>> opus above and allow me to try again.
>>
>> In dev-lang/python*, we use
>>
>> append-ldflags '-L.'
>>
>> to ensure linking is performed against the built libpython.so in-tree,
>> rather than than in the one in $(libdir). But, this doesn't work if
>> LDFLAGS contains "-L$(libdir)".
>
> well, some notes here ...
>
> for users: putting -L<some-system-path> is wrong. it changes the meaning of
> "system paths" when it comes to searching for linking by elevating it to "user
> specified path". imo, you should be fixing the source rather than the symptom.
>
> for packagers: using -L$(libdir) is almost always wrong and pointless. the
> toolchain itself knows the proper system path to search for libraries to link
> against, and when you cross-compile, the --libdir you specify to configure is
> not going to be valid when dereferenced on the build system.
Agreed on these points. Not until we combine the doubly broken
circumstances of (1) User specified -L/usr/$(libdir) in LDPATH and (2)
an ebuild that puts -L. into LDPATH to fix Makefile recipes that appear
more-or-less broken, do we get this conflict. This leads to a certain
odor of "unsupportable circumstances" which, I must admit, makes me
slightly uncomfortable trying to fix it.
Breaking this down:
(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts. I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler. These cases
don't really seem completely "correct" or "incorrect" -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.
As for (2)... meh. LDFLAGS, to my thinking, means something like: "Use
these guidelines and parameters for linking, in preference to the
default guidelines that the build process would normally use." LDFLAGS
coming from make.conf have a slightly different meaning, I guess, but
the gist of it is the same.
So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
complete abomination. Still... I don't like it.
If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.
In the particular case of Gentoo's cpython, I have to admit I'm
concerned that if I start mucking around with the Makefiles/autotool
scaffolding, I'm going to break stuff. But maybe it's worth taking that
risk if it allows us to do things more correctly going forward... I
could at least take a crack at it and see what people think of the results.
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 3:37 ` Gregory M. Turner
@ 2012-10-11 3:50 ` Diego Elio Pettenò
2012-10-11 4:14 ` Mike Frysinger
1 sibling, 0 replies; 20+ messages in thread
From: Diego Elio Pettenò @ 2012-10-11 3:50 UTC (permalink / raw
To: gentoo-dev
On 10/10/2012 20:37, Gregory M. Turner wrote:
>
> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.
Using the -L . -lfoo is a very bad style in general. Just think what
happen if you're trying to link to a libutil.a
--
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 3:37 ` Gregory M. Turner
2012-10-11 3:50 ` Diego Elio Pettenò
@ 2012-10-11 4:14 ` Mike Frysinger
2012-10-11 9:35 ` Gregory M. Turner
1 sibling, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2012-10-11 4:14 UTC (permalink / raw
To: gentoo-dev; +Cc: Gregory M. Turner
[-- Attachment #1: Type: Text/Plain, Size: 2530 bytes --]
On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> (1) is worse than (2), but it does have some quasi-legitimate usages.
> For example, prefix bootstrap does this (or used to), as do many of the
> crossdev-wrapper scripts. I've also resorted to such usage, myself,
> when repairing a prefix after I've broken its compiler. These cases
> don't really seem completely "correct" or "incorrect" -- about the best
> I can say for them it is that they are mostly efficacious, but prone to
> problems.
they're broken. and no, crossdev doesn't do this. it is properly sysrooted.
further, you cannot do multilib with users putting -L<system libdir> into
their LDFLAGS. they just forced a single ABI and any other will simply fail.
the toolchain's compiler driver knows exactly where to find its own libraries.
if it doesn't, it's broken, and it should be fixed.
> So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
> complete abomination. Still... I don't like it.
it's significantly better than the proposed code which tries to parse the
meaning of various flags and insert it at the "right" point. i can't see it
being useful at all. it's a recipe for fragility and being unmaintainable.
> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.
yes, so why is the ebuild specifying -L. for it ? the package should already
have a -L to the right place, and if it wants to make sure it gets used before
the user's LDFLAGS, it should show up in the linking before it (or have the
build system prepend the value to LDFLAGS if it's appending currently).
> In the particular case of Gentoo's cpython, I have to admit I'm
> concerned that if I start mucking around with the Makefiles/autotool
> scaffolding, I'm going to break stuff. But maybe it's worth taking that
> risk if it allows us to do things more correctly going forward... I
> could at least take a crack at it and see what people think of the results.
i'm not sure why this applies to the larger build system. i can understand
that the python build itself might not be putting the -L. in a place where a
user's misconfigured LDFLAGS will cause a problem.
but if you want the right answer, it's to reject broken user LDFLAGS. we
don't support wrong things like -fPIC or -fPIE in global CFLAGS. i don't
think we should be wasting time on LDFLAGS like this either.
-mike
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 4:14 ` Mike Frysinger
@ 2012-10-11 9:35 ` Gregory M. Turner
2012-10-11 15:50 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-11 9:35 UTC (permalink / raw
To: gentoo-dev; +Cc: Mike Frysinger
On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
>> (1) is worse than (2), but it does have some quasi-legitimate usages.
>> For example, prefix bootstrap does this (or used to), as do many of the
>> crossdev-wrapper scripts. I've also resorted to such usage, myself,
>> when repairing a prefix after I've broken its compiler. These cases
>> don't really seem completely "correct" or "incorrect" -- about the best
>> I can say for them it is that they are mostly efficacious, but prone to
>> problems.
>
> and no, crossdev doesn't do this. it is properly sysrooted.
You are right. I guess I was thinking of this:
http://tinderbox.x86.dev.gentoo.org/misc/cmerge,
Which does use -L but looks pretty ancient. I see now that crossdev's
equivalent hacks up the linker scripts -- which seems as "correct" as
one could reasonably hope for... pretty sexy.
> further, you cannot do multilib with users putting -L<system libdir> into
> their LDFLAGS. they just forced a single ABI and any other will simply fail.
> the toolchain's compiler driver knows exactly where to find its own libraries.
> if it doesn't, it's broken, and it should be fixed.
ACK
>> So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
>> complete abomination. Still... I don't like it.
>
> it's significantly better than the proposed code which tries to parse the
> meaning of various flags and insert it at the "right" point. i can't see it
> being useful at all. it's a recipe for fragility and being unmaintainable.
I think I meant the above differently than you interpreted it: I meant
that solving library-search-path selection in autotools steps and/or
Makefile.in would make the question of how to manipulate LDFLAGS
academic, and seems more elegant. Apparently you agree:
>> If the Makefiles are building against libraries expected to be in
>> ${PWD}, it seems to me that the Makefiles should know to look there
>> automatically.
>
> yes, so why is the ebuild specifying -L. for it ? the package should already
> have a -L to the right place, and if it wants to make sure it gets used before
> the user's LDFLAGS, it should show up in the linking before it (or have the
> build system prepend the value to LDFLAGS if it's appending currently).
If we decide that manipulating LDFLAGS is correct, whether we do this
using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.
My point above was that coming to that decision in the first place
suggests some underlying issue exists that arguably ought to be fixed
instead.
> i'm not sure why this applies to the larger build system. i can understand
> that the python build itself might not be putting the -L. in a place where a
> user's misconfigured LDFLAGS will cause a problem.
This only pertains to dev-lang/python AFAIK... there may be others
lurking out there somewhere, but I'm not too worried about them. The
use-cases I have in mind are prefix bootstrap and system-recovery when
gcc breaks.
In general, if python breaks, folks can end up in pretty bad shape due
to chicken-and-egg issues with portage. That's the only reason I've
spent so much effort trying to think this fairly obscure corner-case
through.
-gmt
P.S.: After poking around, the first commit where this appeared is this
one:
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/python/python-2.5.1.ebuild?hideattic=0&r1=1.4&r2=1.5&view=patch,
a mere 5.5 years ago by uberlord.
The cvs ChangeLog references bug 177426 which doesn't seem particularly
illuminating. It also says:
export LDFLAGS=-L. so we link modules correctly on FreeBSD
and possibly other systems where python2.5 isn't installed yet.
Which at least seems to confirm that this was originally a problem not
of search path ordering, but of complete failure to search ${S} in the
first place.
Meanwhile, I've looked into the patches a bit. In regular Gentoo
nothing special happens to LDFLAGS, but in prefix, there is special
handling that suggests some avenues of attack. Before I do anything
further, I want to figure out if it works right without LDPATH
manipulation on linux, and, if so, whether I can't generalize whatever
mechanism allows that to happen.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 9:35 ` Gregory M. Turner
@ 2012-10-11 15:50 ` Mike Frysinger
2012-10-11 20:39 ` Gregory M. Turner
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2012-10-11 15:50 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: Text/Plain, Size: 3322 bytes --]
On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
> On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> > On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> >> (1) is worse than (2), but it does have some quasi-legitimate usages.
> >> For example, prefix bootstrap does this (or used to), as do many of the
> >> crossdev-wrapper scripts. I've also resorted to such usage, myself,
> >> when repairing a prefix after I've broken its compiler. These cases
> >> don't really seem completely "correct" or "incorrect" -- about the best
> >> I can say for them it is that they are mostly efficacious, but prone to
> >> problems.
> >
> > and no, crossdev doesn't do this. it is properly sysrooted.
>
> You are right. I guess I was thinking of this:
>
> http://tinderbox.x86.dev.gentoo.org/misc/cmerge,
>
> Which does use -L but looks pretty ancient. I see now that crossdev's
> equivalent hacks up the linker scripts -- which seems as "correct" as
> one could reasonably hope for... pretty sexy.
by default it doesn't. with the elibtoolize hacks and the drive for .pc files,
this has been largely mitigated. although the latest libtool has a --with-
sysroot option that needs investigating.
> >> If the Makefiles are building against libraries expected to be in
> >> ${PWD}, it seems to me that the Makefiles should know to look there
> >> automatically.
> >
> > yes, so why is the ebuild specifying -L. for it ? the package should
> > already have a -L to the right place, and if it wants to make sure it
> > gets used before the user's LDFLAGS, it should show up in the linking
> > before it (or have the build system prepend the value to LDFLAGS if it's
> > appending currently).
>
> If we decide that manipulating LDFLAGS is correct, whether we do this
> using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.
it's not particularly important, but on one hand, the LDFLAGS parsing logic
should not be in the tree ever. on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python
ebuild really really want to add it.
i'd still lobby for rejecting of invalid LDFLAGS settings and fixing things at
the "right" place.
> In general, if python breaks, folks can end up in pretty bad shape due
> to chicken-and-egg issues with portage. That's the only reason I've
> spent so much effort trying to think this fairly obscure corner-case
> through.
that's really only a problem in the bootstrap case. if an existing system
can't install python, then it isn't like they're screwed since they already
have a working python installed.
> Meanwhile, I've looked into the patches a bit. In regular Gentoo
> nothing special happens to LDFLAGS, but in prefix, there is special
> handling that suggests some avenues of attack. Before I do anything
> further, I want to figure out if it works right without LDPATH
> manipulation on linux, and, if so, whether I can't generalize whatever
> mechanism allows that to happen.
my [limited] understanding of the prefix compiler is that they wrap things with
a custom shell script to inject -L flags behind the back of the compiler at the
very last possible minute and point to the right place.
-mike
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 15:50 ` Mike Frysinger
@ 2012-10-11 20:39 ` Gregory M. Turner
2012-10-11 21:40 ` Marien Zwart
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-11 20:39 UTC (permalink / raw
To: gentoo-dev; +Cc: Mike Frysinger
On 10/11/2012 8:50 AM, Mike Frysinger wrote:
> On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
>> On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> it's not particularly important, but on one hand, the LDFLAGS parsing logic
> should not be in the tree ever.
I've no major attachment to it. Took all of five minutes to code up
reading the ld manpage, and as you pointed out, it's probably solving a
non-problem.
> on the other, i can stomach much smaller one-
> off hacks like prepending -L. to LDFLAGS if the maintainers of the python
> ebuild really really want to add it.
If fixing the python builds proves too onerous then this is what I'll
end up filing a bug for.
> my [limited] understanding of the prefix compiler is that they wrap things with
> a custom shell script to inject -L flags behind the back of the compiler at the
> very last possible minute and point to the right place.
Pretty fuzzy on this myself. Whatever binutils-config does works so
well I've never had to look into it very deeply :)
Above, however, I'm referring to the prefix-specific cpython patch-sets,
where I was "sure" I'd seen patches to project LDFLAGS into setup.py's
compiler invocation tables. But it seems I was mistaken. Perhaps it
was in the main gentoo-patches after all... or maybe I just need more
coffee...
Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!
With any luck, this explains what's going on and suggests that platforms
linking python modules against libpython.so just need to wedge -L${S}
somewhere in configure.in, possibly do-able via sed script (unless we
end up having to mess with setup.py, in which case, things may get a bit
more complicated).
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 20:39 ` Gregory M. Turner
@ 2012-10-11 21:40 ` Marien Zwart
2012-10-12 11:03 ` Gregory M. Turner
0 siblings, 1 reply; 20+ messages in thread
From: Marien Zwart @ 2012-10-11 21:40 UTC (permalink / raw
To: gentoo-dev
I'm going to do something potentially rude and comment on this without
having read the entire thread.
On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:
> Anyhow one thing I have figured out is how things can work correctly on
> Linux wihtout -L.: on Linux, the python plugins aren't actually linked
> against libpython.so!
Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.
However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.
The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).
--
Marien Zwart.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-11 21:40 ` Marien Zwart
@ 2012-10-12 11:03 ` Gregory M. Turner
2012-10-14 8:49 ` Gregory M. Turner
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-12 11:03 UTC (permalink / raw
To: gentoo-dev; +Cc: marienz
On 10/11/2012 2:40 PM, Marien Zwart wrote:
> I'm going to do something potentially rude and comment on this without
> having read the entire thread.
>
> On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:
>> Anyhow one thing I have figured out is how things can work correctly on
>> Linux wihtout -L.: on Linux, the python plugins aren't actually linked
>> against libpython.so!
>
> Python can be built with and without a shared library
> (libpythonx.y.so). If the shared library is built the interpreter
> executable is linked against it (of course), and normally so are the
> extension modules. This is useful because it makes the linker do the
> right thing for applications that embed python (and end up loading
> libpythonx.y via dlopen). If the shared library is not built extension
> modules cannot link against it, and the python executable itself
> exports the relevant symbols. So on a normal shared python build
> extensions should end up linked against libpythonx.y.so.
>
> However, at least one popular distro and presumably most of its
> derivatives (Debian) has a system python built without the shared
> library, plus the shared library built separately, for use by
> applications that embed python. I am not entirely sure why they still
> do this (if I understand correctly it was originally done for
> performance reasons, but I'm not sure if they've checked those are
> still valid). Their extension modules cannot link to libpythonx.y:
> their python interpreter executable provides the same symbols
> libpythonx.y provides, so loading such an extension module into the
> regular interpreter would result in collisions.
>
> The reason I mention this is Gentoo's python is built with the shared
> library, so on Gentoo your extensions *should* end up linked to
> libpythonx.y (if they use a more or less standard build system, at
> least). If yours are not I'm curious what happened, and would
> appreciate it if you gave me some more information off-list (marienz
> on freenode or email).
Thanks a ton for this info, that explains a lot. Actually, got this
notion testing outside of Gentoo, building from a vanilla python 3.3.0
tarball on a Fedora box. I wanted to see what happens in a completely
"Gentoo-free" environment so as to get some kind of baseline
understanding of how upstream is propagating stuff through the Makefiles
to setup.py.
Probably, I screwed up ./configure (I tried to approximate Gentoo's
arguments), and requested a non-shlib build. Guess it would have been
clearer if I'd mentioned that in my post -- sorry if I worried you :)
I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever
see it fail to do as you describe, I'll let you know, Marien.
My i686 prefix is up to date and I can confirm that it is linking
against libpython there.
But... wow. In my almost-vanilla i686 cross-prefix (hosted by my
only-moderately-rice multilib ~amd64 workstation), in
=dev-lang/python-2.7.3-r2, the modules are linked like so:
i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-fno-strict-aliasing \
-O2 -march=i686 -pipe -m32 \
-fwrapv -DNDEBUG \
-I. -IInclude -I./Include \
-I${EPREFIX}/usr/include \
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o \
build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-lm -lpython2.7 \
-o build/lib.linux-i686-2.7/math.so
Which is just shameful. In this case, I'm pretty sure all those
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:
[PREFIX] $ emerge --info --verbose | grep /lib
sys-devel/libtool: 2.4.2::gentoo_prefix
COLLISION_IGNORE="/lib/modules/* *.py[co]"
PKG_CONFIG_PATH="${EPREFIX}/usr/lib/pkgconfig:$( :;
)${EPREFIX}/usr/share/pkgconfig"
PORTAGE_BIN_PATH="${EPREFIX}/usr/lib/portage/bin"
PORTAGE_PYM_PATH="${EPREFIX}/usr/lib/portage/pym"
UNINSTALL_IGNORE="/lib/modules/*"
(snippets edited for readability obv).
Regarding the gcc-as-linker invocation above:
First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).
Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).
Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so). I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.
Plus, it repeats itself like a stuttering Makefool -- but that's pretty
standard and only ever seems to bother me :)
Perhaps, this is a consequence of mine being a cross-prefix, somehow.
That's the only thing significantly off-the-beaten-path thing about it.
Even if so, I'd like amd64->i686 cross-prefixes to work and don't see
why they shouldn't.
The only good news I can report is that changing
append-ldflags "-L."
to
export LDFLAGS="-L. ${LDFLAGS}"
does seem to fix the third issue, above.
Looks like, at least for prefix, it's going to take more than the quick
configure patch I was hoping for, if we're to make things fully right.
We'll see soon enough about ~amd64 linux, and I'm spinning up a
freebsd90 vm to kick around as well.
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-12 11:03 ` Gregory M. Turner
@ 2012-10-14 8:49 ` Gregory M. Turner
2012-10-15 4:29 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-14 8:49 UTC (permalink / raw
To: gentoo-dev
On 10/12/2012 4:03 AM, Gregory M. Turner wrote:
> First, something puts in all kinds of inappropriate amd64 multilib paths
> (this ends up being harmless as wrong-arch libraries get rejected at
> link-time and treated as non-matches for -lclauses... still, WTF?).
>
> Secondly, something puts the built-in ld search paths into the
> command-line ld search path (a practice which has been roundly
> disparaged in this thread, and correctly so, IMO).
>
> Thirdly, prefix x86-linux also clearly suffers from the original problem
> I was seeing in cygwin, and which inspired this thread (putting -L.
> after $(libdir) paths and therefore linking against the wrong
> libpython.so). I never suspected this applied to every prefix python
> ebuild, but that's what the above seems to suggest... not cool.
Just wanted to update this thread with latest current triage and
prognosis data for this patient:
Regarding etiology, for "Firstly" and "Secondly" above: it seems that
these are by design (which is not to say that they should be left: in
each case, the "design" is pretty bad and ought to be fixed, I think).
My hope is that "Firstly" and "Secondly" are needed only to accommodate
prefix bootstrap, in which case we can fix them by detecting whether a
prefix bootstrap is ongoing or not, and conditioning these hacks on that
(improving "Firstly" to not be braindead would be a nice touch, although
doing so in a way that doesn't break anything may be prohibitively
difficult or beyond my abilities).
"Thirdly" has been addressed ad nauseam in this thread and will be
solved by prepending the LDFLAG rather than appending, or, preferably,
by patching autotools (but only if I can find a simple, low-maintenance
approach that is likely to work without building any new per-platform
matrices or case-statements).
Note that, even without this third fix, absent a PEBKAC placement of
"-L${EPREFIX}/usr/$(libdir)" into LDFLAGS, either:
o it's a prefix bootstrap, in which case there probably is no
${EPREFIX}/usr/$(libdir)/libpython${PV}.so, or
o it's /not/ a prefix bootstrap, in which case the above-mentioned
fixes would prevent the "-L${EPREFIX}/usr/$(libdir)" from
getting into LDFLAGS, thus preventing the problem from arising.
So by fixing "Firstly" and "Secondly," we have at least brought things
sufficiently in line with expectations that we are only fixing
user-level configuration mistakes instead of our own, as originally
expected. **
I'm feeling ready, at this point, to file bugs for these issues; once I
do, I'll post bug#'s into this thread and plan to take all off this
chatter off-list, unless something more comes up that seems to require
community review.
Thanks for everyone's help figuring this stuff out,
-gmt
** I suppose a third possibility is that someone is doing an in-place
"re-bootstrap" or restarting a bootstrap build that failed mid-merge,
but ... doesn't matter, it'll be academic once we fix the bug.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-14 8:49 ` Gregory M. Turner
@ 2012-10-15 4:29 ` Mike Frysinger
2012-10-15 8:35 ` Gregory M. Turner
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2012-10-15 4:29 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: Text/Plain, Size: 784 bytes --]
On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> "Thirdly" has been addressed ad nauseam in this thread and will be
> solved by prepending the LDFLAG rather than appending, or, preferably,
> by patching autotools (but only if I can find a simple, low-maintenance
> approach that is likely to work without building any new per-platform
> matrices or case-statements).
i'm fairly certain this isn't autotools. i've poked around the python build
system before in the past and while it uses autoconf to do platform tests, it
doesn't use automake/libtool. make is used to bootstrap python, and then they
descend into a horrible hack of a custom build system written in python. i
suspect much of the pain you're seeing is coming from that last part.
-mike
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-15 4:29 ` Mike Frysinger
@ 2012-10-15 8:35 ` Gregory M. Turner
2012-10-15 17:47 ` Mike Frysinger
0 siblings, 1 reply; 20+ messages in thread
From: Gregory M. Turner @ 2012-10-15 8:35 UTC (permalink / raw
To: gentoo-dev
On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
>> "Thirdly" has been addressed ad nauseam in this thread and will be
>> solved by prepending the LDFLAG rather than appending, or, preferably,
>> by patching autotools (but only if I can find a simple, low-maintenance
>> approach that is likely to work without building any new per-platform
>> matrices or case-statements).
>
> i'm fairly certain this isn't autotools. i've poked around the python build
> system before in the past and while it uses autoconf to do platform tests, it
> doesn't use automake/libtool. make is used to bootstrap python, and then they
> descend into a horrible hack of a custom build system written in python. i
> suspect much of the pain you're seeing is coming from that last part.
> -mike
I guess I should say "in src_prepare" rather than "in autotools."
Specifically, I was thinking some kind of configure.in patch might be
good, since configure.in seems to churn less than Makefile.in, a good
thing if I want to produce a one-size-fits-all patch.
And, yeah, setup.py is definitely behind all this yucky.
Python clearly has an amazing community, so I hate to say anything
negative... but I sometimes wish they would "build" less and "buy" more.
Anyhow, as everyone knows, bitching and moaning about FOSS is pointless
so I'll stop there and spend my time fixing stuff instead :)
-gmt
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-15 8:35 ` Gregory M. Turner
@ 2012-10-15 17:47 ` Mike Frysinger
2012-10-17 10:11 ` Marien Zwart
0 siblings, 1 reply; 20+ messages in thread
From: Mike Frysinger @ 2012-10-15 17:47 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: Text/Plain, Size: 1519 bytes --]
On Monday 15 October 2012 04:35:09 Gregory M. Turner wrote:
> On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> > On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> >> "Thirdly" has been addressed ad nauseam in this thread and will be
> >> solved by prepending the LDFLAG rather than appending, or, preferably,
> >> by patching autotools (but only if I can find a simple, low-maintenance
> >> approach that is likely to work without building any new per-platform
> >> matrices or case-statements).
> >
> > i'm fairly certain this isn't autotools. i've poked around the python
> > build system before in the past and while it uses autoconf to do
> > platform tests, it doesn't use automake/libtool. make is used to
> > bootstrap python, and then they descend into a horrible hack of a custom
> > build system written in python. i suspect much of the pain you're
> > seeing is coming from that last part. -mike
>
> And, yeah, setup.py is definitely behind all this yucky.
>
> Python clearly has an amazing community, so I hate to say anything
> negative... but I sometimes wish they would "build" less and "buy" more.
build systems are hard to get right. python is in the situation where the
setups they care about mostly work and people generally aren't complaining,
but it's more through a hack effort than doing it right which means all the
other cases they haven't considered break horribly. cross-compiling for
example has never worked correctly out of the box.
-mike
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath
2012-10-15 17:47 ` Mike Frysinger
@ 2012-10-17 10:11 ` Marien Zwart
0 siblings, 0 replies; 20+ messages in thread
From: Marien Zwart @ 2012-10-17 10:11 UTC (permalink / raw
To: gentoo-dev
On Monday 15 October 2012 04:35:09 Gregory M. Turner wrote:
> On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> > Python clearly has an amazing community, so I hate to say anything
> > negative... but I sometimes wish they would "build" less and "buy" more.
>
> build systems are hard to get right. python is in the situation where the
> setups they care about mostly work and people generally aren't complaining,
> but it's more through a hack effort than doing it right which means all the
> other cases they haven't considered break horribly. cross-compiling for
> example has never worked correctly out of the box.
The reasons for Python's awkward build system are probably mostly
historical. I'll (unsuccessfully) try to keep this brief, but still:
Python is old: Python 1.0.0 was released January 1994, and Python
existed before that. I don't know what its build system was like back
then, but most systems you might want them to "buy" didn't exist yet.
Distutils (the build system for external extension modules as well as
chunks of Python's standard library) is still old: it was integrated
into Python 1.6 (September 5, 2000). Autoconf 2.13 existed, 2.50
wasn't there yet. Looks like CMake was just turned into a separate
project around that time.
Python's needs are a little special: you can build extension modules
using just Python and a C compiler (and you don't need the C compiler
if your code is pure-python). This is important on awkward platforms
(currently mostly Windows, but I'm pretty sure there were others
similarly awkward). This same Python-powered build system is used to
build part of the standard library, for obvious reasons. And there are
other build systems that make use of python mainly to be
cross-platform (scons and derivatives mostly).
It would probably be possible to redo Python's own build system now
and end up with something better, but I'm not aware of anyone having
seriously attempted this, probably because it's just not quite broken
enough to be worth it. The awkwardness of distutils we're somewhat
stuck with, although there are efforts to modernize and clean up
distutils. I'm not so sure there's anything they could have "bought"
to replace Python's own build system, let alone distutils (build
systems like automake actually use distutils.sysconfig), without a
working time machine (and a rather confusing time loop to allow build
systems that currently support python to do so without
distutils.sysconfig).
--
marienz.
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2012-10-17 10:12 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-06 2:50 [gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath Gregory M. Turner
2012-10-06 7:47 ` [gentoo-dev] " Gregory M. Turner
2012-10-06 8:31 ` Fabian Groffen
2012-10-10 22:47 ` Gregory M. Turner
2012-10-11 3:28 ` Mike Frysinger
2012-10-06 15:13 ` Duncan
2012-10-09 21:26 ` Mike Frysinger
2012-10-11 3:37 ` Gregory M. Turner
2012-10-11 3:50 ` Diego Elio Pettenò
2012-10-11 4:14 ` Mike Frysinger
2012-10-11 9:35 ` Gregory M. Turner
2012-10-11 15:50 ` Mike Frysinger
2012-10-11 20:39 ` Gregory M. Turner
2012-10-11 21:40 ` Marien Zwart
2012-10-12 11:03 ` Gregory M. Turner
2012-10-14 8:49 ` Gregory M. Turner
2012-10-15 4:29 ` Mike Frysinger
2012-10-15 8:35 ` Gregory M. Turner
2012-10-15 17:47 ` Mike Frysinger
2012-10-17 10:11 ` Marien Zwart
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox