[gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

----[ "eclass/flag-o-matic.eclass" ]----->8----->
--- PORTAGE/eclass/flag-o-matic.eclass
+++ OVERLAY/eclass/flag-o-matic.eclass
@@ -117,6 +117,42 @@
         return 0
  }

+# @FUNCTION: prepend-ldpath
+# @USAGE: <path>
+# @DESCRIPTION:
+# Place the specified ldpath into LDFLAGS before any options which could
+# add additional paths to ld's search path.  Specifically, it will place
+# the new <path> "/foo" into LDFLAGS as "-L/foo" just before the first
+# occurance matching any of the globs: '-L*', '-T', '--library-path*',
+# and '--script*', but not matching any of the globs: '-Tbss=*',
+# '-Tdata=*', '-Ttext=*', and '-Ttext-segment=*'.  If no such match is
+# found, then this is equivalent to "append-ldflags -L<path>".
+prepend-ldpath() {
+       local new=()
+       local f
+       local done=no
+       for f in ${LDFLAGS} ; do
+               case "${f}" in
+                       -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+                               new+=( "${f}" )
+                               ;;
+                       -L*|-T*|--library-path*|--script*)
+                               if [[ ${done} == yes ]] ; then
+                                       new+=( "${f}" )
+                               else
+                                       new+=( "-L${1}" "${f}" )
+                                       done=yes
+                               fi
+                               ;;
+                       *)
+                               new+=( "${f}" )
+                               ;;
+               esac
+       done
+       [[ ${done} == no ]] && new+=( "-L${1}" )
+       export LDFLAGS="${new[*]}"
+}
+
  # @FUNCTION: filter-lfs-flags
  # @DESCRIPTION:
  # Remove flags that enable Large File Support.
<-----8<-----

I think my code is probably fine, or if it's buggy, so be it.  A prior 
question is: does this have sufficient utility?

I "need" (not exactly... keep reading) something like this for python. 
Currently python*.ebuild contains:

   append-ldflags "-L."

which is required in order to pull in the newly-rebuilt 
libpython${VERSION}.so at ebuild-time, instead of the system version 
(perhaps it's only so on obscure platforms?).

The problem is, LDFLAGS may already have some library paths coming in 
from the environment.  In this case, we end up with something like:

   LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."

and python goes ahead links everything against, i.e.:

   /random/prefix/usr/lib/libpython3.2.so,

or whatever is the version in question, so we achieve precisely nothing, 
effecting the outcome we are trying to prevent (the problem manifests in 
subtle, hard-to-diagnose ways, unfortunately).

In general, when we say append-ldflags -Lsomething, we may or may not 
mean "always use something/libfoo, not $(libdir)/libfoo to link."  In 
the case where we do mean "always", append-ldflags is a broken approach.

An alternative would be to just replace

   append-ldflags "-L."

with

   export LDFLAGS="-L. ${LDFLAGS}".

however, there are problems with this:

First, it's aesthetically annoying that this spreads out the "-L*" 
arguments into two different "zones" in LDFLAGS.  No big deal, but it 
definitely doesn't make visually scanning the logs any easier.

Second, although I'm not aware of any ld arguments which act as 
"modifiers" to "-L" arguments (except the other -L arguments and 
ldscript arguments, which the above patch scans for), ld does use 
order-dependent patterns for other arguments.

For example: "-lfoo -( -lbar -lbaz -)" is not the same thing as "-( 
-lfoo -lbar -) -lbaz".  So order-dependencies might emerge in some 
future binutils... in that case we'd still probably need to extend the 
case statement above to include the new order-affecting arguments, but 
at least we'd have a place in which to do so.

Third, although the meaning of -L* options may not be affected by other 
arguments in an order-dependant manner, the reverse is not so.  For 
example, see --mri-script, --script (which both can affect and be 
affected by the command-line library search path in an order-dependent 
manner!), -rpath-link (SunOS only), etc...

One could certainly argue that, in practice, points two and three amount 
to hopelessly obscure nitpicking and that nobody in their right mind 
should be relying on this type of stuff in their LDFLAGS.

I'm not sure I'd state it quite so strongly as that, (after all these 
might come just from profile.bashrc and be targeted to a particular 
ebuild) but justification is clearly on the "thin" side.

Indeed, if we're going to worry about side effects, it's not entirely 
clear that what what my patch does is safe.  For example, if the 
environment supplied "-L/foo/bar --script baz", and python for some 
reason had a "baz" file in "${S}", then some kind of breakage would 
surely happen when we did

   prepend-ldpath "."

Also, we might also legitimately worry that the presence of this 
function in flag-o-matic will somehow "encourage" people to do this 
more, and, I'd have to agree that it's a pretty shitty approach to 
getting gcc to find your generated libraries (perhaps I could answer 
this concern by means of some kind of "don't use unless you must" 
disclaimer, or by grafting this code into python.eclass during configure()?)

Gross as it may be, it's the approach we've adopted in dev-lang/python*, 
a critically important, mandatory package, and it's broken.  It's 
definitely not just an academic problem, at least not for my 
cygwin-overlay where it makes it impossible to toggle USE=threads for 
this package.  Presumably the same goes for BSD and any other platforms 
relying on

So we either need to fix the tactics or the strategy -- personally I'm a 
bit reluctant to mess with the latter, which, I guess, is how I end up 
with the above, despite some misgivings.

-gmt

[gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum 
opus above and allow me to try again.

In dev-lang/python*, we use

   append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree, 
rather than than in the one in $(libdir).  But, this doesn't work if 
LDFLAGS contains "-L$(libdir)".

We could try to fix this like:

  export LDFLAGS="-L. ${LDFLAGS}"

or so.  That would cover 99.9% of the cases out there.  But very rarely, 
indiscriminately placing our '-L.' before every other clause in LDFLAGS 
might cause an unanticipated side-effect.

The flag-o-matic patch in my previous post analyses LDPATH and finds the 
first statement that might possibly cause the ld command-line 
library-search-path to be impacted, and inserts the "-L." just before 
that statement, hopefully achieving the desired result with the least 
possible side-effects.

I wonder: do people feel this distinction is meaningful enough to 
justify actually making it?  Or am I just being an anal-retentive nut-job?

If you tried to read my first post, thanks for not offing yourself like 
the guys who get stuck sitting next to Ted Striker in _Airplane!_

-gmt

[gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
> My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum 
> opus above and allow me to try again.
> 
> In dev-lang/python*, we use
> 
>    append-ldflags '-L.'
> 
> to ensure linking is performed against the built libpython.so in-tree, 
> rather than than in the one in $(libdir).  But, this doesn't work if 
> LDFLAGS contains "-L$(libdir)".
> 
> We could try to fix this like:
> 
>   export LDFLAGS="-L. ${LDFLAGS}"
> 
> or so.  That would cover 99.9% of the cases out there.  But very rarely, 
> indiscriminately placing our '-L.' before every other clause in LDFLAGS 
> might cause an unanticipated side-effect.
> 
> The flag-o-matic patch in my previous post analyses LDPATH and finds the 
> first statement that might possibly cause the ld command-line 
> library-search-path to be impacted, and inserts the "-L." just before 
> that statement, hopefully achieving the desired result with the least 
> possible side-effects.

I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it.  I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Duncan]

Gregory M. Turner posted on Sat, 06 Oct 2012 00:47:57 -0700 as excerpted:

> If you tried to read my first post, thanks for not offing yourself like
> the guys who get stuck sitting next to Ted Striker in _Airplane!_

It was worth it just for this:

>>   LDFLAGS="-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L."

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1362 bytes --]

On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
> My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum
> opus above and allow me to try again.
> 
> In dev-lang/python*, we use
> 
>    append-ldflags '-L.'
> 
> to ensure linking is performed against the built libpython.so in-tree,
> rather than than in the one in $(libdir).  But, this doesn't work if
> LDFLAGS contains "-L$(libdir)".

well, some notes here ...

for users: putting -L<some-system-path> is wrong.  it changes the meaning of 
"system paths" when it comes to searching for linking by elevating it to "user 
specified path".  imo, you should be fixing the source rather than the symptom.

for packagers: using -L$(libdir) is almost always wrong and pointless.  the 
toolchain itself knows the proper system path to search for libraries to link 
against, and when you cross-compile, the --libdir you specify to configure is 
not going to be valid when dereferenced on the build system.

> We could try to fix this like:
> 
>   export LDFLAGS="-L. ${LDFLAGS}"
> 
> or so.  That would cover 99.9% of the cases out there.  But very rarely,
> indiscriminately placing our '-L.' before every other clause in LDFLAGS
> might cause an unanticipated side-effect.

i would go with this if you want to support these people doing it wrong
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/6/2012 1:31 AM, Fabian Groffen wrote:
> On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:
>> In dev-lang/python*, we use
>>
>>     append-ldflags '-L.'
>>
>> to ensure linking is performed against the built libpython.so in-tree,
>> rather than than in the one in $(libdir).  But, this doesn't work if
>> LDFLAGS contains "-L$(libdir)".
>>
>> We could try to fix this like:
>>
>>    export LDFLAGS="-L. ${LDFLAGS}"
>>
>> or so.  That would cover 99.9% of the cases out there.  But very rarely,
>> indiscriminately placing our '-L.' before every other clause in LDFLAGS
>> might cause an unanticipated side-effect.
> I think it would make more sense in this case to just add one more patch
> to Python, such that the build-system just inserts it.  I recall it's
> necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
> more why it works/worked on Linux fine.

I was thinking along these lines as well.  As it turned out, however, I 
was not able to find an automagical, PMS-compliant, non-crazy way to do 
it that resolved my test-case (more on that below), without effectively 
hooking econf().

Literally hooking a core API seems like a Pandora's box better left 
unopened, so this is what I came up with (note: the fpectl stuff was 
also in my overlay and the patches got entangled -- rather than editing 
the hunk by hand, I just left it.  It's orthogonal to this issue, 
however, and for present purposes can be ignored):

------>8----->
--- PORTAGE/dev-lang/python/python-2.7.3-r2.ebuild
+++ OVERLAY/dev-lang/python/python-2.7.3-r2.ebuild
@@ -218,13 +255,6 @@
  	# http://bugs.python.org/issue15506
  	export ac_cv_path_PKG_CONFIG=$(tc-getPKG_CONFIG)

-	# Set LDFLAGS so we link modules with -lpython2.7 correctly.
-	# Needed on FreeBSD unless Python 2.7 is already installed.
-	# Please query BSD team before removing this!
-	# On AIX this is not needed, but would record '.' as runpath.
-	[[ ${CHOST} == *-aix* ]] ||
-	append-ldflags "-L."
-
  	local dbmliborder
  	if use gdbm; then
  		dbmliborder+="${dbmliborder:+:}gdbm"
@@ -248,11 +278,18 @@
  		&& myconf="${myconf} --enable-framework=${EPREFIX}/usr/lib" \
  		|| myconf="${myconf} --enable-shared"

+	if [[ ${CHOST} == *-cygwin* ]] ; then
+		fpeconfig="--without-fpectl"
+		myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"
+	else
+		fpeconfig="--with-fpectl"
+	fi
+
  	# note: for a framework build we need to use ucs2 because OSX
  	# uses that internally too:
  	# http://bugs.python.org/issue763708
-	OPT="" econf \
-		--with-fpectl \
+	OPT="" cpython_econf \
+		${fpeconfig} \
  		$(use_enable ipv6) \
  		$(use_with threads) \
  		$( (use wide-unicode && use !aqua) && echo "--enable-unicode=ucs4" 
|| echo "--enable-unicode=ucs2") \
--- PORTAGE/eclass/python.eclass
+++ OVERLAY/eclass/python.eclass
@@ -401,6 +401,64 @@
  	fi
  }

+# see http://thread.gmane.org/gmane.linux.gentoo.devel/80633/focus=80635
+_python_prepend_cwd_ldpath() {
+	local new=()
+	local f
+	local done=no
+	for f in ${LDFLAGS} ; do
+		case "${f}" in
+			-Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+				new+=( "${f}" )
+				;;
+			-L*|-T*|--library-path*|--script*)
+				if [[ ${done} == yes ]] ; then
+					new+=( "${f}" )
+				elif [[ "${f}" == "-L." ]] ; then
+					# if it's somehow already there, don't duplicate it
+					new+=( "-L." )
+					done=yes
+				else
+					new+=( "-L." "${f}" )
+					done=yes
+				fi
+				;;
+			*)
+				new+=( "${f}" )
+				;;
+		esac
+	done
+	[[ ${done} == no ]] && new+=( "-L." )
+	export LDFLAGS="${new[*]}"
+}
+
+# @FUNCTION: cpython_econf
+# @DESCRIPTION:
+# econf() substitute for use in dev-lang/python ebuilds
+#
+# On some platforms, it is neccesary to prepend "-L." to ldpath before
+# proceeding with python's configure process.  Using cpython_econf()
+# instead of econf() will ensure that this is taken care of correctly
+# before python's configure step can proceed.  This is not needed for
+# any python ebuilds other than dev-lang/python; any other ebuild
+# calling this function will receive an error.
+cpython_econf() {
+	if [[ "${CATEGORY}/${PN}" != "dev-lang/python" ]] ; then
+		die "cpython_econf can only be used by dev-lang/python ebuilds"
+	fi
+	# econf will enforce ${EBUILD_PHASE} requirements, so we don't bother.
+	
+	# Set LDFLAGS so we link modules with -lpython2.7 correctly.
+	# Needed on FreeBSD unless Python 2.7 is already installed.
+	# Even if python is already installed, linking against the old
+	# python will cause problems, i.e., when toggling USE="threads".
+	# Also needed on cygwin.  Please query BSD team before removing this!
+	# On AIX this is not needed, and would record '.' as runpath.
+	[[ ${CHOST} == *-aix* ]] || _python_prepend_cwd_ldpath
+
+	econf "$@"
+}
+
  # @FUNCTION: python_pkg_setup
  # @DESCRIPTION:
  # Perform sanity checks and initialize environment.
<-----8<-----

Some notes about the above:

Of course we would presumably want to make a similar change in all of 
the dev-lang/python ebuilds to the one above.

We could do this 100% automagically in python_pkg_setup, which was my 
initial plan.  But my test-case involves LDPATH being "scrubbed" in 
every phase by profile.bashrc.  After python_pkg_setup() changed LDPATH, 
it just broke again in subsequent phases (at least, I think that's what 
would have happened -- I never tested it).  Yes, that's a GIGO problem; 
however, it's debatable whether Gentoo or profile.bashrc is the one 
putting the "G)arbage I)n".  The most robust route is to insert our 
LDFLAG just before econf runs -- that's how the current implementation 
does it, after all, and we potentially create a regression if we move 
this logic to pkg_setup.

Also, I decided to leave my internal API as "_python_foo," although 
today, I'd say "__python_foo" is more correct.  python.eclass is chock 
full of "_python_foo" API's, so I decided it was probably cleaner to 
follow the existing patterns for now, rather than change one lone API 
while the rest remain unchanged.

As for cpython_econf -- is it OK that it doesn't begin with "python"? 
Or should it perhaps be "python_cpython_econf?"

Thanks for your input,

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 973 bytes --]

On Wednesday 10 October 2012 18:47:46 Gregory M. Turner wrote:
> +	if [[ ${CHOST} == *-cygwin* ]] ; then
> +		fpeconfig="--without-fpectl"

just re-use myconf.  this is what it's for.

> +		myconf="${myconf} ac_cv_func_bind_textdomain_codeset=yes"

just export it:
	export ac_cv_func_bind_textdomain_codeset=yes

> As for cpython_econf -- is it OK that it doesn't begin with "python"?
> Or should it perhaps be "python_cpython_econf?"

i think it's all a giant hack for a problem i have yet to see described where 
the user wasn't doing something wrong.  it's one thing to add LDFLAGS="-L. 
${LDFLAGS}" to support their dumbness, but it's a completely different ballgame 
to write an entire flag parser (which i'm going to assume isn't even really 
necessary or covers all possible angles).

another alternative is to modify the python build system so that it uses the 
full path to the library directly rather than searching for it via -L ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/9/2012 2:26 PM, Mike Frysinger wrote:
> On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:
>> My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum
>> opus above and allow me to try again.
>>
>> In dev-lang/python*, we use
>>
>>     append-ldflags '-L.'
>>
>> to ensure linking is performed against the built libpython.so in-tree,
>> rather than than in the one in $(libdir).  But, this doesn't work if
>> LDFLAGS contains "-L$(libdir)".
>
> well, some notes here ...
>
> for users: putting -L<some-system-path> is wrong.  it changes the meaning of
> "system paths" when it comes to searching for linking by elevating it to "user
> specified path".  imo, you should be fixing the source rather than the symptom.
>
> for packagers: using -L$(libdir) is almost always wrong and pointless.  the
> toolchain itself knows the proper system path to search for libraries to link
> against, and when you cross-compile, the --libdir you specify to configure is
> not going to be valid when dereferenced on the build system.

Agreed on these points.  Not until we combine the doubly broken 
circumstances of (1) User specified -L/usr/$(libdir) in LDPATH and (2) 
an ebuild that puts -L. into LDPATH to fix Makefile recipes that appear 
more-or-less broken, do we get this conflict.  This leads to a certain 
odor of "unsupportable circumstances" which, I must admit, makes me 
slightly uncomfortable trying to fix it.

Breaking this down:

(1) is worse than (2), but it does have some quasi-legitimate usages. 
For example, prefix bootstrap does this (or used to), as do many of the 
crossdev-wrapper scripts.  I've also resorted to such usage, myself, 
when repairing a prefix after I've broken its compiler.  These cases 
don't really seem completely "correct" or "incorrect" -- about the best 
I can say for them it is that they are mostly efficacious, but prone to 
problems.

As for (2)... meh.  LDFLAGS, to my thinking, means something like:  "Use 
these guidelines and parameters for linking, in preference to the 
default guidelines that the build process would normally use."  LDFLAGS 
coming from make.conf have a slightly different meaning, I guess, but 
the gist of it is the same.

So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a 
complete abomination.  Still... I don't like it.

If the Makefiles are building against libraries expected to be in 
${PWD}, it seems to me that the Makefiles should know to look there 
automatically.

In the particular case of Gentoo's cpython, I have to admit I'm 
concerned that if I start mucking around with the Makefiles/autotool 
scaffolding, I'm going to break stuff.  But maybe it's worth taking that 
risk if it allows us to do things more correctly going forward... I 
could at least take a crack at it and see what people think of the results.

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Diego Elio Pettenò]

On 10/10/2012 20:37, Gregory M. Turner wrote:
> 
> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.

Using the -L . -lfoo is a very bad style in general. Just think what
happen if you're trying to link to a libutil.a

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 2530 bytes --]

On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> (1) is worse than (2), but it does have some quasi-legitimate usages.
> For example, prefix bootstrap does this (or used to), as do many of the
> crossdev-wrapper scripts.  I've also resorted to such usage, myself,
> when repairing a prefix after I've broken its compiler.  These cases
> don't really seem completely "correct" or "incorrect" -- about the best
> I can say for them it is that they are mostly efficacious, but prone to
> problems.

they're broken.  and no, crossdev doesn't do this.  it is properly sysrooted.

further, you cannot do multilib with users putting -L<system libdir> into 
their LDFLAGS.  they just forced a single ABI and any other will simply fail.  
the toolchain's compiler driver knows exactly where to find its own libraries.  
if it doesn't, it's broken, and it should be fixed.

> So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
> complete abomination.  Still... I don't like it.

it's significantly better than the proposed code which tries to parse the 
meaning of various flags and insert it at the "right" point.  i can't see it 
being useful at all.  it's a recipe for fragility and being unmaintainable.

> If the Makefiles are building against libraries expected to be in
> ${PWD}, it seems to me that the Makefiles should know to look there
> automatically.

yes, so why is the ebuild specifying -L. for it ?  the package should already 
have a -L to the right place, and if it wants to make sure it gets used before 
the user's LDFLAGS, it should show up in the linking before it (or have the 
build system prepend the value to LDFLAGS if it's appending currently).

> In the particular case of Gentoo's cpython, I have to admit I'm
> concerned that if I start mucking around with the Makefiles/autotool
> scaffolding, I'm going to break stuff.  But maybe it's worth taking that
> risk if it allows us to do things more correctly going forward... I
> could at least take a crack at it and see what people think of the results.

i'm not sure why this applies to the larger build system.  i can understand 
that the python build itself might not be putting the -L. in a place where a 
user's misconfigured LDFLAGS will cause a problem.

but if you want the right answer, it's to reject broken user LDFLAGS.  we 
don't support wrong things like -fPIC or -fPIE in global CFLAGS.  i don't 
think we should be wasting time on LDFLAGS like this either.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
>> (1) is worse than (2), but it does have some quasi-legitimate usages.
>> For example, prefix bootstrap does this (or used to), as do many of the
>> crossdev-wrapper scripts.  I've also resorted to such usage, myself,
>> when repairing a prefix after I've broken its compiler.  These cases
>> don't really seem completely "correct" or "incorrect" -- about the best
>> I can say for them it is that they are mostly efficacious, but prone to
>> problems.
>
> and no, crossdev doesn't do this.  it is properly sysrooted.

You are right.  I guess I was thinking of this:

http://tinderbox.x86.dev.gentoo.org/misc/cmerge,

Which does use -L but looks pretty ancient.  I see now that crossdev's 
equivalent hacks up the linker scripts -- which seems as "correct" as 
one could reasonably hope for... pretty sexy.

> further, you cannot do multilib with users putting -L<system libdir> into
> their LDFLAGS.  they just forced a single ABI and any other will simply fail.
> the toolchain's compiler driver knows exactly where to find its own libraries.
> if it doesn't, it's broken, and it should be fixed.

ACK

>> So technically speaking, LDFLAGS="-L. ${LDFLAGS}" or similar is not a
>> complete abomination.  Still... I don't like it.
>
> it's significantly better than the proposed code which tries to parse the
> meaning of various flags and insert it at the "right" point.  i can't see it
> being useful at all.  it's a recipe for fragility and being unmaintainable.

I think I meant the above differently than you interpreted it: I meant 
that solving library-search-path selection in autotools steps and/or 
Makefile.in would make the question of how to manipulate LDFLAGS 
academic, and seems more elegant.  Apparently you agree:

>> If the Makefiles are building against libraries expected to be in
>> ${PWD}, it seems to me that the Makefiles should know to look there
>> automatically.
>
> yes, so why is the ebuild specifying -L. for it ?  the package should already
> have a -L to the right place, and if it wants to make sure it gets used before
> the user's LDFLAGS, it should show up in the linking before it (or have the
> build system prepend the value to LDFLAGS if it's appending currently).

If we decide that manipulating LDFLAGS is correct, whether we do this 
using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.

My point above was that coming to that decision in the first place 
suggests some underlying issue exists that arguably ought to be fixed 
instead.

> i'm not sure why this applies to the larger build system.  i can understand
> that the python build itself might not be putting the -L. in a place where a
> user's misconfigured LDFLAGS will cause a problem.

This only pertains to dev-lang/python AFAIK... there may be others 
lurking out there somewhere, but I'm not too worried about them.  The 
use-cases I have in mind are prefix bootstrap and system-recovery when 
gcc breaks.

In general, if python breaks, folks can end up in pretty bad shape due 
to chicken-and-egg issues with portage.  That's the only reason I've 
spent so much effort trying to think this fairly obscure corner-case 
through.

-gmt

P.S.: After poking around, the first commit where this appeared is this 
one: 
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/python/python-2.5.1.ebuild?hideattic=0&r1=1.4&r2=1.5&view=patch, 
a mere 5.5 years ago by uberlord.

The cvs ChangeLog references bug 177426 which doesn't seem particularly 
illuminating.  It also says:

   export LDFLAGS=-L. so we link modules correctly on FreeBSD
   and possibly other systems where python2.5 isn't installed yet.

Which at least seems to confirm that this was originally a problem not 
of search path ordering, but of complete failure to search ${S} in the 
first place.

Meanwhile, I've looked into the patches a bit.  In regular Gentoo 
nothing special happens to LDFLAGS, but in prefix, there is special 
handling that suggests some avenues of attack.  Before I do anything 
further, I want to figure out if it works right without LDPATH 
manipulation on linux, and, if so, whether I can't generalize whatever 
mechanism allows that to happen.

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 3322 bytes --]

On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
> On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> > On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:
> >> (1) is worse than (2), but it does have some quasi-legitimate usages.
> >> For example, prefix bootstrap does this (or used to), as do many of the
> >> crossdev-wrapper scripts.  I've also resorted to such usage, myself,
> >> when repairing a prefix after I've broken its compiler.  These cases
> >> don't really seem completely "correct" or "incorrect" -- about the best
> >> I can say for them it is that they are mostly efficacious, but prone to
> >> problems.
> > 
> > and no, crossdev doesn't do this.  it is properly sysrooted.
> 
> You are right.  I guess I was thinking of this:
> 
> http://tinderbox.x86.dev.gentoo.org/misc/cmerge,
> 
> Which does use -L but looks pretty ancient.  I see now that crossdev's
> equivalent hacks up the linker scripts -- which seems as "correct" as
> one could reasonably hope for... pretty sexy.

by default it doesn't.  with the elibtoolize hacks and the drive for .pc files, 
this has been largely mitigated.  although the latest libtool has a --with-
sysroot option that needs investigating.

> >> If the Makefiles are building against libraries expected to be in
> >> ${PWD}, it seems to me that the Makefiles should know to look there
> >> automatically.
> > 
> > yes, so why is the ebuild specifying -L. for it ?  the package should
> > already have a -L to the right place, and if it wants to make sure it
> > gets used before the user's LDFLAGS, it should show up in the linking
> > before it (or have the build system prepend the value to LDFLAGS if it's
> > appending currently).
> 
> If we decide that manipulating LDFLAGS is correct, whether we do this
> using my "parser" or LDFLAGS="-L. ${LDFLAGS}" isn't particularly important.

it's not particularly important, but on one hand, the LDFLAGS parsing logic 
should not be in the tree ever.  on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python 
ebuild really really want to add it.

i'd still lobby for rejecting of invalid LDFLAGS settings and fixing things at 
the "right" place.

> In general, if python breaks, folks can end up in pretty bad shape due
> to chicken-and-egg issues with portage.  That's the only reason I've
> spent so much effort trying to think this fairly obscure corner-case
> through.

that's really only a problem in the bootstrap case.  if an existing system 
can't install python, then it isn't like they're screwed since they already 
have a working python installed.

> Meanwhile, I've looked into the patches a bit.  In regular Gentoo
> nothing special happens to LDFLAGS, but in prefix, there is special
> handling that suggests some avenues of attack.  Before I do anything
> further, I want to figure out if it works right without LDPATH
> manipulation on linux, and, if so, whether I can't generalize whatever
> mechanism allows that to happen.

my [limited] understanding of the prefix compiler is that they wrap things with 
a custom shell script to inject -L flags behind the back of the compiler at the 
very last possible minute and point to the right place.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/11/2012 8:50 AM, Mike Frysinger wrote:
> On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:
>> On 10/10/2012 9:14 PM, Mike Frysinger wrote:
> it's not particularly important, but on one hand, the LDFLAGS parsing logic
> should not be in the tree ever.

I've no major attachment to it.  Took all of five minutes to code up 
reading the ld manpage, and as you pointed out, it's probably solving a 
non-problem.

> on the other, i can stomach much smaller one-
> off hacks like prepending -L. to LDFLAGS if the maintainers of the python
> ebuild really really want to add it.

If fixing the python builds proves too onerous then this is what I'll 
end up filing a bug for.

> my [limited] understanding of the prefix compiler is that they wrap things with
> a custom shell script to inject -L flags behind the back of the compiler at the
> very last possible minute and point to the right place.

Pretty fuzzy on this myself.  Whatever binutils-config does works so 
well I've never had to look into it very deeply :)

Above, however, I'm referring to the prefix-specific cpython patch-sets, 
where I was "sure" I'd seen patches to project LDFLAGS into setup.py's 
compiler invocation tables.  But it seems I was mistaken.  Perhaps it 
was in the main gentoo-patches after all... or maybe I just need more 
coffee...

Anyhow one thing I have figured out is how things can work correctly on 
Linux wihtout -L.: on Linux, the python plugins aren't actually linked 
against libpython.so!

With any luck, this explains what's going on and suggests that platforms 
linking python modules against libpython.so just need to wedge -L${S} 
somewhere in configure.in, possibly do-able via sed script (unless we 
end up having to mess with setup.py, in which case, things may get a bit 
more complicated).

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Marien Zwart]

I'm going to do something potentially rude and comment on this without
having read the entire thread.

On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:
> Anyhow one thing I have figured out is how things can work correctly on
> Linux wihtout -L.: on Linux, the python plugins aren't actually linked
> against libpython.so!

Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.

However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.

The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).

-- 
Marien Zwart.

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/11/2012 2:40 PM, Marien Zwart wrote:
> I'm going to do something potentially rude and comment on this without
> having read the entire thread.
>
> On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner <gmt@malth.us> wrote:
>> Anyhow one thing I have figured out is how things can work correctly on
>> Linux wihtout -L.: on Linux, the python plugins aren't actually linked
>> against libpython.so!
>
> Python can be built with and without a shared library
> (libpythonx.y.so). If the shared library is built the interpreter
> executable is linked against it (of course), and normally so are the
> extension modules. This is useful because it makes the linker do the
> right thing for applications that embed python (and end up loading
> libpythonx.y via dlopen). If the shared library is not built extension
> modules cannot link against it, and the python executable itself
> exports the relevant symbols. So on a normal shared python build
> extensions should end up linked against libpythonx.y.so.
>
> However, at least one popular distro and presumably most of its
> derivatives (Debian) has a system python built without the shared
> library, plus the shared library built separately, for use by
> applications that embed python. I am not entirely sure why they still
> do this (if I understand correctly it was originally done for
> performance reasons, but I'm not sure if they've checked those are
> still valid). Their extension modules cannot link to libpythonx.y:
> their python interpreter executable provides the same symbols
> libpythonx.y provides, so loading such an extension module into the
> regular interpreter would result in collisions.
>
> The reason I mention this is Gentoo's python is built with the shared
> library, so on Gentoo your extensions *should* end up linked to
> libpythonx.y (if they use a more or less standard build system, at
> least). If yours are not I'm curious what happened, and would
> appreciate it if you gave me some more information off-list (marienz
> on freenode or email).

Thanks a ton for this info, that explains a lot.  Actually, got this 
notion testing outside of Gentoo, building from a vanilla python 3.3.0 
tarball on a Fedora box.  I wanted to see what happens in a completely 
"Gentoo-free" environment so as to get some kind of baseline 
understanding of how upstream is propagating stuff through the Makefiles 
to setup.py.

Probably, I screwed up ./configure (I tried to approximate Gentoo's 
arguments), and requested a non-shlib build.  Guess it would have been 
clearer if I'd mentioned that in my post -- sorry if I worried you :)

I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever 
see it fail to do as you describe, I'll let you know, Marien.

My i686 prefix is up to date and I can confirm that it is linking 
against libpython there.

But... wow.  In my almost-vanilla i686 cross-prefix (hosted by my 
only-moderately-rice multilib ~amd64 workstation), in 
=dev-lang/python-2.7.3-r2, the modules are linked like so:

   i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 \
	-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
	-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
	-L. \
	-Wl,-O1 \
	-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
	-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
	-L. \
	-fno-strict-aliasing \
	-O2 -march=i686 -pipe -m32 \
	-fwrapv -DNDEBUG \
	-I. -IInclude -I./Include \
	-I${EPREFIX}/usr/include \
	build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o \
	build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o \
	-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
	-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
	-L. \
	-lm -lpython2.7 \
	-o build/lib.linux-i686-2.7/math.so

Which is just shameful.  In this case, I'm pretty sure all those 
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:

   [PREFIX] $ emerge --info --verbose | grep /lib
   sys-devel/libtool:        2.4.2::gentoo_prefix
   COLLISION_IGNORE="/lib/modules/* *.py[co]"
   PKG_CONFIG_PATH="${EPREFIX}/usr/lib/pkgconfig:$( :;
                   )${EPREFIX}/usr/share/pkgconfig"
   PORTAGE_BIN_PATH="${EPREFIX}/usr/lib/portage/bin"
   PORTAGE_PYM_PATH="${EPREFIX}/usr/lib/portage/pym"
   UNINSTALL_IGNORE="/lib/modules/*"

(snippets edited for readability obv).

Regarding the gcc-as-linker invocation above:

First, something puts in all kinds of inappropriate amd64 multilib paths 
(this ends up being harmless as wrong-arch libraries get rejected at 
link-time and treated as non-matches for -lclauses... still, WTF?).

Secondly, something puts the built-in ld search paths into the 
command-line ld search path (a practice which has been roundly 
disparaged in this thread, and correctly so, IMO).

Thirdly, prefix x86-linux also clearly suffers from the original problem 
I was seeing in cygwin, and which inspired this thread (putting -L. 
after $(libdir) paths and therefore linking against the wrong 
libpython.so).  I never suspected this applied to every prefix python 
ebuild, but that's what the above seems to suggest... not cool.

Plus, it repeats itself like a stuttering Makefool -- but that's pretty 
standard and only ever seems to bother me :)

Perhaps, this is a consequence of mine being a cross-prefix, somehow. 
That's the only thing significantly off-the-beaten-path thing about it. 
  Even if so, I'd like amd64->i686 cross-prefixes to work and don't see 
why they shouldn't.

The only good news I can report is that changing

   append-ldflags "-L."

to

   export LDFLAGS="-L. ${LDFLAGS}"

does seem to fix the third issue, above.

Looks like, at least for prefix, it's going to take more than the quick 
configure patch I was hoping for, if we're to make things fully right. 
We'll see soon enough about ~amd64 linux, and I'm spinning up a 
freebsd90 vm to kick around as well.

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/12/2012 4:03 AM, Gregory M. Turner wrote:
> First, something puts in all kinds of inappropriate amd64 multilib paths
> (this ends up being harmless as wrong-arch libraries get rejected at
> link-time and treated as non-matches for -lclauses... still, WTF?).
>
> Secondly, something puts the built-in ld search paths into the
> command-line ld search path (a practice which has been roundly
> disparaged in this thread, and correctly so, IMO).
>
> Thirdly, prefix x86-linux also clearly suffers from the original problem
> I was seeing in cygwin, and which inspired this thread (putting -L.
> after $(libdir) paths and therefore linking against the wrong
> libpython.so).  I never suspected this applied to every prefix python
> ebuild, but that's what the above seems to suggest... not cool.

Just wanted to update this thread with latest current triage and 
prognosis data for this patient:

Regarding etiology, for "Firstly" and "Secondly" above: it seems that 
these are by design (which is not to say that they should be left: in 
each case, the "design" is pretty bad and ought to be fixed, I think).

My hope is that "Firstly" and "Secondly" are needed only to accommodate 
prefix bootstrap, in which case we can fix them by detecting whether a 
prefix bootstrap is ongoing or not, and conditioning these hacks on that 
(improving "Firstly" to not be braindead would be a nice touch, although 
doing so in a way that doesn't break anything may be prohibitively 
difficult or beyond my abilities).

"Thirdly" has been addressed ad nauseam in this thread and will be 
solved by prepending the LDFLAG rather than appending, or, preferably, 
by patching autotools (but only if I can find a simple, low-maintenance 
approach that is likely to work without building any new per-platform 
matrices or case-statements).

Note that, even without this third fix, absent a PEBKAC placement of 
"-L${EPREFIX}/usr/$(libdir)" into LDFLAGS, either:

o it's a prefix bootstrap, in which case there probably is no
   ${EPREFIX}/usr/$(libdir)/libpython${PV}.so, or

o it's /not/ a prefix bootstrap, in which case the above-mentioned
   fixes would prevent the "-L${EPREFIX}/usr/$(libdir)" from
   getting into LDFLAGS, thus preventing the problem from arising.

So by fixing "Firstly" and "Secondly," we have at least brought things 
sufficiently in line with expectations that we are only fixing 
user-level configuration mistakes instead of our own, as originally 
expected. **

I'm feeling ready, at this point, to file bugs for these issues; once I 
do, I'll post bug#'s into this thread and plan to take all off this 
chatter off-list, unless something more comes up that seems to require 
community review.

Thanks for everyone's help figuring this stuff out,

-gmt

** I suppose a third possibility is that someone is doing an in-place 
"re-bootstrap" or restarting a bootstrap build that failed mid-merge, 
but ... doesn't matter, it'll be academic once we fix the bug.

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 784 bytes --]

On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> "Thirdly" has been addressed ad nauseam in this thread and will be
> solved by prepending the LDFLAG rather than appending, or, preferably,
> by patching autotools (but only if I can find a simple, low-maintenance
> approach that is likely to work without building any new per-platform
> matrices or case-statements).

i'm fairly certain this isn't autotools.  i've poked around the python build 
system before in the past and while it uses autoconf to do platform tests, it 
doesn't use automake/libtool.  make is used to bootstrap python, and then they 
descend into a horrible hack of a custom build system written in python.  i 
suspect much of the pain you're seeing is coming from that last part.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
>> "Thirdly" has been addressed ad nauseam in this thread and will be
>> solved by prepending the LDFLAG rather than appending, or, preferably,
>> by patching autotools (but only if I can find a simple, low-maintenance
>> approach that is likely to work without building any new per-platform
>> matrices or case-statements).
>
> i'm fairly certain this isn't autotools.  i've poked around the python build
> system before in the past and while it uses autoconf to do platform tests, it
> doesn't use automake/libtool.  make is used to bootstrap python, and then they
> descend into a horrible hack of a custom build system written in python.  i
> suspect much of the pain you're seeing is coming from that last part.
> -mike

I guess I should say "in src_prepare" rather than "in autotools."

Specifically, I was thinking some kind of configure.in patch might be 
good, since configure.in seems to churn less than Makefile.in, a good 
thing if I want to produce a one-size-fits-all patch.

And, yeah, setup.py is definitely behind all this yucky.

Python clearly has an amazing community, so I hate to say anything 
negative... but I sometimes wish they would "build" less and "buy" more.

Anyhow, as everyone knows, bitching and moaning about FOSS is pointless 
so I'll stop there and spend my time fixing stuff instead :)

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1519 bytes --]

On Monday 15 October 2012 04:35:09 Gregory M. Turner wrote:
> On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> > On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:
> >> "Thirdly" has been addressed ad nauseam in this thread and will be
> >> solved by prepending the LDFLAG rather than appending, or, preferably,
> >> by patching autotools (but only if I can find a simple, low-maintenance
> >> approach that is likely to work without building any new per-platform
> >> matrices or case-statements).
> > 
> > i'm fairly certain this isn't autotools.  i've poked around the python
> > build system before in the past and while it uses autoconf to do
> > platform tests, it doesn't use automake/libtool.  make is used to
> > bootstrap python, and then they descend into a horrible hack of a custom
> > build system written in python.  i suspect much of the pain you're
> > seeing is coming from that last part. -mike
> 
> And, yeah, setup.py is definitely behind all this yucky.
> 
> Python clearly has an amazing community, so I hate to say anything
> negative... but I sometimes wish they would "build" less and "buy" more.

build systems are hard to get right.  python is in the situation where the 
setups they care about mostly work and people generally aren't complaining, 
but it's more through a hack effort than doing it right which means all the 
other cases they haven't considered break horribly.  cross-compiling for 
example has never worked correctly out of the box.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Marien Zwart]

On Monday 15 October 2012 04:35:09 Gregory M. Turner wrote:
> On 10/14/2012 9:29 PM, Mike Frysinger wrote:
> > Python clearly has an amazing community, so I hate to say anything
> > negative... but I sometimes wish they would "build" less and "buy" more.
>
> build systems are hard to get right.  python is in the situation where the
> setups they care about mostly work and people generally aren't complaining,
> but it's more through a hack effort than doing it right which means all the
> other cases they haven't considered break horribly.  cross-compiling for
> example has never worked correctly out of the box.

The reasons for Python's awkward build system are probably mostly
historical. I'll (unsuccessfully) try to keep this brief, but still:

Python is old: Python 1.0.0 was released January 1994, and Python
existed before that. I don't know what its build system was like back
then, but most systems you might want them to "buy" didn't exist yet.

Distutils (the build system for external extension modules as well as
chunks of Python's standard library) is still old: it was integrated
into Python 1.6 (September 5, 2000). Autoconf 2.13 existed, 2.50
wasn't there yet. Looks like CMake was just turned into a separate
project around that time.

Python's needs are a little special: you can build extension modules
using just Python and a C compiler (and you don't need the C compiler
if your code is pure-python). This is important on awkward platforms
(currently mostly Windows, but I'm pretty sure there were others
similarly awkward). This same Python-powered build system is used to
build part of the standard library, for obvious reasons. And there are
other build systems that make use of python mainly to be
cross-platform (scons and derivatives mostly).

It would probably be possible to redo Python's own build system now
and end up with something better, but I'm not aware of anyone having
seriously attempted this, probably because it's just not quite broken
enough to be worth it. The awkwardness of distutils we're somewhat
stuck with, although there are efforts to modernize and clean up
distutils. I'm not so sure there's anything they could have "bought"
to replace Python's own build system, let alone distutils (build
systems like automake actually use distutils.sysconfig), without a
working time machine (and a rather confusing time loop to allow build
systems that currently support python to do so without
distutils.sysconfig).

-- 
marienz.