[gentoo-dev] perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[gentoo-dev] perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 4655 bytes --]


TL;DR: 

Please review and test (also in exotic settings, arches, prefix?) the patch 
below for perl-module.eclass

Sending this to the list since perl is pretty deep down in the system, and 
since I'm not 100% sure of this stuff. Needs review.

Also people familiar with Perl have voiced doubts if we should meddle with the 
perl-module CFLAGS and LDFLAGS at all, maybe making these changes conditional 
on something like I_KNOW_WHAT_I_AM_DOING. Hoping for comments.

Detail discussion and test results follows below the patch.

=======================================

Index: perl-module.eclass
===================================================================
RCS file: /var/cvsroot/gentoo-x86/eclass/perl-module.eclass,v
retrieving revision 1.139
diff -u -B -r1.139 perl-module.eclass
--- perl-module.eclass	30 Mar 2014 19:25:14 -0000	1.139
+++ perl-module.eclass	22 Jun 2014 09:35:16 -0000
@@ -139,6 +139,8 @@
 			--libdoc= \
 			--destdir="${D}" \
 			--create_packlist=0 \
+			--config "optimize=${CFLAGS}" \
+			--config "lddlflags=-shared ${LDFLAGS}" \
 			"${myconf_local[@]}"
 		einfo "perl Build.PL" "$@"
 		perl Build.PL "$@" <<< "${pm_echovar}" \
@@ -150,6 +152,8 @@
 			INSTALLDIRS=vendor \
 			INSTALLMAN3DIR='none' \
 			DESTDIR="${D}" \
+			OPTIMIZE="${CFLAGS}" \
+			LDDLFLAGS="-shared ${LDFLAGS}" \
 			"${myconf_local[@]}"
 		einfo "perl Makefile.PL" "$@"
 		perl Makefile.PL "$@" <<< "${pm_echovar}" \
@@ -178,12 +182,12 @@
 			|| die "Compilation failed"
 	elif [[ -f Makefile ]] ; then
 		set -- \
-			OTHERLDFLAGS="${LDFLAGS}" \
+			OPTIMIZE="${CFLAGS}" \
+			LDDLFLAGS="-shared ${LDFLAGS}" \
 			"${mymake_local[@]}"
 		einfo "emake" "$@"
 		emake "$@" \
 			|| die "Compilation failed"
-#			OPTIMIZE="${CFLAGS}" \
 	fi
 }
 

=======================================

The details: 

Right now, if you build perl modules, e.g. ExtUtils::MakeMaker imports the 
CFLAGS and LDFLAGS used when building dev-lang/perl. They are stored for 
example in /usr/lib*/perl5/*/*/Config_heavy.pl in the "optimize" and 
"lddlflags" variable. 

The CFLAGS variable from the environment is not used at all; the cause for 
this is so far unknown. (If things were working "according to spec", they 
would be appended behind the "optimize" values... which is not really what we 
want either.)

Once OPTIMIZE="${CFLAGS}" is set, it overwrites the value from 
Config_heavy.pl. This means modules get built with the current CFLAGS. 

We can also overwrite lddlflags with the make.conf LDFLAGS by setting the 
mystic environment variable LDDLFLAGS (there's more than one way to do it!), 
but for this to work I have to manually add the parameter "-shared". 

** I'm assuming here that Perl wants to have shared library objects everywhere 
and that this parameter is accepted everywhere. If that is not true, please 
suggest an improvement or cry "foul" ... :| **

https://bugs.gentoo.org/show_bug.cgi?id=261375

=======================================

Testing done 
(in a chroot, amd64, stable except perl modules that are only ~arch and their 
deps): 

1) built a module using ExtUtils::MakeMaker against "perl with wrong flags" 
and checked log for flags (perl-core/Encode)
2) built a module using ModUtils::Build against "perl with wrong flags" and 
checked log for flags (dev-perl/Class-Load-XS)
3) built (nearly) everything using perl-modules.eclass (~1400 packages, either 
newest stable or if no stable ebuild newest ~arch) and checked for obvious 
problems by grepping the logs

Test results from 1)
All good.

Test results from 2)
Both CFLAGS and LDFLAGS are (still) inserted double. Reason so far unclear, 
odd but not problematic.

Test results from 3)

Before patching eclass:
240 packages throwing CFLAGS QA warning

After modification, the following packages still throw CFLAGS QA warning:
app-mobilephone/obexftp-0.23-r2
dev-perl/Wx-Scintilla-0.390.0
media-libs/ming-0.4.4
net-analyzer/net-snmp-5.7.3_pre3
net-analyzer/rrdtool-1.4.8
www-apache/libapreq2-2.13
www-apache/mod_perl-2.0.7

After modification, *IN ADDITION TO THOSE ABOVE* the "bad" flags occur in the 
logs of the following packages:
app-pda/pilot-link-0.12.5
dev-libs/stfl-0.22-r1
dev-perl/Text-BibTeX-0.690.0
dev-perl/math-pari-2.10.806.50
dev-vcs/subversion-1.7.14
games-arcade/frozen-bubble-2.2.1_beta1
sci-biology/vienna-rna-1.8.5
sci-libs/gdal-1.10.1
sys-fs/owfs-2.7_p21-r2
sys-libs/libapparmor-2.8.3

No additional build failures.

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

Dnia 2014-06-22, o godz. 11:41:01
"Andreas K. Huettel" <dilfridge@gentoo.org> napisał(a):

> Also people familiar with Perl have voiced doubts if we should meddle with the 
> perl-module CFLAGS and LDFLAGS at all, maybe making these changes conditional 
> on something like I_KNOW_WHAT_I_AM_DOING. Hoping for comments.

Making them conditional would make them pointless.

Also, how about fixing CC and CXX? That's more important to me than
some disrespected CFLAGS :P.

> Detail discussion and test results follows below the patch.
> 
> =======================================
> 
> Index: perl-module.eclass
> ===================================================================
> RCS file: /var/cvsroot/gentoo-x86/eclass/perl-module.eclass,v
> retrieving revision 1.139
> diff -u -B -r1.139 perl-module.eclass
> --- perl-module.eclass	30 Mar 2014 19:25:14 -0000	1.139
> +++ perl-module.eclass	22 Jun 2014 09:35:16 -0000
> @@ -139,6 +139,8 @@
>  			--libdoc= \
>  			--destdir="${D}" \
>  			--create_packlist=0 \
> +			--config "optimize=${CFLAGS}" \
> +			--config "lddlflags=-shared ${LDFLAGS}" \

Tadaam, you just broke Darwin like we did! Look at [1] for some clues,
though perl may need different values.

[1]:https://bugs.gentoo.org/show_bug.cgi?id=513664#c7

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Duncan]

Andreas K. Huettel posted on Sun, 22 Jun 2014 11:41:01 +0200 as excerpted:

> Also people familiar with Perl have voiced doubts if we should meddle
> with the perl-module CFLAGS and LDFLAGS at all, maybe making these
> changes conditional on something like I_KNOW_WHAT_I_AM_DOING. Hoping for
> comments.

The usual conditional for that is USE=custom-cflags or a similar variant 
like custom-optimization.  See the firefox ebuilds, which use both.

$ equery -N u firefox | grep custom
 - - custom-cflags        : Build with user-specified CFLAGS (unsupported)
 + + custom-optimization  : Fine-tune custom compiler optimizations (-Os, 
-O0, -O1, -O2, -O3)


Generally, such USE flags are discouraged in gentoo, where honoring a 
user's CFLAGS, etc. and letting them keep the pieces if it comes to that 
is the norm.  Where upstream refuses to support custom cflags, however, 
the USE flag solution seems to be an accepted at maintainer discretion.

(FWIW I have USE=custom-cflags set globally, but package.use-disabled it 
for firefox when firefox-28 or so decided it didn't like the ones I was 
using.  At some point I'll probably go back and figure out which one it 
is and filter it specifically, but unsetting that USE flag was both a 
useful first-step debug and a reasonable quick fix for the time being.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On 23 June 2014 01:02, Duncan <1i5t5.duncan@cox.net> wrote:

>
> The usual conditional for that is USE=custom-cflags or a similar variant
> like custom-optimization.  See the firefox ebuilds, which use both.
>
> $ equery -N u firefox | grep custom
>  - - custom-cflags        : Build with user-specified CFLAGS (unsupported)
>  + + custom-optimization  : Fine-tune custom compiler optimizations (-Os,
> -O0, -O1, -O2, -O3)
>
>
> Generally, such USE flags are discouraged in gentoo, where honoring a
> user's CFLAGS, etc. and letting them keep the pieces if it comes to that
> is the norm.  Where upstream refuses to support custom cflags, however,
> the USE flag solution seems to be an accepted at maintainer discretion.



The problem with a USE flag here as such is the change this eclass makes
affects >900 packages in dev-perl/ , and its not easy to casually determine
which of those packages have any C bits where CFLAGS are even meaningful.

Which would mean having >900 packages with a new IUSE value that did
nothing for a majority of them.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1570 bytes --]

Re: [gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Rich Freeman]

On Sun, Jun 22, 2014 at 9:11 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> On 23 June 2014 01:02, Duncan <1i5t5.duncan@cox.net> wrote:
>>
>>
>> The usual conditional for that is USE=custom-cflags or a similar variant
>> like custom-optimization.  See the firefox ebuilds, which use both.
>>
>> $ equery -N u firefox | grep custom
>>  - - custom-cflags        : Build with user-specified CFLAGS (unsupported)
>>  + + custom-optimization  : Fine-tune custom compiler optimizations (-Os,
>> -O0, -O1, -O2, -O3)
>>
>>
>> Generally, such USE flags are discouraged in gentoo, where honoring a
>> user's CFLAGS, etc. and letting them keep the pieces if it comes to that
>> is the norm.  Where upstream refuses to support custom cflags, however,
>> the USE flag solution seems to be an accepted at maintainer discretion.
>
> The problem with a USE flag here as such is the change this eclass makes
> affects >900 packages in dev-perl/ , and its not easy to casually determine
> which of those packages have any C bits where CFLAGS are even meaningful.
>
> Which would mean having >900 packages with a new IUSE value that did nothing
> for a majority of them.

Would it make more sense to move filtering (and thus the flag) to
packages that actually use it, instead of doing it in the eclass?  By
the same logic you have >900 packages filtering CFLAGS when many of
them don't even use CFLAGS.

I definitely wouldn't go the I_KNOW_WHAT_IM_DOING route for something
like flag filtering on something like perl.  Before you know it
everybody will just stick it in their defaults and it will lose all
usefulness.  That should be reserved for things with are somewhat
likely to seriously cripple the system in a manner that isn't easy to
recover from.

Or, perhaps do this in the eclass, but wait until a big perl upgrade
or something to introduce the change so that everybody using emerge -N
doesn't have to re-install all of perl unnecessarily.

Not a pretty solution any which way, but neither is filtering flags at
such a large scale.  It would make much more sense to filter them when
they're known to not work.  Upstream isn't going to "support" Gentoo
no matter what we do anyway (whatever support even means when you're
not paying them).

Rich

Re: [gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 1484 bytes --]

Am Sonntag, 22. Juni 2014, 15:31:43 schrieb Rich Freeman:
> On Sun, Jun 22, 2014 at 9:11 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> > On 23 June 2014 01:02, Duncan <1i5t5.duncan@cox.net> wrote:
> >> The usual conditional for that is USE=custom-cflags or a similar variant
[...]
> > The problem with a USE flag here as such is the change this eclass makes
> > affects >900 packages in dev-perl/ , and its not easy to casually
> > determine which of those packages have any C bits where CFLAGS are even
> > meaningful.
> > 
> > Which would mean having >900 packages with a new IUSE value that did
> > nothing for a majority of them.

The numbers are even worse; the use flag would be added to ~1400 packages, and 
it would actually matter for only ~300.

> Would it make more sense to move filtering (and thus the flag) to
> packages that actually use it, instead of doing it in the eclass?  By
> the same logic you have >900 packages filtering CFLAGS when many of
> them don't even use CFLAGS.

Offhand I don't know any way how the eclass could "know" whether a perl 
package requires C building or not. Any heuristics would be a hack at best. It 
might even be preferable to manually add some variable to 300 packages in the 
ebuild (which then makes the eclass add a use-flag custom-cflags).
i.e.

PERL_CFLAGS_HANDLING=yes
inherit perl-module

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 801 bytes --]

Am Sonntag, 22. Juni 2014, 11:54:33 schrieb Michał Górny:
> 
> Also, how about fixing CC and CXX? That's more important to me than
> some disrespected CFLAGS :P.

Easily doable, I'll come up with an extended patch. 

> > +			--config "lddlflags=-shared ${LDFLAGS}" \
> 
> Tadaam, you just broke Darwin like we did! Look at [1] for some clues,
> though perl may need different values.
> 
> [1]:https://bugs.gentoo.org/show_bug.cgi?id=513664#c7

Ouch. This is one thing I feared. OK, also doable, but I wish there was some 
way to extract "lddlflags minus ldflags" from the Perl configuration... since 
the Perl build system must be able to figure it out. Need to think about it.

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

On 23 June 2014 06:13, Andreas K. Huettel <dilfridge@gentoo.org> wrote:

>
> PERL_CFLAGS_HANDLING=yes
> inherit perl-module
>


I'd probably go with  PERL_C_BINDINGS=yes or PERL_XS=yes  or similar.

Because there's probably other things that we can infer from that property
and use for other things.


And its reasonably straight forward to automatically identify things that
do use XS/C bindings once they're installed, .... just impossible to do in
advance, and difficult to do heuristically during src_configure or similar.

qfile $( find /usr/lib/perl5 -name "*.so"  ) | cut -f 1 -d " "  | sort -u



-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1319 bytes --]

Re: [gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On 23 June 2014 07:10, Kent Fredric <kentfredric@gmail.com> wrote:

> I'd probably go with  PERL_C_BINDINGS=yes or PERL_XS=yes  or similar.
>
> Because there's probably other things that we can infer from that property
> and use for other things.
>
>
> And its reasonably straight forward to automatically identify things that
> do use XS/C bindings once they're installed, .... just impossible to do in
> advance, and difficult to do heuristically during src_configure or similar.
>
> qfile $( find /usr/lib/perl5 -name "*.so"  ) | cut -f 1 -d " "  | sort -u
>

Actually, that said, its probably wise we don't use PERL_ as a prefix,
there's far too much liklihood we'll collide with something in toolchain.

Or if we do, make sure we delete that value from %ENV before doing anything
with perl.

For an example of something we could use GENTOO_PERL_XS=yes for is
potentially exposing a tooling feature that indicates whether or not to
build the C bits at all.

For instance, ExtUtils::MakeMaker responds to the variable PUREPERL_ONLY=1
( Don't ask me where, I don't know, its really poorly documented ) and
Module::Build responds to a parameter --pureperl_only ( badly documented
here:
https://metacpan.org/pod/distribution/Module-Build/lib/Module/Build/API.pod#allow_pureperl
)

( I'm not saying we should do this, or do this yet, or do this on its own,
but it kinda works as justification for the generalist name )

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2359 bytes --]

[gentoo-dev] Re: perl-module.eclass: respect CFLAGS, LDFLAGS - please review

[Duncan]

Andreas K. Huettel posted on Sun, 22 Jun 2014 20:13:59 +0200 as excerpted:

> Offhand I don't know any way how the eclass could "know" whether a perl
> package requires C building or not. Any heuristics would be a hack at
> best. It might even be preferable to manually add some variable to 300
> packages in the ebuild (which then makes the eclass add a use-flag
> custom-cflags).
> i.e.
> 
> PERL_CFLAGS_HANDLING=yes inherit perl-module

Namespace collision issues aside, that would have been my next 
suggestion, yes.  The kde eclasses among others make extended use of this 
sort of "pre-inherit or pre-eclass-function-call declaration" solution.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman