[gentoo-dev] FEATURES use or misuse?

[gentoo-dev] FEATURES use or misuse?

[Patrick Lauer]

Hi there,

a while ago Thilo Bangert spent quite some time on filing lots of bugs. While 
I appreciate such QA efforts I don't agree with those bugs at all.

All of these bugs were for the use of the FEATURES variable in ebuilds, which 
is a very convenient thing to work around issues. 
For example known failures with FEATURE="distcc" or funky things like test 
failures with FEATURES="userpriv" and so on. All other methods of expressing 
that are much more verbose and inherently sucky.

One example of such a bug is https://bugs.gentoo.org/show_bug.cgi?id=278960 
for those too lazy to search.

To quote:
"FEATURES is a portage specific package manager configuration variable not
specified in PMS and cannot reliably be used in ebuilds or eclasses."

Well then, I suggest we finally start documenting reality and fix PMS. The use 
of the FEATURES variable, while it has been there for ... uhm ... as long as I 
can think back, actually :), should not be randomly suppressed. 

So ... what's your opinion? Should we do things as they are correct, or as 
they are specified in PMS? ( /me points at bash 3.0 )

Take care,

Patrick

Re: [gentoo-dev] FEATURES use or misuse?

[Sebastian Pipping]

Patrick Lauer wrote:
> Hi there,
> 
> a while ago Thilo Bangert spent quite some time on filing lots of bugs. While 
> I appreciate such QA efforts I don't agree with those bugs at all.
> 
> All of these bugs were for the use of the FEATURES variable in ebuilds, which 
> is a very convenient thing to work around issues. 
> For example known failures with FEATURE="distcc" or funky things like test 
> failures with FEATURES="userpriv" and so on. All other methods of expressing 
> that are much more verbose and inherently sucky.

What other methods are there?


> One example of such a bug is https://bugs.gentoo.org/show_bug.cgi?id=278960 
> for those too lazy to search.

For that very case I remember that "test" is a global use flag as well
and that therefore at least

  if hasq test ${FEATURES} ; then
      [..]
  fi

has a cleaner use-flag-based equivalent.

  # euse -i test
  global use flags (searching: test)
  ************************************************************
  [-    ] test - Workaround to pull in packages needed to run with
                 FEATURES=test. Portage-2.1.2 handles this internally,
                 so don't set it in make.conf/package.use anymore


> To quote:
> "FEATURES is a portage specific package manager configuration variable not
> specified in PMS and cannot reliably be used in ebuilds or eclasses."

Makes sense to me atm.


> Well then, I suggest we finally start documenting reality and fix PMS. The use 
> of the FEATURES variable, while it has been there for ... uhm ... as long as I 
> can think back, actually :), should not be randomly suppressed. 
> 
> So ... what's your opinion? Should we do things as they are correct, or as 
> they are specified in PMS? ( /me points at bash 3.0 )

My opinion is:  please stop dissing PMS, it doesn't help anybody.
I have requested that from you before.

Would a patch for the next EAPI theoretically impossible?



Sebastian

Re: [gentoo-dev] FEATURES use or misuse?

[Patrick Lauer]

On Tuesday 03 November 2009 18:27:46 Sebastian Pipping wrote:
> Patrick Lauer wrote:
> > Hi there,
> >
> > a while ago Thilo Bangert spent quite some time on filing lots of bugs.
> > While I appreciate such QA efforts I don't agree with those bugs at all.
> >
> > All of these bugs were for the use of the FEATURES variable in ebuilds,
> > which is a very convenient thing to work around issues.
> > For example known failures with FEATURE="distcc" or funky things like
> > test failures with FEATURES="userpriv" and so on. All other methods of
> > expressing that are much more verbose and inherently sucky.
> 
> What other methods are there?

For distcc, you could try some random things like checking the CC variable or 
such. All inherently unreliable methods to guess if the FEATURE variable 
contains a certain string.

Userpriv I've seen the funny idea to check if UID=0 and such. 

I'd guess for every FEATURE there's a similarly confused method to guess if it 
is enabled or disabled, instead of just checking the friggin variable.

> 
> > One example of such a bug is
> > https://bugs.gentoo.org/show_bug.cgi?id=278960 for those too lazy to
> > search.
> 
> For that very case I remember that "test" is a global use flag as well
> and that therefore at least
> 
>   if hasq test ${FEATURES} ; then
>       [..]
>   fi
> 
> has a cleaner use-flag-based equivalent.
> 
>   # euse -i test
>   global use flags (searching: test)
>   ************************************************************
>   [-    ] test - Workaround to pull in packages needed to run with
>                  FEATURES=test. Portage-2.1.2 handles this internally,
>                  so don't set it in make.conf/package.use anymore

That's a creative way to use a side-effect to check. Assumes that the package 
uses that useflag though, what about the case where you want to conditionally 
change a file (or maybe delete it or whatever) in src_prepare when 
FEATURES="test" is set, but USE="test" is not needed?

In short, stop complexifying things.

> 
> > To quote:
> > "FEATURES is a portage specific package manager configuration variable
> > not specified in PMS and cannot reliably be used in ebuilds or eclasses."
> 
> Makes sense to me atm.
Makes no sense to me, but then I seem to be special :)

> My opinion is:  please stop dissing PMS, it doesn't help anybody.
> I have requested that from you before.
It's still broken.
It doesn't document reality.
It's not useful as an implementation guideline or as a reference how to do 
things. In short, it sucks badly. And all my attempts to get it fixed have 
been deflected, so I'll keep ridiculing it until it stops being a failwhale.

After all I'm here to participate in the bestest distro ever, so we deserve 
correct and consistent documentation.
 
> Would a patch for the next EAPI theoretically impossible?
I absolutely fail to see how that helps the current situation.
EAPI is not the right answer to every question. :)


Have fun,

Patrick

Re: [gentoo-dev] FEATURES use or misuse?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Tue, 3 Nov 2009 21:36:18 +0100
Patrick Lauer <patrick@gentoo.org> wrote:
> Userpriv I've seen the funny idea to check if UID=0 and such. 

Yes, and that 'funny idea' has the added advantage of working even if
userpriv is in FEATURES but not actually enabled (yes, that can happen).

> > > To quote:
> > > "FEATURES is a portage specific package manager configuration
> > > variable not specified in PMS and cannot reliably be used in
> > > ebuilds or eclasses."  
> > 
> > Makes sense to me atm.  
> 
> Makes no sense to me, but then I seem to be special :)

PMS doesn't document user configuration. If PMS did document user
configuration, it would mean that user configuration file formats
couldn't arbitrarily be changed between package manager versions as
they are now.

If FEATURES were specified by PMS, Portage wouldn't be able to change
the meaning of its entries without careful EAPI controls. So far as I'm
aware, no-one is in favour of introducing such a restriction. There
are easy alternatives available, and unlike checking FEATURES, those
alternatives actually work.

> And all my attempts to get it fixed have been deflected, so I'll keep
> ridiculing it until it stops being a failwhale.

Patrick, perhaps you would find your efforts more fruitful were you to
respond to reviews of your patches by fixing the issues raised, instead
of using every available opportunity you can find to take pot-shots at
PMS, close off legitimate bugs as INVALID and generally attempt to make
life as hard as possible for those for whom PMS matters most.

Of the small number of patches that have ended up being rejected from
PMS, all but one have been yours, and the one that wasn't was because
the author had mistranslated a phrase. I'd appreciate it if you would
stop to consider why this is the case.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] FEATURES use or misuse?

[Patrick Lauer]

On Tuesday 03 November 2009 21:58:27 Ciaran McCreesh wrote:
> On Tue, 3 Nov 2009 21:36:18 +0100
> 
> Patrick Lauer <patrick@gentoo.org> wrote:
> > Userpriv I've seen the funny idea to check if UID=0 and such.
> 
> Yes, and that 'funny idea' has the added advantage of working even if
> userpriv is in FEATURES but not actually enabled (yes, that can happen).
>
I would consider that a bug. Maybe fixing that bug is easier than 
workarounding it?

> > > > To quote:
> > > > "FEATURES is a portage specific package manager configuration
> > > > variable not specified in PMS and cannot reliably be used in
> > > > ebuilds or eclasses."
> > >
> > > Makes sense to me atm.
> >
> > Makes no sense to me, but then I seem to be special :)
> 
> PMS doesn't document user configuration. If PMS did document user
> configuration, it would mean that user configuration file formats
> couldn't arbitrarily be changed between package manager versions as
> they are now.
I fail to see which part of "It's a key-value list, like the old windows .ini 
files, with comments starting with # allowed" is so specially specific that it 
can't be documented. But then I often fail to notice such obvious 
obviousities.

> If FEATURES were specified by PMS, Portage wouldn't be able to change
> the meaning of its entries without careful EAPI controls. So far as I'm
> aware, no-one is in favour of introducing such a restriction. There
> are easy alternatives available, and unlike checking FEATURES, those
> alternatives actually work.
That is concentrated nonsense. You're implying that an ebuild setting (EAPI) 
can control the package manager configuration. That's so exquisitely backwards 
and incoherent that I'm having a hard time taking you serious.

If PMS defined the existance of a FEATURES variable (like, say, CFLAGS) then 
the contents of that variable could be arbitrary whitespace-separated strings.

Amazingly CFLAGS is not EAPI-controlled and can take arbitrary values, so 
generalizing that amazing functionality to other configuration variables 
should be an easy exercise for the advanced reader.

Once that is done it can be specialized to a FEATURES variable, which is 
exactly what we expected.

> 
> > And all my attempts to get it fixed have been deflected, so I'll keep
> > ridiculing it until it stops being a failwhale.
> 
> Patrick, perhaps you would find your efforts more fruitful were you to
> respond to reviews of your patches by fixing the issues raised, 
I'm trying to do that. And you might want to not patronize me (especially in 
an academic setting that would be terribly rude, on the internet it's just 
silly)

The fact that there are a few dozen violations of PMS that are bastardly 
expensive to rollback suggests that harmonizing PMS to reality may be the 
cheaper method. Trying to bend reality to fit the specification can be an 
amusing game, but has a very high chance of failing hard.

> instead
> of using every available opportunity you can find to take pot-shots at
> PMS, close off legitimate bugs as INVALID and generally attempt to make
> life as hard as possible for those for whom PMS matters most.
I do wonder to whom PMS matters.

It didn't matter to the eclass authors that littered them with "bad" bash 3.2 
features.

It didn't matter to QA when they were notified of that.

It didn't matter to council back then and is still not high up on their 
priority list.

Can it be that the general population of gentoo developers plainly don't care 
about PMS? And if we were to assume that were true, why would they do such a 
thing?

So many questions. Almost like those TV shows where you can win a million 
dollars or a flamingo or a new car.

> 
> Of the small number of patches that have ended up being rejected from
> PMS, all but one have been yours, and the one that wasn't was because
> the author had mistranslated a phrase. I'd appreciate it if you would
> stop to consider why this is the case.
> 
Well, I've not contributed to PMS (like most people) for a long time. Like 
other people I've known that ANY patch I contribute will be denied. Most other 
people don't have the curiosity that drives me to try it to prove my theory, 
so the number of PMS contributors is amazingly small.

Added to that it's atrocious language. Might have been better if native 
english speakers had written it, but beggars can't be choosers. Most people 
don't have the stamina to read it, much less find all the spots they'd need to 
clean up to have a small bit of functionality improved.

And then why bother when the tree doesn't reflect PMS. It's just futile to 
work on a "documentation" that gets the basics wrong. And as soon as you read 
up on prior discussions you find these exhausting discussions that go nowhere 
... why would any sane person want to spend time working on that? Much more 
fun to actually fix bugs or write ebuilds. Or play WoW or whatever.

But I digress. You didn't actually want to have an answer, that was most 
likely a rethorical question. Silly me taking things literally.

Anyway, chill out, enjoy Christmas,

Patrick

[gentoo-dev] Re: FEATURES use or misuse?

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Tue, 3 Nov 2009 23:28:57 +0100
Patrick Lauer <patrick@gentoo.org> wrote:

> And then why bother when the tree doesn't reflect PMS.

Maybe if some people would stop ignoring PMS on whim because they don't agree
with something in it this wouldn't be the case.

Like, when does this end?  Whenever there's a policy you don't agree with,
you do whatever you want?  And it's the policy that's the problem?

I don't get it.

Anyways, this has nothing to do with PMS.  Using FEATURES in the tree was
frowned upon long before it even existed.  The fact that it wasn't documented
as such outside of mailing lists and bug reports is the real bug.


-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: FEATURES use or misuse?

[Patrick Lauer]

On Wednesday 04 November 2009 01:11:39 Ryan Hill wrote:
> On Tue, 3 Nov 2009 23:28:57 +0100
> 
> Patrick Lauer <patrick@gentoo.org> wrote:
> > And then why bother when the tree doesn't reflect PMS.
> 
> Maybe if some people would stop ignoring PMS on whim because they don't
>  agree with something in it this wouldn't be the case.

Well, we have at least one prior discussion and a year of precedent on the 
bash 3.0 / 3.2 thing. Since there were no sanctions for doing it, there's no 
way to break things with it (because you have a recent enough bash guaranteed) 
and it is very convenient people started using it.

After a year of use (and getting used more and more) I just don't see how any 
sane person can think about forbidding it NOW. It's too late. We've moved on. 
You missed your chance.

FEATURES has been used in ebuilds for a loooong time. People were happy with 
it. The only reason it was not properly documented in PMS was because the 
authors didn't agree with it. That's not how you do a standard, but then it 
never was about documenting reality. Now PMS has this hole in it, and instead 
of (1) documenting current behaviour and (2) agreeing on a standard behaviour 
while (3) keeping the historical errata documented ... well, it's kinda, look 
over there ... *runs away*
Not a way to discuss or write a standard, also making things complicated when 
there are known easy ways to fix it. 
> 
> Like, when does this end?  Whenever there's a policy you don't agree with,
> you do whatever you want?  And it's the policy that's the problem?
> 
Well, if everyone else freely ignores it and pointing out that people 
violating the policy has no response I'll consider that policy inactive.

If the Gentoo developers vote with their feet I'm not going to pretend they 
didn't. What you can do then is document what just happened ... maybe ... 
optionally?


> Anyways, this has nothing to do with PMS.  Using FEATURES in the tree was
> frowned upon long before it even existed.  The fact that it wasn't
>  documented as such outside of mailing lists and bug reports is the real
>  bug.
> 
And that usage was tolerated for >2 years. I still don't see what's bad about 
using things as they are, but if people now decide that we need to do complex 
dances instead of fixing things I'll just grab a camera and tape it instead of 
complaining. Life is too short to get worked up about such things :)

[gentoo-dev] Re: FEATURES use or misuse?

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 3237 bytes --]

On Wed, 4 Nov 2009 09:26:10 +0100
Patrick Lauer <patrick@gentoo.org> wrote:

> On Wednesday 04 November 2009 01:11:39 Ryan Hill wrote:
> > On Tue, 3 Nov 2009 23:28:57 +0100
> > 
> > Patrick Lauer <patrick@gentoo.org> wrote:
> > > And then why bother when the tree doesn't reflect PMS.
> > 
> > Maybe if some people would stop ignoring PMS on whim because they don't
> >  agree with something in it this wouldn't be the case.
> 
> Well, we have at least one prior discussion and a year of precedent on the 
> bash 3.0 / 3.2 thing. Since there were no sanctions for doing it, there's no 
> way to break things with it (because you have a recent enough bash guaranteed) 
> and it is very convenient people started using it.
> 
> After a year of use (and getting used more and more) I just don't see how any 
> sane person can think about forbidding it NOW. It's too late. We've moved on. 
> You missed your chance.

Yes, I think we did.  I think any damage that could have been done by people
ignoring policies about sticking to bash 3.0 has been done, and enforcing it
now would be pointless.

That's not to say though that it shouldn't have been enforced.  If anything,
I think this is a textbook example of the kind of corner we paint ourselves
into when people do whatever the fuck they feel like.  In this particular
case, it seems, no real harm was done except some small amount of backwards
compatibility was thrown out the window.  What happens next time?  I'm
surprised you're using this as an example to support your case because if
anything it warns me that we need to be more careful in the future.

> The only reason it was not properly documented in PMS was because the 
> authors didn't agree with it.

Bullshit.  It wasn't documented in PMS because PMS doesn't document user
configuration, because PMS shouldn't document user configuration.  User
configuration is implementation specific.  Do you not realize what a pain in
the ass it would be if we had to do an EAPI bump to slightly change the
semantics of "userpriv" or to change the enabled defaults, not to mention
change any of the environment variables portage uses for configuration?
Making these things independent of the specification allows the package
manager the freedom it needs to make the changes it needs to in order to
continue improving, and frankly, allows other implementations to be something
other than simple portage clones.

And you're still ignoring the fact that this has nothing to do with PMS.  You
have a core portage dev on record, saying "FEATURES are not supposed to have
an effect on the package itself, just how portage handles the package.
Packages behaving differently on certain FEATURES settings are considered
broken by me" back in 2005, before PMS was even a glimmer in an ex-gentoo
dev's eye.

> Well, if everyone else freely ignores it and pointing out that people 
> violating the policy has no response I'll consider that policy inactive.

Then we obviously need more people laying the smack down.
 

-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] a pragmatic approach to FEATURES [was FEATURES use or misuse?]

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 7296 bytes --]

On Wed, Nov 04, 2009 at 07:36:17PM -0600, Ryan Hill wrote:
> On Wed, 4 Nov 2009 09:26:10 +0100
> have a core portage dev on record, saying "FEATURES are not supposed to have
> an effect on the package itself, just how portage handles the package.
> Packages behaving differently on certain FEATURES settings are considered
> broken by me" back in 2005, before PMS was even a glimmer in an ex-gentoo
> dev's eye.

Marius was always quite a bit more hardline about FEATURES then the 
rest of us, but he never did have an answer for the harder questions 
about FEATURES.  Stubbs/myself were more pragmatic- in a purely 
academic sense FEATURES shouldn't exist, but we actually want to get 
shit done so bite the bullet and jump in with the pigs till a better 
solution is found.

Off the top of my head, I don't recall carpaski ever giving a damn 
about ebuild access to FEATURES either, although you'd have to ask 
him.

So you have another core developer from the same era saying "stop 
being academic and start being pragmatic".  Also, get me some pie.

To make it clear- lots of people say things.  Saying something doesn't 
make it right nor make it viable.  Cliche, but it seems to need 
repeating since people are latching onto a 4-5 year old comment as if 
it were the word of law.

And if that's not enough to shut people up, consider splitdebug, 
strip, noman, nodoc, sandbox (yes some ebuilds have had to be aware of 
it), test, etc.

Every one of those features influences packages in ways that at times 
requires the ebuild to be aware of it, and the majority of that list 
predate eapi/pms.  For some, there were solutions that didn't involve 
a hasq on $FEATURES, but not for all (preserve-libs and splitdebug 
come to mind).


> Patrick Lauer <patrick@gentoo.org> wrote:
> > The only reason it was not properly documented in PMS was because the 
> > authors didn't agree with it.
> 
> Bullshit.  It wasn't documented in PMS because PMS doesn't document user
> configuration, because PMS shouldn't document user configuration.  User
> configuration is implementation specific.

That's a mighty fine line to draw.  Pkgcore supports wildly different 
configurations, but still has a FEATURES var it exports to the user 
env.  This was done because ebuilds *expect* FEATURES to be there for 
some of the screwier things that exist.  Point being, user 
configuration *should not* be part of PMS, but that doesn't mean you 
can just label FEATURES as user config and stick your fingers in your 
ears because you don't like it's existance.

I'm not saying I like that fact- I don't particularly like FEATURES.  
That said, it *is* a fact it's out there.  If you're going to argue 
that ebuilds shouldn't know of FEATURES in some fashion, I suggest 
you finish that arguement and say that the majority of RESTRICT
should be pruned (there is a fair amount of parity between the two).


> Do you not realize what a pain in
> the ass it would be if we had to do an EAPI bump to slightly change the
> semantics of "userpriv" or to change the enabled defaults, not to mention
> change any of the environment variables portage uses for configuration?
> Making these things independent of the specification allows the package
> manager the freedom it needs to make the changes it needs to in order to
> continue improving, and frankly, allows other implementations to be something
> other than simple portage clones.

Ebuilds being aware that a space delimited var named FEATURES exists 
does not mean that PMS has to document each/every FEATURE.

If in doubt, consider that PMS doesn't document the installed pkg 
database (at all), but makes the barest mention required.  Or consider 
RESTRICT.  Same deal.  In certain spots, PMS seems willing to document 
the barest bits needed for something while in others it just ignores 
their existance (always found that annoying I might add).

From a purely academic standpoint, FEATURES shouldn't be available to 
ebuilds.  I'd suggest that for those pushing the academic route they 
find equivalent solutions that are of roughly comparable 
implementation cost for the ebuild dev, and push those changes 
through out the tree (after getting the maintainers consent of 
course).

Prove it flies for all scenarioes.  They want the changes primarily, 
and PMS choose to remove FEATURES from eapi0, they do the legwork 
rather then forcing it via a spec.


From a more pragmatic stance, I suggest the following-

1) FEATURES for existing EAPI's is documented as a space delimited 
variable that is effectively an array of 'enabled' 
features/flags/whatever you want to call them.  By enabled I mean the 
barest restriction on the items in the list- can't start with '-' 
(this is due to the incremental nature of FEATURES from configs- it's 
not really a requirement, just basically reality that no FEATURE item 
will start with '-').

2) the contents of FEATURES, the possibilities for flag/feature names 
are left to PM implementation details.

Yes this can become a mess.  I'm well aware it would require paludis 
to export a var they hate, although realistically for the way users 
hook functionality in they ought to be able to do a simple append to 
the ebuild env w/out issue.

If this were done, it would also require the 3 pms to cooperate on 
the naming of what specific functionality would be named.  
ccache/distcc/sandbox/preserve-libs/userpriv being the main ones that 
come to mind.  Even if they didn't, the ebuild author would have the 
option of either leveling a block, or just tweaking their hasq to 
check for an extra name.  Annoying, but pragmatic.

3) via pulling it in and more importantly acknowledging it's 
existance, it opens up the potential for later EAPI's to change it- 
say mandating in EAPI3 that an ebuild must not do 'hasq sandbox 
$FEATURES', instead doing '[ -n "$SANDBOX_ON" ]'.  


Stats time:

Currently, 96 ebuilds out of 26825 directly reference FEATURES via a 
fairly tight grep- of those, we're talking about dbus, mysql, libc, 
sqlite, postgres, mythtv, mono, and gcc.

These are not podunk packages.  At least for gcc/glibc, they've got 
good reasons for the checks they're doing- I'm not aware of 
alternatives for those cases.

For eclasses, around 4% of the eclasses have a reference to 
FEATURES... they're minor ones like python, selinux-policy-2, 
toolchain, mysql, java-utils2, and oh yeah, eutils.

Via 
pquery --attr inherited --raw --all --repo /usr/portage \
| cut -d= -f2- \
| egrep '( |")(eutils|gnatbuild|java-utils|kmod|mysql|myth|python|selinux-policy-2|toolchain(-binutils)?)( |")' \
| wc -l

We get the happy number of 20220, roughly 75% of the tree.  I'm not 
going to do the work, but if someone wants to filter that count down 
to just ebuilds that actually touch FEATURES codepath wise, that stat 
would be useful (and also a bit of a pain in the ass to get I 
suspect).

Either way, academic doesn't seem incredibly sane when we're talking 
about worst case 75% of the largest (and realistically, core) 
repository of that format.

Perhaps the pragmatic approach might be wise.

Donned my absestos suit, so flame way ;)
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] a pragmatic approach to FEATURES [was FEATURES use or misuse?]

[Thilo Bangert]

[-- Attachment #1: Type: Text/Plain, Size: 326 bytes --]


> Perhaps the pragmatic approach might be wise.
> 

when filing the FEATURES bugs, i (IIRC) tried to remain on the pragmatic 
side, recognising the sometimes non-existing alternatives. ie. i didnt 
open bugs for each and every FEATURES usage.

the tracker bug is here:
http://bugs.gentoo.org/show_bug.cgi?id=174335

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] a pragmatic approach to FEATURES [was FEATURES use or misuse?]

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]

On Thu, Nov 05, 2009 at 09:49:07AM +0100, Thilo Bangert wrote:
> 
> > Perhaps the pragmatic approach might be wise.
> > 
> 
> when filing the FEATURES bugs, i (IIRC) tried to remain on the pragmatic 
> side, recognising the sometimes non-existing alternatives. ie. i didnt 
> open bugs for each and every FEATURES usage.
> 
> the tracker bug is here:
> http://bugs.gentoo.org/show_bug.cgi?id=174335

While I appreciate you being pragmatic in your cleanup efforts, you 
miss the point of my post- while FEATURES reliance needed a valid 
reason from a QA standpoint, it was *valid* from a format standpoint 
prior to paludis/PMS (and was the only option in some corner cases 
ebuild wise).

The response these days when it comes to FEATURES is that you cannot 
rely on it's existance ever- this loops back to my point about 
pragmatism.

I understand PMS/paludis wishing to duck the vars existance to make it 
go away, but I don't think it's a tenuable approach- as you yourself 
said above, in trying to do this cleanup you recognized that sometimes 
there was no alternative.

Please understand my point here is QA; not picking a fight.  That 
said, paludis doesn't do anything FEATURES wise due to PMS not 
mandating they do so... as you said, certain ebuilds rely on it's 
existance to work.

This means via PMS being incomplete, paludis isn't going to play nice 
in those cases (corner case- if the user defines the var themselves in 
their config, this would address it).

Essentially, you can't use FEATURES but you have to in some cases.  
Doing so however doesn't necessarily play nice w/ paludis due to 
stepping outside of PMS.  Classic catch 22.

Rather then letting the problem persist, I'd rather see folk take a 
look at FEATURES/PMS and identify what needs to be pushed in to take 
care of the cases where there is no alternative to 'hasq some-feature 
$FEATURES' rather then us just collectively sticking our heads in the 
sand.

Grok?

Or we can just keep ignoring the problem pretending it's a user config 
issue rather then a format level issue...
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] a pragmatic approach to FEATURES [was FEATURES use or misuse?]

[Thilo Bangert]

[-- Attachment #1: Type: Text/Plain, Size: 927 bytes --]

[snip]

> I understand PMS/paludis wishing to duck the vars existance to make it
> go away, but I don't think it's a tenuable approach- as you yourself
> said above, in trying to do this cleanup you recognized that sometimes
> there was no alternative.

yes - however, there not being an alternative at the moment does not 
automatically mean that FEATURES is a good or the best approach. A more 
clean approach still needs to be proposed. 

[snip]

> 
> Rather then letting the problem persist, I'd rather see folk take a
> look at FEATURES/PMS and identify what needs to be pushed in to take
> care of the cases where there is no alternative to 'hasq some-feature
> $FEATURES' rather then us just collectively sticking our heads in the
> sand.

yes - exactly. so which FEATURES are absolutely required in ebuilds / 
eclasses for which an alternative must be developed?

> 
thanks for your input, BTW

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] FEATURES use or misuse?

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 1692 bytes --]

> To quote:
> "FEATURES is a portage specific package manager configuration
> variable not specified in PMS and cannot reliably be used in ebuilds
> or eclasses."

For distcc & ccache, let me quote ebuild.sh code:

if hasq distcc $FEATURES ; then
	export PATH="/usr/lib/distcc/bin:$PATH"
        [[ -n $DISTCC_LOG ]] && addwrite "${DISTCC_LOG%/*}"
fi

if hasq ccache $FEATURES ; then
	export PATH="/usr/lib/ccache/bin:$PATH"
[...]

Do you want an example how to mimic this with portage without having
neither distcc nor ccache in FEATURES?
Even with portage, checking the FEATURES variable isn't reliable. If
you do not want to fix the real bugs and check/disable distcc/ccache
for any reason, then check PATH.

If you want to keep this simple, write an eclass providing functions
for disabling/checking these features.

> Well then, I suggest we finally start documenting reality and fix
> PMS. The use of the FEATURES variable, while it has been there
> for ... uhm ... as long as I can think back, actually :), should not
> be randomly suppressed. 

If you want to fix PMS, then send a patch rephrasing it to explain why
it isn't correct to check FEATURES in some cases. On the other hand, as
its name implies, PMS is a spec, not a documentation on why every
single choice has been made.

> So ... what's your opinion? Should we do things as they are correct,
> or as they are specified in PMS?

PMS may have some flaws, but not these, sorry.

> ( /me points at bash 3.0 )

Ever thought about backward compatibility and keeping a sane upgrade
path? Because that's exactly what this EAPI & PMS debate is all about
IMHO.


Regards,

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] FEATURES use or misuse?

[Patrick Lauer]

On Tuesday 03 November 2009 22:26:24 Alexis Ballier wrote:
> > To quote:
> > "FEATURES is a portage specific package manager configuration
> > variable not specified in PMS and cannot reliably be used in ebuilds
> > or eclasses."
> 
> For distcc & ccache, let me quote ebuild.sh code:
> 
> if hasq distcc $FEATURES ; then
> 	export PATH="/usr/lib/distcc/bin:$PATH"
>         [[ -n $DISTCC_LOG ]] && addwrite "${DISTCC_LOG%/*}"
> fi
> 
> if hasq ccache $FEATURES ; then
> 	export PATH="/usr/lib/ccache/bin:$PATH"
> [...]
> 
> Do you want an example how to mimic this with portage without having
> neither distcc nor ccache in FEATURES?
> Even with portage, checking the FEATURES variable isn't reliable. If
> you do not want to fix the real bugs and check/disable distcc/ccache
> for any reason, then check PATH.
Well, if a user wants to shoot himself hard enough in the foot he (or she, or 
it) can do that. But in the general case we should allow silly assumptions, 
one of them being that if a user sets FEATURES="distcc" that he wants to use 
distcc and will use the portage mechanisms to enable it. The assumption that 
the FEATURE variable actually controls the behaviour of such features seems 
like a good idea to me, almost as good as digital watches.

> If you want to keep this simple, write an eclass providing functions
> for disabling/checking these features.
Wow, that's a nice way to make things complex :)
(and why not let the package manager manage such things?)
 
> > Well then, I suggest we finally start documenting reality and fix
> > PMS. The use of the FEATURES variable, while it has been there
> > for ... uhm ... as long as I can think back, actually :), should not
> > be randomly suppressed.
> 
> If you want to fix PMS, then send a patch 
I tried, and as I've been saying for a long time they get rejected.
Funnily not by any dev but by some random user, but who cares :)
With the current attitude PMS will be marginalized and worked around by 
people. Reality is what you perceive.

> rephrasing it to explain why
> it isn't correct to check FEATURES in some cases. On the other hand, as
> its name implies, PMS is a spec, not a documentation on why every
> single choice has been made.
It's not a spec. Any engineer who has been involved in the spec creation or 
update process will gladly show you where it fails (for example not 
documenting, on purpose, some bits instead of documenting them with the note 
that this is nonstandard behaviour. A real specification will document such 
errata because then anyone working with it knows the potential issues ...)

 
> > ( /me points at bash 3.0 )
> 
> Ever thought about backward compatibility and keeping a sane upgrade
> path? Because that's exactly what this EAPI & PMS debate is all about
> IMHO.
Well, I don't want to be rude to you. But please, try using the KDE eclass 
with bash 3.0. Or maybe, just for fun (yeah, I know) portage.

Please do it and then tell me how forcing bash 3.2, then documenting the need 
for bash 3.0, is in any way sane or consistent.

We broke backwards compatibility about a year ago. Since then we've collected 
so many bash-3.0-incompatible bits that a migration back is technically 
possible, but practically no longer an acceptable solution (And trying to 
force it will make you lots of new "friends"). Your objections come a year to 
late. Now we have to accept reality and decide how to live with it.

Calling EAPI is ... well ... I can't even think of a place to start to explain 
how wrong it is. How on earth are you going to parse an eclass that supports 
multiple EAPIs where one EAPI were to support features of bash 4? 
The only way to do it would be to force bash 4 on all lower EAPIs, or make 
per-EAPI eclasses, or forbid use of new bash features in eclasses.
All horrible ways to avoid fixing the problem. All workaroundable by just 
accepting things as they are.

Sometimes EAPI is not a viable solution to a problem.

Take care,

Patrick

Re: [gentoo-dev] FEATURES use or misuse?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

On Tue, 3 Nov 2009 23:04:58 +0100
Patrick Lauer <patrick@gentoo.org> wrote:
> > If you want to fix PMS, then send a patch 
> I tried, and as I've been saying for a long time they get rejected.

Yes, let's look at the patches that you sent that were rejected:

The first was one where you asked the PMS team to make a change that
goes directly against a Council decision, and that the PMS team does
not have the authority to accept on its own. You were told to bring
the issue to gentoo-dev@ for further attention. You never did.

The second patch you sent:

* incorrectly removed as "no content" two sentences that had specific
  and necessary meaning.

* didn't apply against any version of PMS except your much-modified
  local version, and that didn't make sense when applied to version of
  PMS that anyone else had.

Again, no updated patch was sent.

The third patch you sent had a number of small issues and could have
been improved. You did not send an updated patch with those issues
fixed.

> Funnily not by any dev but by some random user, but who cares :)

Everyone on the PMS mailing list is more than welcome to do patch
reviews. The more eyes we have looking at things the better. Policy is
not to apply patches until everyone is happy with them; objections are
considered on their merits, not on their author.

Every other person who has had people comment upon a patch they
submitted has either addressed the issues and submitted an updated
patch, or explained why they feel the patch is correct as-is.

Every other person who has submitted a patch has worked with all the
reviewers to reach a conclusion that is acceptable to all involved.

Every other person has an ultimate patch acceptance rate of at least
98%. You are the *only* person to have had patches rejected where the
submitter did not agree with the rejection.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] FEATURES use or misuse?

[Sebastian Pipping]

Patrick Lauer wrote:
> Calling EAPI is ... well ... I can't even think of a place to start to explain 
> how wrong it is. How on earth are you going to parse an eclass that supports 
> multiple EAPIs where one EAPI were to support features of bash 4? 
> The only way to do it would be to force bash 4 on all lower EAPIs, or make 
> per-EAPI eclasses, or forbid use of new bash features in eclasses.
> All horrible ways to avoid fixing the problem.

I find restricting the eclass to Bash 3 is a natural, maintainable
approach to this.  How would "fixing he problem" work from your perspective?


> All workaroundable by just 
> accepting things as they are.

What do you mean by "accepting things as they are"?
You have been talking of "accepting reality" repeatedly and I'm left
wondering what you actually mean by that.  I especially fail to see who
is trying to conceal(?) reality and reality about what.



Sebastian

Re: [gentoo-dev] FEATURES use or misuse?

[Patrick Lauer]

On Wednesday 04 November 2009 01:33:23 Sebastian Pipping wrote:
> Patrick Lauer wrote:
> > Calling EAPI is ... well ... I can't even think of a place to start to
> > explain how wrong it is. How on earth are you going to parse an eclass
> > that supports multiple EAPIs where one EAPI were to support features of
> > bash 4? The only way to do it would be to force bash 4 on all lower
> > EAPIs, or make per-EAPI eclasses, or forbid use of new bash features in
> > eclasses. All horrible ways to avoid fixing the problem.
> 
> I find restricting the eclass to Bash 3 is a natural, maintainable
> approach to this.  How would "fixing he problem" work from your
>  perspective?
It doesn't :)
You can't use the "new" features in the "old" eclass, even with conditionals 
separating the execution paths. Which means you'd have to either not use them 
(which makes me wonder why we allow features when they can't be used).
Or you clone the eclass and now maintain the code in two places (wheee, bad 
engineering!)

So we end up with a bad solution either way. There are some clean options, but 
they tend to be a bit more complex. For example globally forcing minimum 
versions (which makes upgrade paths a bit more interesting). 

> 
> > All workaroundable by just
> > accepting things as they are.
> 
> What do you mean by "accepting things as they are"?

People have been doing things (in this case using bash 3.2 features) for a 
long time (about a year now). Even when some people warned about the impact 
noone cared.

So more and more these "illegal" features get used, and as there are no 
sanctions for it (not even from QA!) they are accepted as allowed.

Fast forward and you have an informal standard (using += in ebuilds is ok) 
that is agreed on by everyone. Yes, everyone, because when people pointed out 
that it was a Bad Thing there was no reaction, no opposition, nothing.

So the Gentoo developer community agreed on it. The only thing not reflecting 
that agreement is PMS. So we fix it.

Same with FEATURES variable. Been used for the last few years. Works. 
Most reliable way to do a few things if you assume that users don't actively 
try to break things. And instead of properly documenting it we pretend it 
never happened?


> You have been talking of "accepting reality" repeatedly and I'm left
> wondering what you actually mean by that.  I especially fail to see who
> is trying to conceal(?) reality and reality about what.
Ok,

from stable portage ebuilds:
RDEPEND="  [snip]
>=app-shells/bash-3.2_p17

from KDE eclass:
RESTRICT+=" test"

gentoo-x86/app-shells/bash $ ls -1 *.ebuild
bash-3.1_p17.ebuild
bash-3.2_p39.ebuild
bash-3.2_p48-r1.ebuild
bash-3.2_p48.ebuild
bash-4.0_p28.ebuild
bash-4.0_p33.ebuild
bash-4.0_p35.ebuild

So we can either dance around all day and pretend bash 3.0 still has any 
relevance, or we stop the nonsense and tolerate reality as it is.

We can also pretend FEATURES never served a purpose and doesn't fix any 
issues, then spend lots of time workarounding around working solutions because 
we just declared them illegal.

I don't know how much time you have and what your priorities are, but I'm not 
going to care about such a waste of time, and it goes very low on my list of 
priorities. If there's a decision on this I doubt most devs will care much, so 
anyone wanting to have the FEATURES use removed will end up having to do it 
himself, against the resistance of package maintainers (don't touch my package 
etc. etc.)

Have fun,

Patrick

Re: [gentoo-dev] FEATURES use or misuse?

[David Leverton]

On Tuesday 03 November 2009 15:48:03 Patrick Lauer wrote:
> To quote:
> "FEATURES is a portage specific package manager configuration variable not
> specified in PMS and cannot reliably be used in ebuilds or eclasses."

This has been the Portage team's position for years, since long before there 
were PMS and other package managers.  See 
http://bugs.gentoo.org/show_bug.cgi?id=82513#c16 for example.

[gentoo-dev] Re: FEATURES use or misuse?

[Duncan]

David Leverton posted on Tue, 03 Nov 2009 23:04:33 +0000 as excerpted:

> On Tuesday 03 November 2009 15:48:03 Patrick Lauer wrote:
>> To quote:
>> "FEATURES is a portage specific package manager configuration variable
>> not specified in PMS and cannot reliably be used in ebuilds or
>> eclasses."
> 
> This has been the Portage team's position for years, since long before
> there were PMS and other package managers.  See
> http://bugs.gentoo.org/show_bug.cgi?id=82513#c16 for example.

Indeed, that comment is from 2005, but I've been around since 2004 and 
remember no other position on FEATURES, ever.  There are others who 
predate me.  Idle question, when was FEATURES introduced as a portage 
variable?

Of course, one way to end the debate would be for portage to unset the 
variable before calling ebuild.sh, on environment pollution grounds...  
Given the history of (ab)use, there'd be solid reason to do that.  Maybe 
I should file a bug...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: FEATURES use or misuse?

[Peter Hjalmarsson]

tis 2009-11-03 klockan 16:48 +0100 skrev Patrick Lauer:
> Hi there,
> 
> All of these bugs were for the use of the FEATURES variable in ebuilds, which 
> is a very convenient thing to work around issues. 
> For example known failures with FEATURE="distcc" or funky things like test 
> failures with FEATURES="userpriv" and so on. All other methods of expressing 
> that are much more verbose and inherently sucky.

I ask myself if what we really want is many different and strange
approaches to handle FEATURES?
Would it not be better to actually expand some eclasses to be able to
say something about your build environment?
I mean where the checks for "userpriv" is needed also prefix will fail,
because AFAIK it can be used to build and install programs in an
non-root environment? Or if you just test an ebuild and runs it as your
user? So would it not just be better to have a check for which privs the
user has so it covers more fields?
For "ccache" and for "distcc" would it not be better to expand
toolchain-funcs so you can have a function like "tc-getCC" from which
you can get that sort of information?

Also if the ebuilds does not already have a comment about it, do not
forget to comment on why these checks are there, and why they are a must
(i.e. a broken buildsystem should be fixed, not worked around - while
tests that are designed to run as root should not be run as a user even
if in the best of worlds all testsuits should test and skip those tests
themselves).
I would not like to see a new kind of hell where when something is
broken it is not fixed properly, but in a strange ways worked around in
ways that does not always work.
qemu and kvm is good examples on how NOT to do this these with regards
to hardened.

qemu (which kvm apparently has used as a template) has a broken build
system (it does not link with CFLAGS, only LDFLAGS, which is something
that also the gcc devs say you should not if you want a predictable
result), and it also invokes filter-flags at the wrong place in the
ebuild (hint: int should be invoked before the command that sets the
CFLAGS, in this case ./configure and not after like in these ebuilds).
Instead it has some strange logic to unset/change the GCC_SPECS which if
it ever worked certainly does not anymore (bugs filed for both, qemu bug
is really old, but very noisy, bug for kvm has been open for about an
week which the qemu maintainer may want to check out, with a clean patch
of one added sed a move of filter-flags and the removal of the whole
src_compile block to make the ebuild install something which (in case of
qemu) actually build, and does not segfault as soon as it uses a bit
softmmu).

Re: [gentoo-dev] Re: FEATURES use or misuse?

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2001 bytes --]

On Wed, Nov 04, 2009 at 11:12:05PM +0100, Peter Hjalmarsson wrote:
> tis 2009-11-03 klockan 16:48 +0100 skrev Patrick Lauer:
> > Hi there,
> > 
> > All of these bugs were for the use of the FEATURES variable in ebuilds, which 
> > is a very convenient thing to work around issues. 
> > For example known failures with FEATURE="distcc" or funky things like test 
> > failures with FEATURES="userpriv" and so on. All other methods of expressing 
> > that are much more verbose and inherently sucky.
> 
> I ask myself if what we really want is many different and strange
> approaches to handle FEATURES?
> Would it not be better to actually expand some eclasses to be able to
> say something about your build environment?

This is a *really* bad approach- pkgcore/paludis have supported 
standalone repositories basically from their inception, portage (via 
repos.conf) has developed equivalent support, and from the looks of it 
is moving towards such capabilities.

What this means is that eclasses aren't automatically mashed together 
across all trees and shared.  Meaning each repository would have to 
carry those eclasses, or require that they always be slaved against a 
specific repository carrying said eclasses (again, bad mojo).

The code duplication there is annoying, but the potential range of 
screwups this can lead to is even worse.  Further, via proper 
environment saving/restoration for installed pkgs/binpkgs, this means 
that one screwed up FEATURES detection (or that things have changed 
since then and a new check is required) lives on forever, no 
possibility of being updated/fixed in the env dump.

Shorter version, eclasses are a fun idea at first until you dig in and 
realize they'll blow your leg off for things like this.

If it's any consolation, I proposed moving large chunks of eapi0 
(prior to eapi existing) into the tree- the idea died off for the same 
reasons your "FEATURE awareness in eclasses" must die off.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]