[gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hola all-

For those who aren't familiar, pkg_pretend is in EAPI4- the main usage 
of it is will be use dep checking- this email is specifically 
regarding an alternative to it that *should* be superior for that use 
case, but I'm looking for feedback.

Basically, we use the original VALID_USE proposal from way back in 
'05- if you're familiar w/ MYOPTIONS, they're reasonably similar.

Roughly, VALID_USE is a list of constraints stating what the allowed 
use flag combinations are for this pkg.  If you think of normal 
depdencies (I must have openssl and python merged prior), it's the 
same machinery.

Examples:

# if build is set, python and openssl must be unset
VALID_USE="build? ( !python !openssl )"

# if mysql is set, sqlite must not be, and vice versa.
# note mysql/sqlite do *not* have to be set also.
VALID_USE="mysql? ( !sqlite ) !sqlite? ( mysql )"

# mysql or sqlite must be set; exclusive or.
VALID_USE="mysql? ( !sqlite ) !mysql ( sqlite )"

# alternative syntax, adding an xor group operator
# note xor isn't required- you can do the same thing
# via spelling it out, it's just a convenient thing to have.
VALID_USE="^^ ( mysql sqlite )"

# if gui is enabled, a widget set must be specified- can build
# multiple widget bindings
VALID_USE="X? ( || ( gtk qt motif ) )"


Note that like dependencies, these are assertions.  More importantly, 
they're also data rather then executable code.  Via it being data, up 
front GUI's can tell you exactly what conflicts arise, and what needs 
to be adjusted.

Doing it as executable, can, but it's iterative and reliant on the dev 
writing a clear message every time (rather then the UI tool getting 
it right once).  Clarifying iterative, consider

USE="build X"
w/ valid use states being-
VALID_USE="build? ( !X !python ) X? ( ^^ ( gtk qt ) ) gtk? ( ssl )"

The user first flips off build.  They rerun emerge- next they're told 
"you must choose gtk or qt, not both".  They change to USE="X gtk".  
Next they're told "you need ssl turned on if you want gtk".

At this point, the user goes and gets a beer because aparently Murphy 
hates them and it's going to be a long, long night.


There also is one major issue with relying on pkg_pretend 
(executable) for use state validation vs doing it as VALID_USE 
(data)- the package manager cannot know what states are valid thus 
limiting the things it can do.  Literally the pkg manager is screwed 
when it comes to use cycle breaking.  If the pkg manager doesn't know 
what states are valid, when it encounters a use cycle it doesn't know 
what intermediate builds it can do to break that cycle- literally if 
USE state validation is in pkg_pretend, the *user* will have to walk 
the PM through breaking the cycle instead of the PM figuring out the 
proper steps to break it.

Executive summary: if use validation is implemented via pkg_pretend,
1) it still has the iterative issue
2) it's harder for configuration tools to deal with (iterative issues 
above, plus the fact they get a blob of text they're stuck trying to 
parse)
3) it pretty much eliminates the possibility of doing use cycle 
breaking properly (which is already an issue, only going to get worse- 
hence why this was proposed in '04/'05 when we started playing w/ use 
deps in portage 3 at the time).


Comments desired; assuming no significant blowback, I'll be pushing 
this to the council level since eapi4 is annoying feature locked right 
now.

Thanks-
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Ulrich Mueller]

>>>>> On Wed, 31 Mar 2010, Brian Harring wrote:

> Roughly, VALID_USE is a list of constraints stating what the allowed
> use flag combinations are for this pkg. If you think of normal
> depdencies (I must have openssl and python merged prior), it's the
> same machinery.

Maybe we should first discuss if we want to drop the following
rule [1] which your proposal seems to contradict:

| Occasionally, ebuilds will have conflicting USE flags for
| functionality. Checking for them and returning an error is not a
| viable solution. Instead, you must pick one of the USE flags in
| conflict to favour.

Ulrich

[1] <http://devmanual.gentoo.org/general-concepts/use-flags/>

[gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

Note that while I inadvertantly cross posted (I was intending on 
cc'ing council@gentoo.org, not the ml), doubt they need to be cc'd 
further- my original attention was to effectively ensure they were 
paying aware of the details of this so that when I took it to them 
folk were informed.

CC'ing gentoo-council so folk following it there know it moved 
over to -dev.  Your discussion of devmanual relevance needs some -dev 
consensus anyways before the council should be deciding on it.

Also the cross posting is making betelgeuse cry anyways (and pissing 
off my procmail setup) ;)


On Wed, Mar 31, 2010 at 11:48:37AM +0200, Ulrich Mueller wrote:
> >>>>> On Wed, 31 Mar 2010, Brian Harring wrote:
> 
> > Roughly, VALID_USE is a list of constraints stating what the allowed
> > use flag combinations are for this pkg. If you think of normal
> > depdencies (I must have openssl and python merged prior), it's the
> > same machinery.
> 
> Maybe we should first discuss if we want to drop the following
> rule [1] which your proposal seems to contradict:

Not just my proposal- council contradicted it via even letting 
pkg_pretend into EAPI3 (now EAPI4):

http://www.mail-archive.com/gentoo-council@lists.gentoo.org/msg00493.html


> | Occasionally, ebuilds will have conflicting USE flags for
> | functionality. Checking for them and returning an error is not a
> | viable solution. Instead, you must pick one of the USE flags in
> | conflict to favour.
> 
> [1] <http://devmanual.gentoo.org/general-concepts/use-flags/>

I honestly consider the ebuild silently making decisions on the user's
behalf *worse*.  Consider if openoffice silently made decisions like 
that- 4 hours later it'll wind up choosing the option you didn't 
really want and you'll be in a foul mood.

Frankly is the devmanual even relevant on at this point beyond good 
practices btw?  Last I looked through it, there was a rather unhealthy 
mix of good policy that we follow, and policy that isn't relevant 
anymore- in need of some cleanup at the very least.


~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: pkg_pretend USE validation and VALID_USE alternative

[Ulrich Mueller]

>>>>> On Wed, 31 Mar 2010, Brian Harring wrote:

> Not just my proposal- council contradicted it via even letting 
> pkg_pretend into EAPI3 (now EAPI4):

> http://www.mail-archive.com/gentoo-council@lists.gentoo.org/msg00493.html

It says "displaying conflicting USE flags" which doesn't necessarily
imply that the ebuild should fail.

> I honestly consider the ebuild silently making decisions on the
> user's behalf *worse*.

Right, this is exactly what we should decide on, before talking about
possible implementations.

We already have enough issues with circular dependencies, and I'm
sceptical about adding additional failures on USE flag conflicts.
Display a warning, but don't error out.

Ulrich

[gentoo-dev] Re: pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On Wed, Mar 31, 2010 at 01:04:39PM +0200, Ulrich Mueller wrote:
> We already have enough issues with circular dependencies, and I'm
> sceptical about adding additional failures on USE flag conflicts.
> Display a warning, but don't error out.

Solve the cyclical dependency via breaking the use cycle and doing two 
builds of the target instead.

It *does* work.  Now I'm reasonably sure I've since *broken* that 
support in pkgcore since stubbs/myself worked it out (resolver still 
supports it though), but that doesn't mean restoring it is hard.  

Frankly I'd rather see us support these things then have the PM bitch 
at the user or make guesses on the users behalf.

My 2 cents either way- mainly responding since you seem to be ignoring 
that as long as pkg_pretend isn't fricking used, we've got a way to 
break use induced cyclical deps automatically via the PM.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: pkg_pretend USE validation and VALID_USE alternative

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On 3/31/10 1:04 PM, Ulrich Mueller wrote:
> We already have enough issues with circular dependencies, and I'm
> sceptical about adding additional failures on USE flag conflicts.
> Display a warning, but don't error out.

How about only allowing local USE flags to conflict? This also seems to
be the most common use case.

Anyway, the earlier the user can react to USE flags conflict, the better.

Paweł Hajdan jr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Piotr Jaroszyński]

>> | Occasionally, ebuilds will have conflicting USE flags for
>> | functionality. Checking for them and returning an error is not a
>> | viable solution. Instead, you must pick one of the USE flags in
>> | conflict to favour.
>>
>> [1] <http://devmanual.gentoo.org/general-concepts/use-flags/>
>
> I honestly consider the ebuild silently making decisions on the user's
> behalf *worse*.  Consider if openoffice silently made decisions like
> that- 4 hours later it'll wind up choosing the option you didn't
> really want and you'll be in a foul mood.

I'm pretty sure it says that because there was no way to fail early
before. And failing in the middle of 300 packages upgrade because some
useflags are in conflict wasn't reasonable.

-- 
Best Regards
Piotr Jaroszyński

Re: [gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Róbert Čerňanský]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

On Wed, 31 Mar 2010 03:46:47 -0700
Brian Harring <ferringb@gmail.com> wrote:

> 
> On Wed, Mar 31, 2010 at 11:48:37AM +0200, Ulrich Mueller wrote:
> > >>>>> On Wed, 31 Mar 2010, Brian Harring wrote:
> > 
> > | Occasionally, ebuilds will have conflicting USE flags for
> > | functionality. Checking for them and returning an error is not a
> > | viable solution. Instead, you must pick one of the USE flags in
> > | conflict to favour.
> > 
> > [1] <http://devmanual.gentoo.org/general-concepts/use-flags/>
> 
> I honestly consider the ebuild silently making decisions on the user's
> behalf *worse*.  Consider if openoffice silently made decisions like 
> that- 4 hours later it'll wind up choosing the option you didn't 
> really want and you'll be in a foul mood.

If I'm getting this right the proposed behavior is such that in case of
conflicting use flags emerge fails and user gets a message that he
has to set use flags as required. If so then I think it is not the right
way to handle it. A package manager should be able do deal with (use
flag) dependencies automatically. Similarly as it deals with "normal"
package dependenicies.

It should not do this silenly though. emerge -pv should display real
state of use flags; so if some use flag has to be turned on
automatically due to dependency/conflict then it has to be shown so.

This apply also for package[use_flag] deps. It is not very convenient
to fiddle use flags for individual packages that I basically do not care
about because they are just dependencies; so natural expectation is
that package manager pulls required deps. automatically (whether it
means install a package or install a package _with_ switched use flag).

I hope this does not sound that I'm dictating you what is the right way
to do things. I just wanted to express my opinion. And I admit that
perhaps I do not see possible negative consequences of such behaviour.

Regards,
Robert


-- 
Robert Cernansky
E-mail: hslists2@zoznam.sk
Jabber: hs@jabber.sk

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Zac Medico]

On 04/01/2010 01:44 PM, Róbert Čerňanský wrote:
> If I'm getting this right the proposed behavior is such that in case of
> conflicting use flags emerge fails and user gets a message that he
> has to set use flags as required. If so then I think it is not the right
> way to handle it. A package manager should be able do deal with (use
> flag) dependencies automatically. Similarly as it deals with "normal"
> package dependenicies.
> 
> It should not do this silenly though. emerge -pv should display real
> state of use flags; so if some use flag has to be turned on
> automatically due to dependency/conflict then it has to be shown so.
> 
> This apply also for package[use_flag] deps. It is not very convenient
> to fiddle use flags for individual packages that I basically do not care
> about because they are just dependencies; so natural expectation is
> that package manager pulls required deps. automatically (whether it
> means install a package or install a package _with_ switched use flag).
> 
> I hope this does not sound that I'm dictating you what is the right way
> to do things. I just wanted to express my opinion. And I admit that
> perhaps I do not see possible negative consequences of such behaviour.

I appreciate your input, and I think you're right about this issue.
However, it seems to be essentially a user interface issue, and
therefore it doesn't necessarily need to be included in the
specification (except maybe as a side note).
-- 
Thanks,
Zac

[gentoo-dev] Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

Note I inadvertantly cross posted, I was intending on cc'ing 
council@gentoo.org.

As such one final cc to that ml to end this subthread while pulling 
this back to -dev.


On Wed, Mar 31, 2010 at 11:16:22PM +1300, Alistair Bush wrote:
> > Hola all-
> >  
> > Comments desired; assuming no significant blowback, I'll be pushing
> > this to the council level since eapi4 is annoying feature locked right
> 
> I think this solution is far better,  until someone smarter than me tells me 
> otherwise.
> 
> Don't know whether I like the VALID_USE var name,  so please at least think 
> about something a little better if you can ;).  ( I like green,  but not too 
> green if you know what I mean )
>
> Will we still have to define the use flags in IUSE?

Yes, although if folks have a better proposal that incorporates 
VALID_USE into IUSE I'm definitely open to it- the original proposal 
for VALID_USE tried to inline it into IUSE, but it got ugly (hence it 
mutating it this form).

The problem w/ trying to reuse IUSE is the following (sorry, I like 
lists of assertions)-

*) IUSE currently serves as a list of valid USE flags, just that.  No 
repeat specification of a flag (which means the dev in question is 
unlikely to typo a flag).

*) having a single specified list of valid use flags is the basis for 
doing validation of use flags used in all other metadata.  In other 
words, that list of valid use flags *really* needs to go out of it's 
way to make human error hard.

*) VALID_USE is a set of assertions on the allowed state of USE; IUSE 
is just a list of flags.  Intermixing the two in a way that is still 
readable is really ugly

*) given a library that has optional perl and python bindings 
(which can be toggled freely, they're standalone flags) I've nfc how 
one would sanely specify that w/in IUSE while adding VALID_USE 
semantics.  Possibly, you could include use conditionals into IUSE, 
and treat the () contents as assertions- but that makes adding xor in 
hard, is rather ugly/hard on the eyes, and violates the DRY (Don't 
Repeat Yourself) principle from above.

Definitely open to counterproposals that address those 
concerns however...


> I'm guessing we can't use IUSE to store this information because of the  whole 
> glep-55 thing.

glep-55 is unrelated to this as far as I can tell- if you think 
otherwise please clarify.

Thanks
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Alex Alexander]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

On Wed, Mar 31, 2010 at 02:20:35AM -0700, Brian Harring wrote:
> Hola all-
> 
> For those who aren't familiar, pkg_pretend is in EAPI4- the main usage 
> of it is will be use dep checking- this email is specifically 
> regarding an alternative to it that *should* be superior for that use 
> case, but I'm looking for feedback.
> 
> Basically, we use the original VALID_USE proposal from way back in 
> '05- if you're familiar w/ MYOPTIONS, they're reasonably similar.
> 
> Roughly, VALID_USE is a list of constraints stating what the allowed 
> use flag combinations are for this pkg.  If you think of normal 
> depdencies (I must have openssl and python merged prior), it's the 
> same machinery.
> 
> [snip]
> 
> Comments desired; assuming no significant blowback, I'll be pushing 
> this to the council level since eapi4 is annoying feature locked right 
> now.

I like the whole concept, it is clean and straightforward.

Also, notifying the user of any possible issues and breaking so he can
make the actual decision sounds far better than making the choice for him.

VALID_USE does look a bit strange.

how about
	IUSE_RULES
or
	IUSE_RESTRICTIOMS
or
	RUSE
?

-- 
Alex Alexander :: wired
Gentoo Developer
www.linuxized.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Wed, Mar 31, 2010 at 08:49:26PM +0300, Alex Alexander wrote:
> VALID_USE does look a bit strange.
> 
> how about
> 	IUSE_RULES
> or
> 	IUSE_RESTRICTIOMS
> or
> 	RUSE
> ?
It's not really IUSE; the constraints it specifies applies to USE 
only.

USE_STATES, VALID_USES, VALID_USE_STATES, ALLOWED_USE, etc.

Actual name I don't hugely care about, I'm more interested in ensuring 
we don't rule out doing use cycle breaking via a bad design decision.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

On Wed, 31 Mar 2010 12:46:26 -0700
Brian Harring <ferringb@gmail.com> wrote:
> Actual name I don't hugely care about, I'm more interested in
> ensuring we don't rule out doing use cycle breaking via a bad design
> decision.

Cycle breaking requires explicit instructions from the ebuilds in
question (many of which are system things, which further complicates it)
along with support from Portage, so it's a distant future, lot of work
thing.

Since we need pkg_pretend to cover all the things that aren't use flag
related anyway, it makes sense to just go with that rather than
delaying things even further. When in the distant future Portage
becomes able to deal with cycle breaking, ebuilds can be converted to
use something like VALID_USE when they're also updated to export
information on which of their flags can safely be toggled.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3239 bytes --]

On Wed, Mar 31, 2010 at 08:56:28PM +0100, Ciaran McCreesh wrote:
> On Wed, 31 Mar 2010 12:46:26 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> > Actual name I don't hugely care about, I'm more interested in
> > ensuring we don't rule out doing use cycle breaking via a bad design
> > decision.
> 
> Cycle breaking requires explicit instructions from the ebuilds in
> question (many of which are system things, which further complicates it)
> along with support from Portage, so it's a distant future, lot of work
> thing.

Nonsense.  Note I said 'use cycle', not the generic 'cycle breaking'.  
USE induced cycles don't require explicit instructions from the 
ebuild at all- the PM itself can search the solution space (toggling 
flags as needed) to search out a way around the cycle.

Consider user configuration w/ USE=X, pkg_a w/ DEPEND "X? ( pkg_b )", 
pkg_b w/ DEPEND "pkg_a".  To be clear, you're claiming that the 
ebuild itself (and only the ebuild) is the the one able to state-

emerge pkg_a[-X]
emerge pkg_a[X]

As demonstrated, that cycle is easily broken.  A lot of the cycles 
users run into originate that way also.

Reiterating a point you're missing also, any use cycle a user hits is 
currently requires the *user* to sort it out anyways- what VALID_USE 
adds is the ability for the package manager to do it itself.

As for the "portage is developmentally slow" contribute frankly- per 
the norm w/ open source, you want something, ultimately you're the one 
responsible for the work.

Less contentious answer, I've already gotten an estimate of 2 weeks 
out of Luther (the person who has been knocking out EAPI4 features in 
the last month or so)- I'm not that concerned about it.  Actual work 
is a few days, motivation per the norm is the main time sink.


> Since we need pkg_pretend to cover all the things that aren't use flag
> related anyway, it makes sense to just go with that rather than
> delaying things even further.

And as I've already laid out in the bug, pkg_pretend has it's own set 
of issues when compared to pkg_setup due to it being non temporal, 
thus having high false positive potentials.

The main council push for pkg_pretend was to move use constraint 
checking to pre build.  VALID_USE does that cleaner and enabling use 
cycle breaking to be built; as such I'm pushing it up to them unless 
someone can find significant *real* flaws.

Soo... as I've described on the bug and here (repeatedly), exempting 
5-10 cases from the tree, what pkg_pretend enables can either be done 
better via VALID_USE, or is a degradation due to temporal concerns 
when you move the code out of pkg_setup.

Short version: it's a step backwards.


> When in the distant future Portage
> becomes able to deal with cycle breaking, ebuilds can be converted to
> use something like VALID_USE when they're also updated to export
> information on which of their flags can safely be toggled.

You're being short sighted.  VALID_USE is useful now for representing 
use states that are allowed; that data itself is useful for use cycle 
breaking.  Added bonus of enabling better functionality via a superior 
solutions, basically.


~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2594 bytes --]

On Thu, 1 Apr 2010 00:31:09 -0700
Brian Harring <ferringb@gmail.com> wrote:
> > Cycle breaking requires explicit instructions from the ebuilds in
> > question (many of which are system things, which further
> > complicates it) along with support from Portage, so it's a distant
> > future, lot of work thing.
> 
> Nonsense.  Note I said 'use cycle', not the generic 'cycle
> breaking'. USE induced cycles don't require explicit instructions
> from the ebuild at all- the PM itself can search the solution space
> (toggling flags as needed) to search out a way around the cycle.
> 
> Consider user configuration w/ USE=X, pkg_a w/ DEPEND "X? ( pkg_b )", 
> pkg_b w/ DEPEND "pkg_a".  To be clear, you're claiming that the 
> ebuild itself (and only the ebuild) is the the one able to state-
> 
> emerge pkg_a[-X]
> emerge pkg_a[X]
> 
> As demonstrated, that cycle is easily broken.  A lot of the cycles 
> users run into originate that way also.

Congratulations. You just turned on 'build' and 'bootstrap', and turned
off 'acl'.

> And as I've already laid out in the bug, pkg_pretend has it's own set 
> of issues when compared to pkg_setup due to it being non temporal, 
> thus having high false positive potentials.

These are exactly the same issues that pkg_setup has. You can't block a
useful feature simply because developers could theoretically screw
things up by using it.

> The main council push for pkg_pretend was to move use constraint 
> checking to pre build.  VALID_USE does that cleaner and enabling use 
> cycle breaking to be built; as such I'm pushing it up to them unless 
> someone can find significant *real* flaws.

No, VALID_USE addresses a *subset* of the issues. It's not a
replacement for pkg_pretend.

> > When in the distant future Portage
> > becomes able to deal with cycle breaking, ebuilds can be converted
> > to use something like VALID_USE when they're also updated to export
> > information on which of their flags can safely be toggled.
> 
> You're being short sighted.  VALID_USE is useful now for representing 
> use states that are allowed; that data itself is useful for use cycle 
> breaking.  Added bonus of enabling better functionality via a
> superior solutions, basically.

Simply adding VALID_USE won't let you do cycle breaking. You also need
extensive lists of which flags for which packages can safely be toggled
and when without breaking the system, and the only way you'll get those
lists is if developers care enough to update their ebuilds to provide
them.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 4916 bytes --]

On Thu, Apr 01, 2010 at 08:41:02AM +0100, Ciaran McCreesh wrote:
> On Thu, 1 Apr 2010 00:31:09 -0700
> > As demonstrated, that cycle is easily broken.  A lot of the cycles 
> > users run into originate that way also.
> 
> Congratulations. You just turned on 'build' and 'bootstrap', and turned
> off 'acl'.

Actually, I'm well aware I did.  See, if PMS wasn't developed in a 
void you'd know build, bootstrap, acl and friends were already a known 
issue with use cycle breaking.

Thing is, in checking the problem through we couldn't find a single 
instance where it was the *wrong* thing to do.

Build, boostrap, acl, etc, they're relevant only when you've got 
basically a from scratch system, or are rebuilding enough things you 
need to revert to that level.

Meaning it's the right damn thing to do- same thing the user would 
have to do.


> > And as I've already laid out in the bug, pkg_pretend has it's own set 
> > of issues when compared to pkg_setup due to it being non temporal, 
> > thus having high false positive potentials.
> 
> These are exactly the same issues that pkg_setup has. You can't block a
> useful feature simply because developers could theoretically screw
> things up by using it.

Ciaran, seriously stop lieing.  Your own words disprove this claim, 
I quote from pms/ebuild-functions.tex:

"""
The \t{pkg\_pretend} function may be used to carry out sanity 
checks early on in the install process. For example, if an ebuild 
requires a particular kernel configuration, it may perform that
check in \t{pkg\_pretend} and call \t{eerror} and then \t{die} with 
appropriate messages if the requirement is not met.

\t{pkg\_pretend} is run separately from the main phase function 
sequence, and does not participate in any kind of environment saving. 
There is no guarantee that any of an ebuild's dependencies will
be met at this stage, and no guarantee that the system state will not 
have changed substantially before the next phase is executed.
"""

It's ironic that the only example PMS can give for this functionality 
is als one that should not be implemented in pkg_pretend, but I 
digress.  Pure/simple, as I've explained repeatedly:

pkg_setup: ran just before the build of the pkg, after the pkg's 
DEPENDS are all built.  Meaning you *can* do has_version checks, 
kernel config checks, etc, because the proceeding deps are now 
satisfied.

pkg_pretend: ran before *every* *single* *build* has been ran, meaning 
the has_version check, the kernel config check, etc, all can invalidly 
die.

Had they been pkg_setup (check after DEPENDs are satisfied), the 
majority of the checks would pass, but because they're ran prior to 
DEPENDs satisfied, users will have to wind up breaking what was a 
single emerge invocation into multiple to satisfy pkg_pretend being 
wrong.


> > The main council push for pkg_pretend was to move use constraint 
> > checking to pre build.  VALID_USE does that cleaner and enabling use 
> > cycle breaking to be built; as such I'm pushing it up to them unless 
> > someone can find significant *real* flaws.
> 
> No, VALID_USE addresses a *subset* of the issues. It's not a
> replacement for pkg_pretend.

Cherry picking the argument again.  Main != whole, meaning the 
majority reason I could see w/in council logs for supporting 
pkg_pretend was USE constraint validation.

As I've said, and as you seem to finally understand, VALID_USE isn't a 
replacement for pkg_pretend- it just replaces the *main* usage of it.


> > > When in the distant future Portage
> > > becomes able to deal with cycle breaking, ebuilds can be converted
> > > to use something like VALID_USE when they're also updated to export
> > > information on which of their flags can safely be toggled.
> > 
> > You're being short sighted.  VALID_USE is useful now for representing 
> > use states that are allowed; that data itself is useful for use cycle 
> > breaking.  Added bonus of enabling better functionality via a
> > superior solutions, basically.
> 
> Simply adding VALID_USE won't let you do cycle breaking. You also need
> extensive lists of which flags for which packages can safely be toggled
> and when without breaking the system, and the only way you'll get those
> lists is if developers care enough to update their ebuilds to provide
> them.

That's one view, but sure, I'll run with it.

The thing is, *without* VALID_USE you cannot do use cycle breaking 
*period*.  executable vs data for the representation of the 
constraints (as I've spelled out for you 3 times now).

So your arguement against VALID_USE basically comes down to "it may 
not be able to do USE cycle breaking"- fine, I disagree, but whatever.  

pkg_pretend however completely disallows even *doing* use cycle 
breaking.  How in the hell is that a better next step?

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Gilles Dartiguelongue]

Le jeudi 01 avril 2010 à 00:56 -0700, Brian Harring a écrit :
[snip]
> pkg_setup: ran just before the build of the pkg, after the pkg's 
> DEPENDS are all built.  Meaning you *can* do has_version checks, 
> kernel config checks, etc, because the proceeding deps are now 
> satisfied.
> 
> pkg_pretend: ran before *every* *single* *build* has been ran,
> meaning 
> the has_version check, the kernel config check, etc, all can
> invalidly 
> die.
> 
> Had they been pkg_setup (check after DEPENDs are satisfied), the 
> majority of the checks would pass, but because they're ran prior to 
> DEPENDs satisfied, users will have to wind up breaking what was a 
> single emerge invocation into multiple to satisfy pkg_pretend being 
> wrong.
> 

jumping on the train here, but who said PM would not feed proper data to
pkg_pretend so it would behave like the DEPEND were already built. Could
some guy involved in a PM development tell us about how this would be
handled ?

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Thu, Apr 01, 2010 at 12:10:20PM +0200, Gilles Dartiguelongue wrote:
> jumping on the train here, but who said PM would not feed proper data to
> pkg_pretend so it would behave like the DEPEND were already built. Could
> some guy involved in a PM development tell us about how this would be
> handled ?

Good idea, but not really viable.  The only scenario where this would 
work cleanly is in has_version checks which most of the time should be 
blockers/deps anyways.

Basically, you want the PM to lie to the ebuild in some fashion.  
Since pkg_pretend is free form, it's effectively impossible to cover 
the scenarios it could check on- consider checking the kernel 
config/version, or checking the active jvm/python version.

Some of those can sort of be handled, but it requires a lot of custom 
code (code that has to change as the tools involved change) to pull it 
off.

As said, good idea, but it was ruled out due to it being techically 
infeasible considering the gains.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Gilles Dartiguelongue]

Le jeudi 01 avril 2010 à 03:18 -0700, Brian Harring a écrit :
> On Thu, Apr 01, 2010 at 12:10:20PM +0200, Gilles Dartiguelongue wrote:
> > jumping on the train here, but who said PM would not feed proper data to
> > pkg_pretend so it would behave like the DEPEND were already built. Could
> > some guy involved in a PM development tell us about how this would be
> > handled ?
> 
> Good idea, but not really viable.  The only scenario where this would 
> work cleanly is in has_version checks which most of the time should be 
> blockers/deps anyways.

That's indeed the only thing I was thinking of

> Basically, you want the PM to lie to the ebuild in some fashion.  
> Since pkg_pretend is free form, it's effectively impossible to cover 
> the scenarios it could check on- consider checking the kernel 
> config/version, or checking the active jvm/python version.

except the kernel will not change during the upgrade, unless you reboot
in the middle of the upgrade but I would expect the PM to recompute
pkg_pretend in resume.

> Some of those can sort of be handled, but it requires a lot of custom 
> code (code that has to change as the tools involved change) to pull it 
> off.
> 
> As said, good idea, but it was ruled out due to it being techically 
> infeasible considering the gains.

Since I have little insight on the rest I will trust whatever decision
has been taken.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

On Thu, Apr 01, 2010 at 12:42:10PM +0200, Gilles Dartiguelongue wrote:
> > Basically, you want the PM to lie to the ebuild in some fashion.  
> > Since pkg_pretend is free form, it's effectively impossible to cover 
> > the scenarios it could check on- consider checking the kernel 
> > config/version, or checking the active jvm/python version.
> 
> except the kernel will not change during the upgrade,

Pardon, I wasn't clear- I was referring to kernel sources, not 
the running kernel.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Thu, 1 Apr 2010 03:59:54 -0700
Brian Harring <ferringb@gmail.com> wrote:
> On Thu, Apr 01, 2010 at 12:42:10PM +0200, Gilles Dartiguelongue wrote:
> > > Basically, you want the PM to lie to the ebuild in some fashion.  
> > > Since pkg_pretend is free form, it's effectively impossible to
> > > cover the scenarios it could check on- consider checking the
> > > kernel config/version, or checking the active jvm/python version.
> > 
> > except the kernel will not change during the upgrade,
> 
> Pardon, I wasn't clear- I was referring to kernel sources, not 
> the running kernel.

But if the kernel sources symlink is changed by installing new kernel
sources, there won't be a valid .config in the new directory anyway.
Thus, pkg_pretend doesn't introduce any new breakage.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On Thu, Apr 01, 2010 at 12:23:42PM +0100, Ciaran McCreesh wrote:
> On Thu, 1 Apr 2010 03:59:54 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> > On Thu, Apr 01, 2010 at 12:42:10PM +0200, Gilles Dartiguelongue wrote:
> > > > Basically, you want the PM to lie to the ebuild in some fashion.  
> > > > Since pkg_pretend is free form, it's effectively impossible to
> > > > cover the scenarios it could check on- consider checking the
> > > > kernel config/version, or checking the active jvm/python version.
> > > 
> > > except the kernel will not change during the upgrade,
> > 
> > Pardon, I wasn't clear- I was referring to kernel sources, not 
> > the running kernel.
> 
> But if the kernel sources symlink is changed by installing new kernel
> sources, there won't be a valid .config in the new directory anyway.

Oddly enough, I actually have an ebuild that directly contradicts 
that- used for managing my sources w/in kvms.

> Thus, pkg_pretend doesn't introduce any new breakage.

Regardless of my own usage, ironically you just inadvertantly pointed 
out a whole class of false negatives pkg_pretend has.

Specifically, 

1) starting w/ a configured kernel at /usr/src/linux
2) merging a version of aufs2 requiring new kernel sources
3) emerge runs pkg_pretend.  aufs2 does it's checks w/in pkg_pretend, 
sees the old configured kernel and thinks things are fine
4) new kernel sources get merged.  /usr/src/linux is no lnger a 
configured kernel.
5) aufs2 blows up during its build due to an unconfigured 
kernel.

Since I'm in the mood for a Scooby Doo quote, "wraut-wroh".

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

On Thu, 1 Apr 2010 04:38:36 -0700
Brian Harring <ferringb@gmail.com> wrote:
> > But if the kernel sources symlink is changed by installing new
> > kernel sources, there won't be a valid .config in the new directory
> > anyway.
> 
> Oddly enough, I actually have an ebuild that directly contradicts 
> that- used for managing my sources w/in kvms.

Your own personal ebuilds are an exception that don't apply to
everyone else. For ebuilds using kernel-2.eclass with USE=symlink, a
missing .config is exactly what happens.

> > Thus, pkg_pretend doesn't introduce any new breakage.
> 
> Regardless of my own usage, ironically you just inadvertantly pointed 
> out a whole class of false negatives pkg_pretend has.
> 
> Specifically, 
> 
> 1) starting w/ a configured kernel at /usr/src/linux
> 2) merging a version of aufs2 requiring new kernel sources
> 3) emerge runs pkg_pretend.  aufs2 does it's checks w/in pkg_pretend, 
> sees the old configured kernel and thinks things are fine
> 4) new kernel sources get merged.  /usr/src/linux is no lnger a 
> configured kernel.
> 5) aufs2 blows up during its build due to an unconfigured 
> kernel.
> 
> Since I'm in the mood for a Scooby Doo quote, "wraut-wroh".

This isn't a breakage that's introduced by pkg_pretend. It's a breakage
that's already there.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3066 bytes --]

On Thu, 1 Apr 2010 00:56:08 -0700
Brian Harring <ferringb@gmail.com> wrote:
> Actually, I'm well aware I did.  See, if PMS wasn't developed in a 
> void you'd know build, bootstrap, acl and friends were already a
> known issue with use cycle breaking.

So since it's a known issue, why are you pushing for VALID_USE "because
it allows cycle breaking" without also pushing for a solution to flags
that can't be toggled at the same time?

> pkg_setup: ran just before the build of the pkg, after the pkg's 
> DEPENDS are all built.  Meaning you *can* do has_version checks, 
> kernel config checks, etc, because the proceeding deps are now 
> satisfied.

Except that they might change, because, as we established on the bug,
two packages that aren't interdependent can affect each other's
assumptions, and can be built in parallel. pkg_pretend does not alter
the problem here.

> Cherry picking the argument again.  Main != whole, meaning the 
> majority reason I could see w/in council logs for supporting 
> pkg_pretend was USE constraint validation.
> 
> As I've said, and as you seem to finally understand, VALID_USE isn't
> a replacement for pkg_pretend- it just replaces the *main* usage of
> it.

You said on the bug that you wanted pkg_pretend removed in favour of
VALID_USE. I don't object to VALID_USE; I object to you claiming that
it replaces pkg_pretend, and I object to you claiming that using
VALID_USE instead of pkg_pretend is enough to allow cycle breaking.

> > Simply adding VALID_USE won't let you do cycle breaking. You also
> > need extensive lists of which flags for which packages can safely
> > be toggled and when without breaking the system, and the only way
> > you'll get those lists is if developers care enough to update their
> > ebuilds to provide them.
> 
> That's one view, but sure, I'll run with it.
> 
> The thing is, *without* VALID_USE you cannot do use cycle breaking 
> *period*.  executable vs data for the representation of the 
> constraints (as I've spelled out for you 3 times now).

You also can't do it *with* VALID_USE, unless you also have extensive
help from ebuilds. Why are you pushing for VALID_USE without also
proposing a way for the package mangler to be told which flags it can
change?

> pkg_pretend however completely disallows even *doing* use cycle 
> breaking.  How in the hell is that a better next step?

pkg_pretend is a pragmatic, cheap solution that solves a larger number
of problems, whilst not ruling out anything that Portage will
realistically be able to do in a relevant timeframe.

If, in the distant future, Portage supports use cycle breaking, then
people can switch their ebuilds to use VALID_USE when they're also
updating their ebuilds to export the cycle breaking information the
package mangler requires to do it without trashing a system. But since
we don't know exactly what that information looks like yet, we might as
well just stick with the single solution that solves all of the
problems.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Summarizing ciaran's claim to end this nonsense-

VALID_USE isn't useful because use cycle breaking can't be done 
according to strictures he desires, as such VALID_USE is pointless 
because pkg_pretend can cover it.

It's a bit brief and likely left out an insult or two, but it's to 
the point at least and doesn't need more expounding of his views.

So does anyone _other_ then ciaran care to comment?  Thinking I may 
write this up as a glep as an attempt to start pushing PMS changes 
into a more visible and documented light.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

On Thu, 1 Apr 2010 04:18:27 -0700
Brian Harring <ferringb@gmail.com> wrote:
> Summarizing ciaran's claim to end this nonsense-
> 
> VALID_USE isn't useful because use cycle breaking can't be done 
> according to strictures he desires, as such VALID_USE is pointless 
> because pkg_pretend can cover it.
> 
> It's a bit brief and likely left out an insult or two, but it's to 
> the point at least and doesn't need more expounding of his views.

Brian, you may find it beneficial to learn that criticism of a proposal
is not the same as an insult, and that correct criticism remains valid
regardless of where it comes from. Once you learn to accept that, you
may start finding it considerably easier to work with large teams on
technical issues.

And no. My claim is that since VALID_USE on its own doesn't allow for
anything that can't also be done by pkg_pretend, and since pkg_pretend
can do things that developers and users need that VALID_USE can't, and
since pkg_pretend doesn't break anything that's not already broken,
VALID_USE shouldn't be a priority. The issue can be revisited for EAPI 5
when there's more Portage development time available, and when it is
revisited, it should be done with related issues such as ensuring that
cycle breaking is possible without randomly breaking user systems in
horrible ways.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[David Leverton]

On Thursday 01 April 2010 12:18:27 Brian Harring wrote:
> It's a bit brief and likely left out an insult or two

If anyone's been personal and insulting in this discussion, it isn't Ciaran.

I've seen this attitude on IRC too.  Funnily enough, you don't speak for other 
people, you don't decide what their opinions are, and you don't get to decree 
that they're insulting you (or worse, that they're /going to/ insult you) 
unless they are, in fact, insulting you.

As for the topic: the only real concern about VALID_USE that I've seen from 
anyone is about whether Portage can implement it reasonably soon.  Since some 
people think it can, how about picking some reasonable time period (allowing 
for bikeshedding about the syntax, of course), and if VALID_USE isn't 
supported in Portage by then but EAPI 4 is otherwise ready then we postpone 
VALID_USE until later?

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Dror Levin]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

 On Thu, Apr 1, 2010 at 20:04, David Leverton <levertond@googlemail.com>wrote:

> If anyone's been personal and insulting in this discussion, it isn't
> Ciaran.
>
I'll take that as an April Fools' day joke.


> As for the topic: the only real concern about VALID_USE that I've seen from
> anyone is about whether Portage can implement it reasonably soon.  Since
> some
> people think it can, how about picking some reasonable time period
> (allowing
> for bikeshedding about the syntax, of course), and if VALID_USE isn't
> supported in Portage by then but EAPI 4 is otherwise ready then we postpone
> VALID_USE until later?
>
Here's another suggestion: how about we don't impose any ridiculous
constraints on development and keep this discussion on the technological
side of the original proposal? If the only concerns are about the time it
will take to implement, then this suggestion must be perfect in all aspects
and so we should probably wait until it's done, even if is takes a loooong
time!

I really like this attitude, though. Once you're done criticizing the
technological aspects of some proposal you start raising concerns about how
hard it is to implement features for Portage, how long that takes, etc.
Well, since that's not really constructive, I suggest you keep those
concerns to yourself.

Dror Levin

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

On Thu, 1 Apr 2010 21:39:43 +0300
Dror Levin <spatz@gentoo.org> wrote:
> > If anyone's been personal and insulting in this discussion, it isn't
> > Ciaran.
>  
> I'll take that as an April Fools' day joke.

Could you point out where I've been personal and insulting in this
discussion please? I'd like to learn for future reference the kind of
technical criticisms that you mistake for insults so that I can phrase
them in a way less likely to mislead in the future.

> I really like this attitude, though. Once you're done criticizing the
> technological aspects of some proposal you start raising concerns
> about how hard it is to implement features for Portage, how long that
> takes, etc. Well, since that's not really constructive, I suggest you
> keep those concerns to yourself.

So you're saying that when designing EAPIs, we should no longer
consider Portage implementation time?

Currently, one of the requirements for including a feature in an EAPI
is that the Portage people expect to be able to deliver it quickly.
We've left out a huge number of widely requested features from previous
EAPIs simply because they weren't considered deliverable by Portage in
a realistic timeframe, and when selecting features we've been careful
to pick those that require the minimum total amount of work on the
Portage side. Hence pkg_pretend -- although a subset of its
functionality could be handled in other ways, it's considered most
practical to go for the single cheapest feature that implements
everything people need.

Would you prefer a perfect EAPI ten years from now and nothing until
then, or a better EAPI than one that we currently have one year from
now? The Council has been pretty explicit in wanting the latter, so if
you want policy to be changed to the former then you'll need to take it
up with them.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] pkg_pretend USE validation and VALID_USE alternative

[David Leverton]

On Thursday 01 April 2010 19:39:43 Dror Levin wrote:
> Here's another suggestion: how about we don't impose any ridiculous
> constraints on development and keep this discussion on the technological
> side of the original proposal?

It's not ridiculous to expect to have a new EAPI in a reasonable amount of 
time.  Other features are already done, so holding them back until this one 
is complete would itself be placing constraints on developers who have a use 
for the other features.