[gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hola all-

For those who aren't familiar, pkg_pretend is in EAPI4- the main usage 
of it is will be use dep checking- this email is specifically 
regarding an alternative to it that *should* be superior for that use 
case, but I'm looking for feedback.

Basically, we use the original VALID_USE proposal from way back in 
'05- if you're familiar w/ MYOPTIONS, they're reasonably similar.

Roughly, VALID_USE is a list of constraints stating what the allowed 
use flag combinations are for this pkg.  If you think of normal 
depdencies (I must have openssl and python merged prior), it's the 
same machinery.

Examples:

# if build is set, python and openssl must be unset
VALID_USE="build? ( !python !openssl )"

# if mysql is set, sqlite must not be, and vice versa.
# note mysql/sqlite do *not* have to be set also.
VALID_USE="mysql? ( !sqlite ) !sqlite? ( mysql )"

# mysql or sqlite must be set; exclusive or.
VALID_USE="mysql? ( !sqlite ) !mysql ( sqlite )"

# alternative syntax, adding an xor group operator
# note xor isn't required- you can do the same thing
# via spelling it out, it's just a convenient thing to have.
VALID_USE="^^ ( mysql sqlite )"

# if gui is enabled, a widget set must be specified- can build
# multiple widget bindings
VALID_USE="X? ( || ( gtk qt motif ) )"


Note that like dependencies, these are assertions.  More importantly, 
they're also data rather then executable code.  Via it being data, up 
front GUI's can tell you exactly what conflicts arise, and what needs 
to be adjusted.

Doing it as executable, can, but it's iterative and reliant on the dev 
writing a clear message every time (rather then the UI tool getting 
it right once).  Clarifying iterative, consider

USE="build X"
w/ valid use states being-
VALID_USE="build? ( !X !python ) X? ( ^^ ( gtk qt ) ) gtk? ( ssl )"

The user first flips off build.  They rerun emerge- next they're told 
"you must choose gtk or qt, not both".  They change to USE="X gtk".  
Next they're told "you need ssl turned on if you want gtk".

At this point, the user goes and gets a beer because aparently Murphy 
hates them and it's going to be a long, long night.


There also is one major issue with relying on pkg_pretend 
(executable) for use state validation vs doing it as VALID_USE 
(data)- the package manager cannot know what states are valid thus 
limiting the things it can do.  Literally the pkg manager is screwed 
when it comes to use cycle breaking.  If the pkg manager doesn't know 
what states are valid, when it encounters a use cycle it doesn't know 
what intermediate builds it can do to break that cycle- literally if 
USE state validation is in pkg_pretend, the *user* will have to walk 
the PM through breaking the cycle instead of the PM figuring out the 
proper steps to break it.

Executive summary: if use validation is implemented via pkg_pretend,
1) it still has the iterative issue
2) it's harder for configuration tools to deal with (iterative issues 
above, plus the fact they get a blob of text they're stuck trying to 
parse)
3) it pretty much eliminates the possibility of doing use cycle 
breaking properly (which is already an issue, only going to get worse- 
hence why this was proposed in '04/'05 when we started playing w/ use 
deps in portage 3 at the time).


Comments desired; assuming no significant blowback, I'll be pushing 
this to the council level since eapi4 is annoying feature locked right 
now.

Thanks-
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Ulrich Mueller]

>>>>> On Wed, 31 Mar 2010, Brian Harring wrote:

> Roughly, VALID_USE is a list of constraints stating what the allowed
> use flag combinations are for this pkg. If you think of normal
> depdencies (I must have openssl and python merged prior), it's the
> same machinery.

Maybe we should first discuss if we want to drop the following
rule [1] which your proposal seems to contradict:

| Occasionally, ebuilds will have conflicting USE flags for
| functionality. Checking for them and returning an error is not a
| viable solution. Instead, you must pick one of the USE flags in
| conflict to favour.

Ulrich

[1] <http://devmanual.gentoo.org/general-concepts/use-flags/>

Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Alistair Bush]

> Hola all-
>  
> Comments desired; assuming no significant blowback, I'll be pushing
> this to the council level since eapi4 is annoying feature locked right
> now.
> 

I think this solution is far better,  until someone smarter than me tells me 
otherwise.

Don't know whether I like the VALID_USE var name,  so please at least think 
about something a little better if you can ;).  ( I like green,  but not too 
green if you know what I mean )

Will we still have to define the use flags in IUSE?

I'm guessing we can't use IUSE to store this information because of the  whole 
glep-55 thing.  IUSE would be my preferred var.  both defining the 
"relationships" and the use flags in only one go.

> Thanks-
> ~harring

Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

Note that while I inadvertantly cross posted (I was intending on 
cc'ing council@gentoo.org, not the ml), doubt they need to be cc'd 
further- my original attention was to effectively ensure they were 
paying aware of the details of this so that when I took it to them 
folk were informed.

CC'ing gentoo-council so folk following it there know it moved 
over to -dev.  Your discussion of devmanual relevance needs some -dev 
consensus anyways before the council should be deciding on it.

Also the cross posting is making betelgeuse cry anyways (and pissing 
off my procmail setup) ;)


On Wed, Mar 31, 2010 at 11:48:37AM +0200, Ulrich Mueller wrote:
> >>>>> On Wed, 31 Mar 2010, Brian Harring wrote:
> 
> > Roughly, VALID_USE is a list of constraints stating what the allowed
> > use flag combinations are for this pkg. If you think of normal
> > depdencies (I must have openssl and python merged prior), it's the
> > same machinery.
> 
> Maybe we should first discuss if we want to drop the following
> rule [1] which your proposal seems to contradict:

Not just my proposal- council contradicted it via even letting 
pkg_pretend into EAPI3 (now EAPI4):

http://www.mail-archive.com/gentoo-council@lists.gentoo.org/msg00493.html


> | Occasionally, ebuilds will have conflicting USE flags for
> | functionality. Checking for them and returning an error is not a
> | viable solution. Instead, you must pick one of the USE flags in
> | conflict to favour.
> 
> [1] <http://devmanual.gentoo.org/general-concepts/use-flags/>

I honestly consider the ebuild silently making decisions on the user's
behalf *worse*.  Consider if openoffice silently made decisions like 
that- 4 hours later it'll wind up choosing the option you didn't 
really want and you'll be in a foul mood.

Frankly is the devmanual even relevant on at this point beyond good 
practices btw?  Last I looked through it, there was a rather unhealthy 
mix of good policy that we follow, and policy that isn't relevant 
anymore- in need of some cleanup at the very least.


~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-council] pkg_pretend USE validation and VALID_USE alternative

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

Note I inadvertantly cross posted, I was intending on cc'ing 
council@gentoo.org.

As such one final cc to that ml to end this subthread while pulling 
this back to -dev.


On Wed, Mar 31, 2010 at 11:16:22PM +1300, Alistair Bush wrote:
> > Hola all-
> >  
> > Comments desired; assuming no significant blowback, I'll be pushing
> > this to the council level since eapi4 is annoying feature locked right
> 
> I think this solution is far better,  until someone smarter than me tells me 
> otherwise.
> 
> Don't know whether I like the VALID_USE var name,  so please at least think 
> about something a little better if you can ;).  ( I like green,  but not too 
> green if you know what I mean )
>
> Will we still have to define the use flags in IUSE?

Yes, although if folks have a better proposal that incorporates 
VALID_USE into IUSE I'm definitely open to it- the original proposal 
for VALID_USE tried to inline it into IUSE, but it got ugly (hence it 
mutating it this form).

The problem w/ trying to reuse IUSE is the following (sorry, I like 
lists of assertions)-

*) IUSE currently serves as a list of valid USE flags, just that.  No 
repeat specification of a flag (which means the dev in question is 
unlikely to typo a flag).

*) having a single specified list of valid use flags is the basis for 
doing validation of use flags used in all other metadata.  In other 
words, that list of valid use flags *really* needs to go out of it's 
way to make human error hard.

*) VALID_USE is a set of assertions on the allowed state of USE; IUSE 
is just a list of flags.  Intermixing the two in a way that is still 
readable is really ugly

*) given a library that has optional perl and python bindings 
(which can be toggled freely, they're standalone flags) I've nfc how 
one would sanely specify that w/in IUSE while adding VALID_USE 
semantics.  Possibly, you could include use conditionals into IUSE, 
and treat the () contents as assertions- but that makes adding xor in 
hard, is rather ugly/hard on the eyes, and violates the DRY (Don't 
Repeat Yourself) principle from above.

Definitely open to counterproposals that address those 
concerns however...


> I'm guessing we can't use IUSE to store this information because of the  whole 
> glep-55 thing.

glep-55 is unrelated to this as far as I can tell- if you think 
otherwise please clarify.

Thanks
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]