[gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring.

[gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring.

[Brian Harring]

Hola.

CC'ing pms since obviously they should comment, although the 
discussion should be on -dev (aka, public, not an alias).

Sorry, this is a long email; condensing it down into a glep is viable, 
just my time is limited (it's telling I started the gentoo stuff at 
2am, wrapping up likely at 4am).

The length of this email basically relates to the subtlety of this 
proposal; at first glance it may appear as if I'm proposing a syntax 
shuffle just for aesthetic reasons, but as elaborated in the tail end, 
that subtle syntax change opens up a lot of possibilities.


Either way, what I propose, is pretty simple.  Currently USE_EXPAND'd 
groups are pushed into USE in the form ${use_group}_${setting}.

Thus VIDEO_CARDS="intel nvidia" becomes video_cards_intel 
video_cards_nvidia.  This should be reasonably well known and 
understood from the dev standpoint; from the PM side of things, this 
actually seriously sucks ass; if we see video_cards_intel in an IUSE 
somewhere, we don't know if it's a USE flag, or a use_expanded group; 
to figure that out, we basically have to do a longest prefix matching 
(or suffix tree).

There in is one of the subtle design flaws of it; two USE_EXPAND'd 
targets, VIDEO_CARDS and VIDEO, can intersect in their namespace.  
While we (devs) intentionally avoid this potential via careful name 
choice, the fact the possibility exists is a bit of a sign that 
USE_EXPAND was a bit of a hack.

What I propose is simple enough, and is for >=EAPI5:

1) We disallow '@' in USE flags (yes, a use flag can actually have 
'@' in it's name according to PMS; someone was hitting the crack 
pipe pretty damn hard when they allowed that one).  This doesn't 
impact anything in gentoo-x86, nor any tree I've ever seen.

2) If <EAPI5, we continue exporting video_cards_nvidia 
(${use_group}_${setting}).

3) If >=EAPI5, we export ${use_group}_${setting}, *and* 
${use_group}@${setting}.  While there may be complaints about having 
it exported in two fashions, the reason for this is simple- it makaes 
transition way the fuck easier, meaning we don't make devs wrie pain 
in the ass "if eapi x, else blah" code.  Basically, for eclass code, 
the dev would do thus:
3.a) If the eclass supports <EAPI5, they check for the old form; 
${use_group}_${setting}.
3.b) If however that eclass is >=EAPI5, they should check for the new 
form; ${use_group}@{setting}.

4) At some point in the future, when either code involved has migrated 
all to EAPI5, or we do something in a later EAPI that renders 
supporting ${use_group}_${setting} untenuable, we stop exporting 
${use_group}_${setting} for those EAPIs.  Preferably with one EAPI 
cycle w/ deprecation warnings.

Basically, this approach is zero impact for devs for the most part; 
you don't have to do multiple paths for checking use wise if 
supporting EAPI5 in parallel to others.  Once your code is EAPI5 (and 
above), you use the new syntax- or worst case, someone interested goes 
and converts your >=EAPI5 ebuild/eclass to use the new syntax on your 
behalf.

Thus far, I've not particularly sold this notion in this email- I 
focused on making damn clear this cleanup is basically no cost for 
everyone but PM authors (and it's easy enough for us to handle).

Aside from the cleanup/sanity improvements to our algos for 
backmatching USE_EXPAND groups, this syntax gets us actually some 
rathre nifty shit; it basically allows us to introduce a generation 
configuration space.

So... onto examples of why we should do this beyond the basic "it's a 
cleaner syntax/design" argument:

--Per phase RESTRICT usage--
Exactly as it sounds.  Consider userpriv- most of the time that's 
turned off, that's purely because one phase- literally just one 
phase- needs to do something via root.  Thus we weaken the entirety 
of the protection.  Now, we could do some bastardly shit w/ existing 
use_expand for this (which has some conflict potentials for the PM), 
or via this syntax, we could do:

RESTRICT="phase@src_install? ( userpriv )"

Now, not all restricts are phase level- fetch, strip, installsources, 
etc, operate on a global level.  userpriv, usersandbox, sandbox 
(latter two aren't used in gentoo-x86, although PM's support it still) 
are phase specific.  That may seem like a small list, but consider 
that

1) pkgcore has fakeroot code; it's had it for a while, I'm intending 
on resurrecting that; this requires a RESTRICT='fakeroot' however.
2) I intend to start abusing the 'unshare' utility (man 1 unshare) to 
contain ebuilds build process; basically gives us a way to level some 
protections for the build host.  I'd be very surprised if other PM's 
don't follow suit (or do it before I do even); it's a simple 
protection that's worth doing.
3) There's been repeated talk of using containers/overlays to 
constrain the build process; sooner or later someones going to do it, 
and you can be damn sure a restrict will be necessary.

Etc, you get the idea; the point of the new syntax is general 
flexibility and allowing us to be finegrained in RESTRICT where 
useful.


--A form of unified DEPENDENCIES that fits into our existing style--

First, I'd appreciate it if the frothing over DEPENDENCIES wasn't 
involved here; this syntax lends itself to DEPENDENCIES, the debate 
about whether or not to do DEPENDENCIES is orthogonal.

Either way, we could do thus:

DEPENDENCIES="
dep@build,run? ( common deps/rdeps )
dep@run? ( rdep specific )"

Etc.  There's a subtly there; specifically that it's a 
conditional w/ multiple settings being checked.  Now, we could either 
do that as per USE_GROUP (moreso, per conditional namespace- dep being 
a namespace, just like 'phase' was one from earlier) rules, which I 
think sucks, or we could come up with common syntax there, which leads 
into the next potential feature we can build via this:

-- Reducing duplication, and avoiding multi-slot issues --

Picking on mesa, the following deps:

>=x11-libs/libdrm-2.4.24[video_cards_nouveau?,video_cards_vmware?]
video_cards_nouveau? ( <x11-libs/libdrm-2.4.34 )
video_cards_intel? ( >=x11-libs/libdrm-2.4.24[video_cards_intel] )
video_cards_radeon? ( >=x11-libs/libdrm-2.4.24[video_cards_radeon] )

Could instead be written as:

>=x11-libs/libdrm-2.4.24[video_cards@nouveau,vmware?]
video_cards@nouveau,intel,radeon? ( 
  <x11-libs/libdrm-2.4.34[video_cards@intel,radeon?] )

Another example, just from randomly poking through the tree: 
app-admin/chef-server-api; I get the feeling this likely applies to 
most ruby pkgs.  Abbreviated rdeps are thus:

ruby_targets_ruby18? (
  ~app-admin/chef-10.12.0[ruby_targets_ruby18]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets_ruby18]
  .. 12 more pkgs w/ similar use enforcement ..
)

ruby_targets_ruby19? (
  ~app-admin/chef-10.12.0[ruby_targets_ruby19]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets_ruby19]
  .. 12 more pkgs w/ similar use enforcement ..
)

Instead, it could be written as thus:

ruby_targets@ruby18,ruby19? (
  ~app-admin/chef-10.12.0[ruby_targets@ruby18,ruby19?]]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets@ruby18,ruby19?]
  .. 12 more pkgs w/ similar use enforcement ..
)

Upshot of that approach, it's effectively a single atom- meaning we 
don't have to worry about the scenario where (just hypothetically 
speaking) app-admin/chef was a slotted pkg; the current form they 
have in the tree, were it slotted, the resolver could pull in two 
there even if the pkg didn't want it to.

I'd be curious exactly what people think of that extension; 
technically, there is no reason it couldn't be used outside of just 
use_deps also.

Finally, keep in mind that the syntax for multiple flags there is 
intentionally a bit loose; that can be debated/bikeshedded to hell; 
the point however is that while we could try doing similar w/ old 
USE_EXPAND syntax, that's fairly ugly; the new '@' syntax makes this 
easier and opens up potentials.


-- ability to do testing of use groups (multiple flags) as a whole --

Continuing from the ruby example, we have a ruby use flag, then 
use_expanded ruby_targets_ruby${version}.  There's no reason the ruby 
use flag couldn't be killed off, and instead, allow 'ruby@? ( 
ruby-deps )' syntax- or something possibly more syntactically 
tasteful, 'ruby@*? ( ruby-deps )'.


-- Sub use groups --

There's no reason we couldn't do nesting of namespaces here; 
apache2_modules for example has the following optionals:

proxy - HTTP/1.1 proxy/gateway server
proxy_ajp - AJP support module for mod_proxy
proxy_balancer - mod_proxy extension for load balancing
proxy_connect - mod_proxy extension for CONNECT request handling
proxy_ftp - FTP support module for mod_proxy
proxy_http - HTTP support module for mod_proxy
proxy_scgi - SCGI gateway module for mod_proxy

If we wanted to, w/ the new syntax we could fairly easily nest that, 
such that it's apache2_modules@proxy@ajp, 
apache2_modules@proxy@balancer... you get the idea.  Note this isn't 
required- it just becomes sanely possible w/ the syntax.

-- REQUIRED_USE enhancement --

It's late, so I'm going light on the details for this; essentially, 
the ability to address a bunch of flags at once I'm strongly wager 
devs will find a use for.  My brains a bit fried, but if folks have an 
example in this case, please chime in.


Again, sorry for the length, but hopefully having read this far, 
it wasn't a waste of your time. ;)

If folks want to pick at specific sub-uses of the syntax, please feel 
free to change the subject for that feature and derive a thread from 
it (think git patchset uploads, and you'll grok the notion).

~harring

Re: [gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring.

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]

On Thu, 2012-09-13 at 03:39 -0700, Brian Harring wrote:
> Hola.
> 
> CC'ing pms since obviously they should comment, although the 
> discussion should be on -dev (aka, public, not an alias).
> 
> Sorry, this is a long email; condensing it down into a glep is viable, 
> just my time is limited (it's telling I started the gentoo stuff at 
> 2am, wrapping up likely at 4am).
> 
> The length of this email basically relates to the subtlety of this 
> proposal; at first glance it may appear as if I'm proposing a syntax 
> shuffle just for aesthetic reasons, but as elaborated in the tail end, 
> that subtle syntax change opens up a lot of possibilities.
> 
> 
> Either way, what I propose, is pretty simple.  Currently USE_EXPAND'd 
> groups are pushed into USE in the form ${use_group}_${setting}.
> 
> Thus VIDEO_CARDS="intel nvidia" becomes video_cards_intel 
> video_cards_nvidia.  This should be reasonably well known and 
> understood from the dev standpoint; from the PM side of things, this 
> actually seriously sucks ass; if we see video_cards_intel in an IUSE 
> somewhere, we don't know if it's a USE flag, or a use_expanded group; 
> to figure that out, we basically have to do a longest prefix matching 
> (or suffix tree).
> 
> There in is one of the subtle design flaws of it; two USE_EXPAND'd 
> targets, VIDEO_CARDS and VIDEO, can intersect in their namespace.  
> While we (devs) intentionally avoid this potential via careful name 
> choice, the fact the possibility exists is a bit of a sign that 
> USE_EXPAND was a bit of a hack.
> 
> What I propose is simple enough, and is for >=EAPI5:
> 
> 1) We disallow '@' in USE flags (yes, a use flag can actually have 
> '@' in it's name according to PMS; someone was hitting the crack 
> pipe pretty damn hard when they allowed that one).  This doesn't 
> impact anything in gentoo-x86, nor any tree I've ever seen.
> 
> 2) If <EAPI5, we continue exporting video_cards_nvidia 
> (${use_group}_${setting}).
> 
> 3) If >=EAPI5, we export ${use_group}_${setting}, *and* 
> ${use_group}@${setting}.  While there may be complaints about having 
> it exported in two fashions, the reason for this is simple- it makaes 
> transition way the fuck easier, meaning we don't make devs wrie pain 
> in the ass "if eapi x, else blah" code.  Basically, for eclass code, 
> the dev would do thus:
> 3.a) If the eclass supports <EAPI5, they check for the old form; 
> ${use_group}_${setting}.
> 3.b) If however that eclass is >=EAPI5, they should check for the new 
> form; ${use_group}@{setting}.
> 
> 4) At some point in the future, when either code involved has migrated 
> all to EAPI5, or we do something in a later EAPI that renders 
> supporting ${use_group}_${setting} untenuable, we stop exporting 
> ${use_group}_${setting} for those EAPIs.  Preferably with one EAPI 
> cycle w/ deprecation warnings.
> 
...
> ~harring
> 

+1

This would make things easier for PM consumer apps to process use flags
without specialized code to figure out if the flag was use_expanded or
just had an "_" in it's name...

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring: bikeshedding the separator

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Thu, 13 Sep 2012 03:39:19 -0700
Brian Harring <ferringb@gmail.com> wrote:
> 1) We disallow '@' in USE flags (yes, a use flag can actually have 
> '@' in it's name according to PMS; someone was hitting the crack 
> pipe pretty damn hard when they allowed that one).  This doesn't 
> impact anything in gentoo-x86, nor any tree I've ever seen.

No crack pipe was involved in that decision! It's because of LINGUAS.

(Incidentally, we used : rather than @ for separation for exheres-0 for
that reason.)

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring: bikeshedding the separator

[Brian Harring]

On Thu, Sep 13, 2012 at 04:24:27PM +0100, Ciaran McCreesh wrote:
> On Thu, 13 Sep 2012 03:39:19 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> > 1) We disallow '@' in USE flags (yes, a use flag can actually have 
> > '@' in it's name according to PMS; someone was hitting the crack 
> > pipe pretty damn hard when they allowed that one).  This doesn't 
> > impact anything in gentoo-x86, nor any tree I've ever seen.
> 
> No crack pipe was involved in that decision! It's because of LINGUAS.
> 
> (Incidentally, we used : rather than @ for separation for exheres-0 for
> that reason.)

Who says Linguas definition didn't involve a bit 'o crack? ;)  On that 
subject, not entirely sure how the hell a grepp'ing of profiles on my 
part missed that file; worse, I distinctly recall this coming up in 
the past.  Suspect it's time we add a footnote to that section of 
names since it's non-obvious.  Sigh...

Two angles;

1) Mind you, the woken up/caffeine ratio currently blows so this 
could be quite off kilter- but at first glance the '@' linguas usage 
actually seems to map to sub use groups (both in intent and 
grouping), at least for the quick scan I did of what we use.  Might 
not actually be an issue, iow if we allow that, although that's 
assuming the '@' subgroup approach translates reasonably cleanly.

The potential failure I'd see with that approach is it's a bit reliant 
on the assumption that the rules:

`language[_territory[.codeset]][@modifier]`

have been abided by- that the modifier is just that, a modifier.  
Were we to do this, which, at least at first sight, seems like a nifty 
solution that addresses it, we'd *likely* want sub groups to have a 
rule such that if you try to expand the subgroup of a setting, and 
that setting isn't a subgroup, it is considered 'on'.  Basically, for 
the linguas case, it kind of sucks if we get into the situation where 
the consuming ebuilds/eclasses has to be sensitive to which modifiers 
were exposed.  Essentially:

linguas@de_DE@, if not a subgroup to expand to, is treated as scalar, 
rather than list.  Impliciations of that I've not yet fully thought 
out- note that tweak isn't necessary for the basic notion of #1 to be 
usable also.


2) bikeshedding potentials: just spitballing a couple of potentials 
if '@' subgroups there doesn't fly for folks reading (mean it nicely, 
we shouldn't diverge arbitarily, but in the same instant we shouldn't 
import syntax/notions from exherbo unless they explicitly make sense 
in the gentoo/PMS context; the formats diverge enough the "reuse for 
compatibility" argument doesn't hold much water):

ruby_targets@ruby_18
ruby_targets:ruby_18
ruby_targets|ruby_18
ruby_targets(ruby_18)

Potentially fun thought, although the syntax is kind of ugly; 
basically we treat 'ruby_target' as a matching target (restriction in 
pkgcore vernacular, something that matches something else); the nice 
thing about this is it naturally plugins into the notion of multiple 
settings:

ruby_targets[ruby18]
ruby_targets[ruby18,ruby19]

In the same angle, while it's partially valid bash (and not in the 
single setting case):
ruby_targets{ruby_18}
ruby_targets{ruby_18,ruby_19}

That said, I consider the [] and {} syntax absolutely freaking ugly to 
the eye.  I mention it should others think it's not butt-fugly.

If approach #1 doesn't fly, using ':' as the delimiter gets my vote.

~harring