[gentoo-pms] USE_EXPAND / USE 'configuration space' refactoring.

[Brian Harring <ferringb@gmail.com>]

Hola.

CC'ing pms since obviously they should comment, although the 
discussion should be on -dev (aka, public, not an alias).

Sorry, this is a long email; condensing it down into a glep is viable, 
just my time is limited (it's telling I started the gentoo stuff at 
2am, wrapping up likely at 4am).

The length of this email basically relates to the subtlety of this 
proposal; at first glance it may appear as if I'm proposing a syntax 
shuffle just for aesthetic reasons, but as elaborated in the tail end, 
that subtle syntax change opens up a lot of possibilities.


Either way, what I propose, is pretty simple.  Currently USE_EXPAND'd 
groups are pushed into USE in the form ${use_group}_${setting}.

Thus VIDEO_CARDS="intel nvidia" becomes video_cards_intel 
video_cards_nvidia.  This should be reasonably well known and 
understood from the dev standpoint; from the PM side of things, this 
actually seriously sucks ass; if we see video_cards_intel in an IUSE 
somewhere, we don't know if it's a USE flag, or a use_expanded group; 
to figure that out, we basically have to do a longest prefix matching 
(or suffix tree).

There in is one of the subtle design flaws of it; two USE_EXPAND'd 
targets, VIDEO_CARDS and VIDEO, can intersect in their namespace.  
While we (devs) intentionally avoid this potential via careful name 
choice, the fact the possibility exists is a bit of a sign that 
USE_EXPAND was a bit of a hack.

What I propose is simple enough, and is for >=EAPI5:

1) We disallow '@' in USE flags (yes, a use flag can actually have 
'@' in it's name according to PMS; someone was hitting the crack 
pipe pretty damn hard when they allowed that one).  This doesn't 
impact anything in gentoo-x86, nor any tree I've ever seen.

2) If <EAPI5, we continue exporting video_cards_nvidia 
(${use_group}_${setting}).

3) If >=EAPI5, we export ${use_group}_${setting}, *and* 
${use_group}@${setting}.  While there may be complaints about having 
it exported in two fashions, the reason for this is simple- it makaes 
transition way the fuck easier, meaning we don't make devs wrie pain 
in the ass "if eapi x, else blah" code.  Basically, for eclass code, 
the dev would do thus:
3.a) If the eclass supports <EAPI5, they check for the old form; 
${use_group}_${setting}.
3.b) If however that eclass is >=EAPI5, they should check for the new 
form; ${use_group}@{setting}.

4) At some point in the future, when either code involved has migrated 
all to EAPI5, or we do something in a later EAPI that renders 
supporting ${use_group}_${setting} untenuable, we stop exporting 
${use_group}_${setting} for those EAPIs.  Preferably with one EAPI 
cycle w/ deprecation warnings.

Basically, this approach is zero impact for devs for the most part; 
you don't have to do multiple paths for checking use wise if 
supporting EAPI5 in parallel to others.  Once your code is EAPI5 (and 
above), you use the new syntax- or worst case, someone interested goes 
and converts your >=EAPI5 ebuild/eclass to use the new syntax on your 
behalf.

Thus far, I've not particularly sold this notion in this email- I 
focused on making damn clear this cleanup is basically no cost for 
everyone but PM authors (and it's easy enough for us to handle).

Aside from the cleanup/sanity improvements to our algos for 
backmatching USE_EXPAND groups, this syntax gets us actually some 
rathre nifty shit; it basically allows us to introduce a generation 
configuration space.

So... onto examples of why we should do this beyond the basic "it's a 
cleaner syntax/design" argument:

--Per phase RESTRICT usage--
Exactly as it sounds.  Consider userpriv- most of the time that's 
turned off, that's purely because one phase- literally just one 
phase- needs to do something via root.  Thus we weaken the entirety 
of the protection.  Now, we could do some bastardly shit w/ existing 
use_expand for this (which has some conflict potentials for the PM), 
or via this syntax, we could do:

RESTRICT="phase@src_install? ( userpriv )"

Now, not all restricts are phase level- fetch, strip, installsources, 
etc, operate on a global level.  userpriv, usersandbox, sandbox 
(latter two aren't used in gentoo-x86, although PM's support it still) 
are phase specific.  That may seem like a small list, but consider 
that

1) pkgcore has fakeroot code; it's had it for a while, I'm intending 
on resurrecting that; this requires a RESTRICT='fakeroot' however.
2) I intend to start abusing the 'unshare' utility (man 1 unshare) to 
contain ebuilds build process; basically gives us a way to level some 
protections for the build host.  I'd be very surprised if other PM's 
don't follow suit (or do it before I do even); it's a simple 
protection that's worth doing.
3) There's been repeated talk of using containers/overlays to 
constrain the build process; sooner or later someones going to do it, 
and you can be damn sure a restrict will be necessary.

Etc, you get the idea; the point of the new syntax is general 
flexibility and allowing us to be finegrained in RESTRICT where 
useful.


--A form of unified DEPENDENCIES that fits into our existing style--

First, I'd appreciate it if the frothing over DEPENDENCIES wasn't 
involved here; this syntax lends itself to DEPENDENCIES, the debate 
about whether or not to do DEPENDENCIES is orthogonal.

Either way, we could do thus:

DEPENDENCIES="
dep@build,run? ( common deps/rdeps )
dep@run? ( rdep specific )"

Etc.  There's a subtly there; specifically that it's a 
conditional w/ multiple settings being checked.  Now, we could either 
do that as per USE_GROUP (moreso, per conditional namespace- dep being 
a namespace, just like 'phase' was one from earlier) rules, which I 
think sucks, or we could come up with common syntax there, which leads 
into the next potential feature we can build via this:

-- Reducing duplication, and avoiding multi-slot issues --

Picking on mesa, the following deps:

>=x11-libs/libdrm-2.4.24[video_cards_nouveau?,video_cards_vmware?]
video_cards_nouveau? ( <x11-libs/libdrm-2.4.34 )
video_cards_intel? ( >=x11-libs/libdrm-2.4.24[video_cards_intel] )
video_cards_radeon? ( >=x11-libs/libdrm-2.4.24[video_cards_radeon] )

Could instead be written as:

>=x11-libs/libdrm-2.4.24[video_cards@nouveau,vmware?]
video_cards@nouveau,intel,radeon? ( 
  <x11-libs/libdrm-2.4.34[video_cards@intel,radeon?] )

Another example, just from randomly poking through the tree: 
app-admin/chef-server-api; I get the feeling this likely applies to 
most ruby pkgs.  Abbreviated rdeps are thus:

ruby_targets_ruby18? (
  ~app-admin/chef-10.12.0[ruby_targets_ruby18]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets_ruby18]
  .. 12 more pkgs w/ similar use enforcement ..
)

ruby_targets_ruby19? (
  ~app-admin/chef-10.12.0[ruby_targets_ruby19]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets_ruby19]
  .. 12 more pkgs w/ similar use enforcement ..
)

Instead, it could be written as thus:

ruby_targets@ruby18,ruby19? (
  ~app-admin/chef-10.12.0[ruby_targets@ruby18,ruby19?]]
  >=dev-ruby/dep_selector-0.0.3[ruby_targets@ruby18,ruby19?]
  .. 12 more pkgs w/ similar use enforcement ..
)

Upshot of that approach, it's effectively a single atom- meaning we 
don't have to worry about the scenario where (just hypothetically 
speaking) app-admin/chef was a slotted pkg; the current form they 
have in the tree, were it slotted, the resolver could pull in two 
there even if the pkg didn't want it to.

I'd be curious exactly what people think of that extension; 
technically, there is no reason it couldn't be used outside of just 
use_deps also.

Finally, keep in mind that the syntax for multiple flags there is 
intentionally a bit loose; that can be debated/bikeshedded to hell; 
the point however is that while we could try doing similar w/ old 
USE_EXPAND syntax, that's fairly ugly; the new '@' syntax makes this 
easier and opens up potentials.


-- ability to do testing of use groups (multiple flags) as a whole --

Continuing from the ruby example, we have a ruby use flag, then 
use_expanded ruby_targets_ruby${version}.  There's no reason the ruby 
use flag couldn't be killed off, and instead, allow 'ruby@? ( 
ruby-deps )' syntax- or something possibly more syntactically 
tasteful, 'ruby@*? ( ruby-deps )'.


-- Sub use groups --

There's no reason we couldn't do nesting of namespaces here; 
apache2_modules for example has the following optionals:

proxy - HTTP/1.1 proxy/gateway server
proxy_ajp - AJP support module for mod_proxy
proxy_balancer - mod_proxy extension for load balancing
proxy_connect - mod_proxy extension for CONNECT request handling
proxy_ftp - FTP support module for mod_proxy
proxy_http - HTTP support module for mod_proxy
proxy_scgi - SCGI gateway module for mod_proxy

If we wanted to, w/ the new syntax we could fairly easily nest that, 
such that it's apache2_modules@proxy@ajp, 
apache2_modules@proxy@balancer... you get the idea.  Note this isn't 
required- it just becomes sanely possible w/ the syntax.

-- REQUIRED_USE enhancement --

It's late, so I'm going light on the details for this; essentially, 
the ability to address a bunch of flags at once I'm strongly wager 
devs will find a use for.  My brains a bit fried, but if folks have an 
example in this case, please chime in.


Again, sorry for the length, but hopefully having read this far, 
it wasn't a waste of your time. ;)

If folks want to pick at specific sub-uses of the syntax, please feel 
free to change the subject for that feature and derive a thread from 
it (think git patchset uploads, and you'll grok the notion).

~harring