Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring <ferringb@gmail.com>]

On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote:
> On 03/11/2012 06:55 PM, Brian Harring wrote:
> > On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote:
> >> Yeah. Another way of putting it is that the requirement to spawn a bash
> >> process and source the ebuild adds a ridiculous amount of unnecessary
> >> complexity, in violation of the KISS principle [1].
> > 
> > This statement is incorrect.
> > 
> > Even if EAPI could be parsed via some non sourcing approach, we 
> > *still* have to source the ebuild to get the metadata for when the 
> > EAPI is supported (the vast majority of usage).  That complexity is 
> > there one way or another- we wouldn't be trying to extract the EAPI 
> > from the ebuild unless the cache was invalid/missing.
> 
> There are a couple of other cases worth considering:
> 
> 1) User downloads an overlay that doesn't provide cache. We want the
> package manager to give a pretty "EAPI unsupported" message, rather than
> spit out some bash syntax errors.

This criticsm pretty much applies *strictly to the existing 
implementation*.  It's disenguous busting it in this fashion.

EAPI as a function explicitly gives it an out before hitting any of 
that, eliminating your entire critique.  Same goes for parsing it out 
of the ebuild, or renaming the extension.


> 2) You're assuming that a package manager can validate cache for an EAPI
> that it doesn't necessarily support.

Actually, I'm not.

> That's a somewhat fragile
> assumption, given the complexities of cache validation, which involve
> verification all files that affect metadata and those files may vary
> depending on the EAPI.

This is a fair bit of handwavey bullshit.  The same "complexities of 
cache validation" we have to by default deal with for valid/supported 
EAPIs; the case for unsupported EAPIs is actually simpler than 
for supported EAPis, and up until 7ddb7d30, was likely working fine in 
portage.  Tweaking portage to restore that support is pretty easy.

Also note that with the sole exception of g55, every implementation 
has to deal with this specific issue, even g55 isn't fully exempt from 
it since in cleansing a cache, there is the open question of pruning 
cache entries for ebuilds known, but not understood by the local PM.

Either way, the algo is as follows: PM pulls EAPI from the ebuild 
(regardless of method); PM recognizes that it can't handle it- thus 
it stores EAPI:-$EAPI, and the normal cache ebuild checksumming; 
whether it be mtime, md5 for md5-dict, or whatever form new caches 
choose to use.

For attempts to load, the cache subsystem checks that validation 
before trusting the cache entry- this is in place and has been for a 
very, very long time.  This isn't anything new.

So... the entry point is the ebuild; we have checksums for it, and a 
prefixed versions of it's EAPI stored.  The sole deal breaker here is
if we were to decide to allow eclasses to set EAPI (rather than the
current "you're not supposed to, although we don't explicitly block 
it"- which could be tightened to be disallowed at the implementation 
level for EAPI>4).

The following scenarios exist:

1) PM still doesn't support that EAPI, looks at the cache/ebuild: 
checksums are the same, thus the stored EAPI is trustable, leading to 
the PM knowing it still can't process that ebuild and masking it 
appropriately.

2) PM still doesn't support that EAPI, looks, sees that the checksums
no longer match the ebuild.  Re-pulls metadata/EAPI; if the ebuild has 
been changed to a supported EAPI, continues on its way.  If not, 
stores the checksum/negated EAPI, and masks appropriately.

3) PM now supports that EAPI, looks, sees the negated eapi and 
recognizes it as one it knowns, and forces regeneration.

All of those scenarios are there, and easy enough to handle- this 
applies for the existing EAPI implementation additionally.


Now, this isn't to say that someone can't go and try to structure new 
features explicitly to defeat the existing validations.  They could.

The question here is whether or not new features would actually 
*break* that w/out intentionally going out of their way to do so (aka, 
skip the strawman arguments, actual examples required).


> > Phrasing it more bluntly: you can only avoid the sourcing step if you 
> > can isolate that the EAPI is unsupported (which is extremely rare in 
> > terms of actual user experience).  For the rest of the time (well past 
> > the 99% mark) sourcing is the immediate step following.
> 
> For the sake of being robust in all possible cases, it's just a lot
> simpler if we can obtain the EAPI simply and reliably, without spawning
> bash to source the ebuild.

This still is arguable; any performant PM is going to have to do the 
cache dance I mentioned above, or maintain an in memory list of 
ebuilds it knows it can't handle (which if you want to talk robust, 
fails miserably if the API isn't designed for sync invalidation of 
that global list).  Complexity, oh my.

That's the kicker from where I'm sitting; you're so focused on 
avoiding bash that you ignore the complexities surrounding it that are 
unavoidable regardless.

The EAPI function route resolve this issue w/out penalizing the normal 
(aka supported) pathways, and does so at the same robustness as 
everything sans g55 (which aims to allow us to use things other 
than bash such as the mocked xml ebuild format).  Frankly, the 
function route gives us a bit more robustness- rather than allowing 
the scenario of a PM that doesn't parse seeing a version that requires 
it, via the function route doing `eapi 5 || die` has no such failure 
mode.


> > Also, stop referencing wikipedia.  People know what "trivial 
> > objection" and "KISS" is.
> 
> You can't assume that. On this list we've got potential to have readers
> and responders with lots of different backgrounds.

You might want to include links to Linux, gentoo, and bash while 
you're at it.

Ya know, just in case.


> Your insistence on
> using bash to obtain the EAPI would make me wonder if you understood the
> KISS principle, if I didn't know you better.

Zing.

What I'd like to see, is accuracy in this discussion.  Skip the 
handwavey "complexity! complexity! complexity!" crap, same for 
selective robustness definitions.  Past attempts at this discussion 
mostly failed due to people pulling crap like this and frankly it just 
pisses people off. 

Like it or not, each solution has technical pros/cons, more 
importantly, it has an impact on developers meaning what they *like* 
carries weight (rather than what you want them to do).

Shebang/parsing EAPI w/out bash, EAPI as a function, all are reliant 
on the same cache mechanism.  That's fact.  G55 is the sole exception.

The question of what mechanism to use is dependant on what people 
actually want, and how far we're willing to stretch the ebuild format 
(versus defining a new one rather than new versions of it).

For the lifespan of gentoo thus far, our format has been bash; trying 
to shift away from bash is fine, but as long as it *is* bash we should
be designing w/ that flow, including what is least likely to have a 
developer screw up.  Pragmatic reality, especially since we're still 
stuck w/ compatibility issues getting to the new mechanism.


Frankly I'm starting to think I should just write up a table of 
pros/cons, extending what Ulm has started; in particular, including 
accurate statement of the transition costs of each (something people 
seem to be ignoring- compatibility mechanisms are *not* cheap to get 
in place).

~harring