TLDR: rant in support of overlays (was: Re: [gentoo-dev] Over-reliance of Gentoo projects on overlays)

[Greg Turner <gmt@malth.us>]

On Wed, Jun 12, 2013 at 10:23 AM, Michael Orlitzky <michael@orlitzky.com> wrote:
> On 06/12/2013 01:13 PM, Ciaran McCreesh wrote:
>> On Wed, 12 Jun 2013 19:05:29 +0200 hasufell <hasufell@gentoo.org>
>> wrote:
>>>> Isn't it more an indication that Gentoo needs better package
>>>> management support for overlays?
>>
>>> No.
>>
>
> We need worse support for overlays, i.e. no. Having to use >3 overlays
> defeats the purpose of a QA'd tree.

Disclaimer: I'm not a Gentoo developer -- actually, fuck that noise, I
am a Gentoo developer.  But you know what I mean.  My knowledge of the
gentoo QA process is limited to what I've been able to glean as a
beneficiary of the work and a limited participant via the bugzilla
system.  The precise mechanisms and policies that drive Gentoo QA are
actually fairly opaque to me.

Anyhow... wrt overlays "defeating the purpose" of QA: overlays may or
may not prevent the QA process from pertaining to users of overlays,
depending on the nature of the overlays.  But, in fact, far from
defeating the purpose and integrity of Gentoo QA, my belief is that by
providing a standard baseline that QA may rely upon, overlays serve to
enhance and protect Gentoo's quality.

consider: emerge --info provides the overlays in bug reports to gx86
package maintainers and, if there is doubt about the sanity of the
overlays, maintainers are (I presume) free not to support nonstandard
configurations.  But if a bug-reporter has this problem, the overlay
system actually protects them.  If they feel they are left
high-and-dry due to their nonstandard gentoo installation, and are
sure that their bug is a legitimate gx86 bug, they are free to whip up
a virtual machine or to temporarily drop their overlays and CFLAG rice
and emerge --depclean && emerge -e system.

Assuming they turn out to be right, bug reporters and package
maintainers can be sure to be roughly on the same page wrt
reproducibility.  Indeed, no matter what kind of personality conflicts
or other nontechnical issues may be at play, the reporter of a
legitimate bug is pretty much guaranteed to get some kind of
resolution to his issue, or at least that has been my experience.  If
bug reporters don't like those results and want to implement a
different solution than the one they got, overlays enable them to do
that as well.

In short, overlays permit Gentoo to maintain reasonable quality
standards while encouraging innovation and casual experimentation.
Larry the Cow approves of them.

> Everything in an (official)
> overlay should be in package.mask instead. The main reason it isn't is
> because nobody wants to use CVS. For good examples, see sunrise or
> gentoo-haskell.

Such a policy might be OK for developers who are able to just hop on
and make changes to gx86 without going though the whole bugzilla
process, hence "official".

However, it seems like you're thinking of overlays as piles of package
ebuilds which haven't yet made it into stable.  They may be that, or
they may not -- overlays can add profiles, modify core eclasses, or
even replace portage itself with a customized variant, and who knows
what else.  As another poster pointed out sarcastically, the GSOC
types of projects clearly don't comport well with this, even if
certain things like, i.e., sunrise, arguably might.

Anyhow, isn't the gentoo-x86 tree already plenty big enough, without
every single overlay's ebuilds and eclasses in there too?  Personally,
I'm inclined to wish it was smaller, even if that meant more stuff was
pushed into overlays, although I suppose that might have a negative
impact on QA coverage without some corresponding changes on the QA end
of things... I guess I don't know enough about it to speak
confidently.

As a huge consumer of the overlay and layman mechanisms, both as a
user and a developer, there is absolutely no doubt in my mind that by
far the gravest problem with the overlay architecture is its inability
to create direct VCS connectivity between an overlay and its upstream
PORTDIR (coupled with it's requirement to clone entire package
directories instead of overriding them on a per-file basis).  FWIW, I
have nascent ideas about how to fix that, but they are quite radical,
probably half-baked (even just as vaporware ideas), and arguably
off-topic, so I won't elaborate.

-gmt