TLDR: rant in support of overlays (was: Re: [gentoo-dev] Over-reliance of Gentoo projects on overlays)

TLDR: rant in support of overlays (was: Re: [gentoo-dev] Over-reliance of Gentoo projects on overlays)

[Greg Turner]

On Wed, Jun 12, 2013 at 10:23 AM, Michael Orlitzky <michael@orlitzky.com> wrote:
> On 06/12/2013 01:13 PM, Ciaran McCreesh wrote:
>> On Wed, 12 Jun 2013 19:05:29 +0200 hasufell <hasufell@gentoo.org>
>> wrote:
>>>> Isn't it more an indication that Gentoo needs better package
>>>> management support for overlays?
>>
>>> No.
>>
>
> We need worse support for overlays, i.e. no. Having to use >3 overlays
> defeats the purpose of a QA'd tree.

Disclaimer: I'm not a Gentoo developer -- actually, fuck that noise, I
am a Gentoo developer.  But you know what I mean.  My knowledge of the
gentoo QA process is limited to what I've been able to glean as a
beneficiary of the work and a limited participant via the bugzilla
system.  The precise mechanisms and policies that drive Gentoo QA are
actually fairly opaque to me.

Anyhow... wrt overlays "defeating the purpose" of QA: overlays may or
may not prevent the QA process from pertaining to users of overlays,
depending on the nature of the overlays.  But, in fact, far from
defeating the purpose and integrity of Gentoo QA, my belief is that by
providing a standard baseline that QA may rely upon, overlays serve to
enhance and protect Gentoo's quality.

consider: emerge --info provides the overlays in bug reports to gx86
package maintainers and, if there is doubt about the sanity of the
overlays, maintainers are (I presume) free not to support nonstandard
configurations.  But if a bug-reporter has this problem, the overlay
system actually protects them.  If they feel they are left
high-and-dry due to their nonstandard gentoo installation, and are
sure that their bug is a legitimate gx86 bug, they are free to whip up
a virtual machine or to temporarily drop their overlays and CFLAG rice
and emerge --depclean && emerge -e system.

Assuming they turn out to be right, bug reporters and package
maintainers can be sure to be roughly on the same page wrt
reproducibility.  Indeed, no matter what kind of personality conflicts
or other nontechnical issues may be at play, the reporter of a
legitimate bug is pretty much guaranteed to get some kind of
resolution to his issue, or at least that has been my experience.  If
bug reporters don't like those results and want to implement a
different solution than the one they got, overlays enable them to do
that as well.

In short, overlays permit Gentoo to maintain reasonable quality
standards while encouraging innovation and casual experimentation.
Larry the Cow approves of them.

> Everything in an (official)
> overlay should be in package.mask instead. The main reason it isn't is
> because nobody wants to use CVS. For good examples, see sunrise or
> gentoo-haskell.

Such a policy might be OK for developers who are able to just hop on
and make changes to gx86 without going though the whole bugzilla
process, hence "official".

However, it seems like you're thinking of overlays as piles of package
ebuilds which haven't yet made it into stable.  They may be that, or
they may not -- overlays can add profiles, modify core eclasses, or
even replace portage itself with a customized variant, and who knows
what else.  As another poster pointed out sarcastically, the GSOC
types of projects clearly don't comport well with this, even if
certain things like, i.e., sunrise, arguably might.

Anyhow, isn't the gentoo-x86 tree already plenty big enough, without
every single overlay's ebuilds and eclasses in there too?  Personally,
I'm inclined to wish it was smaller, even if that meant more stuff was
pushed into overlays, although I suppose that might have a negative
impact on QA coverage without some corresponding changes on the QA end
of things... I guess I don't know enough about it to speak
confidently.

As a huge consumer of the overlay and layman mechanisms, both as a
user and a developer, there is absolutely no doubt in my mind that by
far the gravest problem with the overlay architecture is its inability
to create direct VCS connectivity between an overlay and its upstream
PORTDIR (coupled with it's requirement to clone entire package
directories instead of overriding them on a per-file basis).  FWIW, I
have nascent ideas about how to fix that, but they are quite radical,
probably half-baked (even just as vaporware ideas), and arguably
off-topic, so I won't elaborate.

-gmt

Re: TLDR: rant in support of overlays (was: Re: [gentoo-dev] Over-reliance of Gentoo projects on overlays)

[Michael Orlitzky]

On 06/12/2013 06:31 PM, Greg Turner wrote:
> On Wed, Jun 12, 2013 at 10:23 AM, Michael Orlitzky <michael@orlitzky.com> wrote:
>> On 06/12/2013 01:13 PM, Ciaran McCreesh wrote:
>>> On Wed, 12 Jun 2013 19:05:29 +0200 hasufell <hasufell@gentoo.org>
>>> wrote:
>>>>> Isn't it more an indication that Gentoo needs better package
>>>>> management support for overlays?
>>>
>>>> No.
>>>
>>
>> We need worse support for overlays, i.e. no. Having to use >3 overlays
>> defeats the purpose of a QA'd tree.
> 
> Disclaimer: I'm not a Gentoo developer -- actually, fuck that noise, I
> am a Gentoo developer.  But you know what I mean.  My knowledge of the
> gentoo QA process is limited to what I've been able to glean as a
> beneficiary of the work and a limited participant via the bugzilla
> system.  The precise mechanisms and policies that drive Gentoo QA are
> actually fairly opaque to me.
> 
> Anyhow... wrt overlays "defeating the purpose" of QA: overlays may or
> may not prevent the QA process from pertaining to users of overlays,
> depending on the nature of the overlays.  But, in fact, far from
> defeating the purpose and integrity of Gentoo QA, my belief is that by
> providing a standard baseline that QA may rely upon, overlays serve to
> enhance and protect Gentoo's quality.

E.g. https://bugs.gentoo.org/show_bug.cgi?id=341807

Overlays aren't tested against one another. As you add more overlays,
the probability of brokenness approaches unity.

You're not allowed to commit broken shit to the tree, so adding your
packages to gx86 implies that it works with the rest of the packages in
gx86.

Re: TLDR: rant in support of overlays (was: Re: [gentoo-dev] Over-reliance of Gentoo projects on overlays)

[yac]

On Wed, 12 Jun 2013 15:31:57 -0700
Greg Turner <gmt@malth.us> wrote:

> Anyhow, isn't the gentoo-x86 tree already plenty big enough, without
> every single overlay's ebuilds and eclasses in there too?  Personally,
> I'm inclined to wish it was smaller, even if that meant more stuff was
> pushed into overlays

Actually, this is something I expected to happen soon after we got
overlays but for some reason it haven't. I imagine we would
not have a single gx86 official tree but rather a bunch of official
overlays. For basic installation one would need just the system
overlay. Then everypony could add official overlay for KDE, or gnome or
whatnot as one desires.

I haven't thought this through in any way but it feels like better
design.

[gentoo-dev] Re: TLDR: rant in support of overlays (was: Re: Over-reliance of Gentoo projects on overlays)

[Duncan]

yac posted on Thu, 13 Jun 2013 12:15:26 +0200 as excerpted:

> On Wed, 12 Jun 2013 15:31:57 -0700 Greg Turner <gmt@malth.us> wrote:
> 
>> Anyhow, isn't the gentoo-x86 tree already plenty big enough, without
>> every single overlay's ebuilds and eclasses in there too?  Personally,
>> I'm inclined to wish it was smaller, even if that meant more stuff was
>> pushed into overlays
> 
> Actually, this is something I expected to happen soon after we got
> overlays but for some reason it haven't. I imagine we would not have a
> single gx86 official tree but rather a bunch of official overlays. For
> basic installation one would need just the system overlay. Then
> everypony could add official overlay for KDE, or gnome or whatnot as one
> desires.
> 
> I haven't thought this through in any way but it feels like better
> design.

Someone else already mentioned the problem with that.  At least 
currently, only the official tree is tested against, so at least in 
theory it's quite easy to have conflicts between overlays, and it's 
certainly much more likely to have packages broken for some usage as they 
simply haven't been tested against packages in that overlay.

The more overlays, the more likely the conflicts and breakage.

The obvious way around that is to have a set of "blessed" overlays that 
get tested against, much like the main tree (only) today.  However, that 
seriously complexifies (good) testing as now every dev has to pull down 
the whole set of "blessed" overlays instead of just the main tree plus 
whatever overlays he happens to work on (with some devs doing no overlays 
at all), as is the case now.

The last thing we should be doing is throwing additional roadblocks into 
the way of reasonable testing, and I believe that's why the split you 
expected hasn't happened -- people realize that and decide the main 
tree's the best idea after all.

Tho CVS is a enough of a pain that I'm sure that alone keeps some 
packages and potential devs away.  Once it's git, that problem too will 
disappear, and there will be less pressure to split off overlays than 
there is now.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman