[gentoo-dev] the graveyard overlay

[gentoo-dev] the graveyard overlay

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

I'm starting a new thread so this will be a completely separate
discussion.

On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
> On Fri, 8 Jul 2016 10:42:14 -0400 Rich Freeman wrote:
> > On Fri, Jul 8, 2016 at 10:30 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
> > >
> > > Also there's some debate in IRC about whether or not these packages
> > > should be lastrited or dropped to maintainer-needed.  These forks are
> > > not in good shape upstream, so I think it makes better sense to
> > > p.mask/lastrite and then move them to the graveyard overlay when I
> > > remove them from the tree in 30 days.
> > >
> > 
> > IMO the criteria should be whether they work or not.  Not whether
> > upstream is more or less active.
> > 
> > If they're blockers on other work, by all means cull them.  However,
> > if the biggest problem with them is that they're using a few inodes in
> > the repo, then they should probably stay.
>  
> +1
> 
> Best regards,
> Andrew Savchenko

There is also an overlay for packages that are removed from the official
tree [1], and imo that is where old software should go if it doesn't
have an active maintainer.

I don't know why we haven't been using this, but using it more than we
have makes a lot of sense.

William

[1] https://github.com/gentoo/graveyard

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

On Fri, 8 Jul 2016 10:11:45 -0500 William Hubbs wrote:
> I'm starting a new thread so this will be a completely separate
> discussion.
> 
> On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
> > On Fri, 8 Jul 2016 10:42:14 -0400 Rich Freeman wrote:
> > > On Fri, Jul 8, 2016 at 10:30 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
> > > >
> > > > Also there's some debate in IRC about whether or not these packages
> > > > should be lastrited or dropped to maintainer-needed.  These forks are
> > > > not in good shape upstream, so I think it makes better sense to
> > > > p.mask/lastrite and then move them to the graveyard overlay when I
> > > > remove them from the tree in 30 days.
> > > >
> > > 
> > > IMO the criteria should be whether they work or not.  Not whether
> > > upstream is more or less active.
> > > 
> > > If they're blockers on other work, by all means cull them.  However,
> > > if the biggest problem with them is that they're using a few inodes in
> > > the repo, then they should probably stay.
> >  
> > +1
> > 
> > Best regards,
> > Andrew Savchenko
> 
> There is also an overlay for packages that are removed from the official
> tree [1], and imo that is where old software should go if it doesn't
> have an active maintainer.
> 
> I don't know why we haven't been using this, but using it more than we
> have makes a lot of sense.

When software is in the main tree, it is a subject of tree-wide
changes like GLEP 67 update, package moves and so on. In a
separated overlay it will be completely abandoned and it may create
inter-overlay dependencies issues (e.g. when A is an old
package from the tree and package B from some overlay depends on A,
so if A will move to graveyard, B will be broken).

I completely do not understand why having "old" software in tree is
a problem, if such software have no serious issues and is not
blocking major progress. If software _is_ sufficiently broken, then
indeed move it to graveyard.

As I said yesterday on IRC, one of the greatest virtues of Gentoo
is its ample spectra of packages available in the main tree. I do
not understand why it should be killed for nothing.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: the graveyard overlay

[»Q«]

On Fri, 8 Jul 2016 10:11:45 -0500
William Hubbs <williamh@gentoo.org> wrote:

> There is also an overlay for packages that are removed from the
> official tree [1], and imo that is where old software should go if it
> doesn't have an active maintainer.
> 
> I don't know why we haven't been using this, but using it more than we
> have makes a lot of sense.

Completely aside from the question of criteria for removing stuff from
the main tree, it would make a lot of users happy if every package
which *is* removed were added to the graveyard overlay.

Re: [gentoo-dev] Re: the graveyard overlay

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

On Fri, 8 Jul 2016 10:55:49 -0500 »Q« wrote:
> On Fri, 8 Jul 2016 10:11:45 -0500
> William Hubbs <williamh@gentoo.org> wrote:
> 
> > There is also an overlay for packages that are removed from the
> > official tree [1], and imo that is where old software should go if it
> > doesn't have an active maintainer.
> > 
> > I don't know why we haven't been using this, but using it more than we
> > have makes a lot of sense.
> 
> Completely aside from the question of criteria for removing stuff from
> the main tree, it would make a lot of users happy if every package
> which *is* removed were added to the graveyard overlay.

Good idea, since we have attic no more, and users need this [1].
While any removed package can be extracted from git, this is not
a trivial operation and is definitely harder then fetching filesv
from overlay.

But the problem is that this way overlay will become completely
broken in terms of both QA and security.

[1] https://archives.gentoo.org/gentoo-dev/message/660e7f65e1b4d3f932aacfb69e273e3b

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4690 bytes --]

On Fri, 8 Jul 2016 18:33:35 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Fri, 8 Jul 2016 10:11:45 -0500 William Hubbs wrote:
> > I'm starting a new thread so this will be a completely separate
> > discussion.
> > 
> > On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:  
> > > On Fri, 8 Jul 2016 10:42:14 -0400 Rich Freeman wrote:  
> > > > On Fri, Jul 8, 2016 at 10:30 AM, Anthony G. Basile <blueness@gentoo.org> wrote:  
> > > > >
> > > > > Also there's some debate in IRC about whether or not these packages
> > > > > should be lastrited or dropped to maintainer-needed.  These forks are
> > > > > not in good shape upstream, so I think it makes better sense to
> > > > > p.mask/lastrite and then move them to the graveyard overlay when I
> > > > > remove them from the tree in 30 days.
> > > > >  
> > > > 
> > > > IMO the criteria should be whether they work or not.  Not whether
> > > > upstream is more or less active.
> > > > 
> > > > If they're blockers on other work, by all means cull them.  However,
> > > > if the biggest problem with them is that they're using a few inodes in
> > > > the repo, then they should probably stay.  
> > >  
> > > +1
> > > 
> > > Best regards,
> > > Andrew Savchenko  
> > 
> > There is also an overlay for packages that are removed from the official
> > tree [1], and imo that is where old software should go if it doesn't
> > have an active maintainer.
> > 
> > I don't know why we haven't been using this, but using it more than we
> > have makes a lot of sense.  
> 
> When software is in the main tree, it is a subject of tree-wide
> changes like GLEP 67 update, package moves and so on. In a
> separated overlay it will be completely abandoned and it may create
> inter-overlay dependencies issues (e.g. when A is an old
> package from the tree and package B from some overlay depends on A,
> so if A will move to graveyard, B will be broken).

And that is the exact problem. As long as it's in Gentoo, it creates
work for others. Others who have enough work already without your help,
thank you.

There is a major difference between doing a global changes involving
few dozen or hundred packages when they are maintained, and having few
dozen more unmaintained packages to care for. The changes already
require a lot of effort, you know.

Now, there's a significant difference between lastriting unmaintained
packages at treecleaner's leisure and having a clean tree to work on,
and having to figure out how many of the packages blocking some global
change are unmaintained and if they can be cleaned, and cleaning them
while doing something completely different, then checking again,
then...

> I completely do not understand why having "old" software in tree is
> a problem, if such software have no serious issues and is not
> blocking major progress. If software _is_ sufficiently broken, then
> indeed move it to graveyard.

Right now, we have over 1500 unmaintained packages. Please tell me, how
that speaks of Gentoo? Because as far as I can see, we have 1500
packages which nobody looks after unless forced to.

You say that there are no bugs in those packages. How do you know? You
don't know unless you test it, and no maintainer means nobody is known
to test it regularly. The package can be pretty much completely broken
and we'll not know unless someone tests it.

Now, as long as package is in ::gentoo it is somehow officially
supported. Which means it pops up for user when he is looking for
something, and he assumes it's going to solve his problem. This is good
if it works. But considering it's unmaintained, nobody's testing it.

So the package might work, or it might not. It might have major
problems but nobody may notice them before user learns about them
the hard way. If he reports a bug, the bug goes to /dev/null, unless
some developer accidentally notices it and decides to fix it. Or just
lastrite the package.

Let me summarize the user experience. User was looking for a good tool.
Instead, he found a well-advertised unmaintained old piece of software
that promised to solve his problems and instead created more problems
for him. Nevertheless, he decided to be a good user and reported
the bug. Then he learned that nobody cares to fix the bug, the package
is long forgotten, and all his effort was for nothing.

Then he has to look for an alternative... and he starts to wonder how
to filter out those unmaintained packages because he'd rather use
something that somebody has really tested in the last 5 years.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Rich Freeman]

On Fri, Jul 8, 2016 at 12:51 PM, Michał Górny <mgorny@gentoo.org> wrote:
>
> Now, there's a significant difference between lastriting unmaintained
> packages at treecleaner's leisure and having a clean tree to work on,
> and having to figure out how many of the packages blocking some global
> change are unmaintained and if they can be cleaned, and cleaning them
> while doing something completely different, then checking again,
> then...

Packages that are unmaintained are tagged as such.  Perhaps we need
better reporting tools to make it easier to identify blockers
associated with these?

Perhaps some kind of QA tool that lists all unmaintained packages that
are a blocker for anything and auto-treecleans them?

>
> You say that there are no bugs in those packages. How do you know? You
> don't know unless you test it, and no maintainer means nobody is known
> to test it regularly. The package can be pretty much completely broken
> and we'll not know unless someone tests it.
>

This sounds like a tree falling in the forest with nobody around...

If a package is in the tree, and it has no known bugs, and no users,
who cares?

If somebody tries to use it, and it doesn't work, then they can file a
bug, and then we can treeclean it.

I tend to favor just leaving things alone in the tree unless there is
a serious bug, or it is something sensitive enough that it just isn't
worth taking the chance.

However, I think broadly speaking there are two approaches that I
think make some sort of sense:
1.  Packages stay in tree until there is a problem (serious bug,
blocker, etc), then rapid and aggressive cleanout.
2.  Only maintained packages in tree.  Have a graveyard repo for
unmaintained stuff that doesn't have serious bugs (including not
working due to some change introduced in the main tree), and
aggressive cleanout if these criteria fail.

I don't think that keeping stuff in the tree until it is broken and
then moving them to a graveyard makes sense, at least not if they're
seriously broken.  The only use case I really see for this is people
who don't know how to work git.  I think a better solution for that
use case is for somebody to just make a repo that contains the latest
version of every file that has ever been in the gentoo repo (so,
basically a replay of the scm history minus delete operations, except
maybe for moves).  That really should just be used as a convenience
and not as an actual overlay.

Having a graveyard that ONLY contains broken stuff as an overlay just
doesn't make sense to me.  Why would you install packages directly
from it without fixing them first?  Certainly for build failures you'd
be forced to do this.  I guess for security issues you could decide
that you don't care, or that your host is in a locked room with no
network access or something.  However, these seem like such minor use
cases that somebody could just stick the ebuilds in their own overlay
if they needed them.

-- 
Rich

Re: [gentoo-dev] the graveyard overlay

[james]

On 07/08/2016 10:33 AM, Andrew Savchenko wrote:
> On Fri, 8 Jul 2016 10:11:45 -0500 William Hubbs wrote:
>> I'm starting a new thread so this will be a completely separate
>> discussion.
>>
>> On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>>> On Fri, 8 Jul 2016 10:42:14 -0400 Rich Freeman wrote:
>>>> On Fri, Jul 8, 2016 at 10:30 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
>>>>>
>>>>> Also there's some debate in IRC about whether or not these packages
>>>>> should be lastrited or dropped to maintainer-needed.  These forks are
>>>>> not in good shape upstream, so I think it makes better sense to
>>>>> p.mask/lastrite and then move them to the graveyard overlay when I
>>>>> remove them from the tree in 30 days.
>>>>>
>>>>
>>>> IMO the criteria should be whether they work or not.  Not whether
>>>> upstream is more or less active.
>>>>
>>>> If they're blockers on other work, by all means cull them.  However,
>>>> if the biggest problem with them is that they're using a few inodes in
>>>> the repo, then they should probably stay.
>>>
>>> +1
>>>
>>> Best regards,
>>> Andrew Savchenko
>>
>> There is also an overlay for packages that are removed from the official
>> tree [1], and imo that is where old software should go if it doesn't
>> have an active maintainer.

I disagree with the criteria. Old is not necessarily sufficiently 
broken. Granted every package should have a maintainer, but the lack 
thereof is not sufficient for removal.

>>
>> I don't know why we haven't been using this, but using it more than we
>> have makes a lot of sense.

NP-H. just published a basic guide on how to resurrect  packages from 
github. Perhaps that should be tested by both devs and users, and 
modified and further documented to be compatible with 'the graveyard 
overlay' and those instructions linked to the graveyard.

In the future, the graveyard will itself look like the gentoo tree, 
except for the actual package contained therein, including  categories 
and files ? Patches? Sources ? How is the graveyard going to look and 
function for retrieval a year or two from now? Bare-bones copies of 
ebuilds only? Just like the attic, this 'graveyard' will be visited
often for package to be returned at least to a user's 
/usr/local/portage. So why not implement the full set of tooling for 
such needs
into the graveyard concept ?


> When software is in the main tree, it is a subject of tree-wide
> changes like GLEP 67 update, package moves and so on. In a
> separated overlay it will be completely abandoned and it may create
> inter-overlay dependencies issues (e.g. when A is an old
> package from the tree and package B from some overlay depends on A,
> so if A will move to graveyard, B will be broken).

> I completely do not understand why having "old" software in tree is
> a problem, if such software have no serious issues and is not
> blocking major progress. If software _is_ sufficiently broken, then
> indeed move it to graveyard.

> As I said yesterday on IRC, one of the greatest virtues of Gentoo
> is its ample spectra of packages available in the main tree. I do
> not understand why it should be killed for nothing.

+1

Clearly we have significant differences in the dev and user communities 
on viability and usefulness of old packages. Perhaps the solution is to 
be conservative on tree removal, but aggressive on concurrent status via 
labeling of packages. For example, all of these issues  could be added 
to a simple tool like app-portage/eix to show, for example security, age 
  of last update and maintainer status, and any other issues deemed 
significant.


> Best regards,
> Andrew Savchenko

hth,
James

Re: [gentoo-dev] the graveyard overlay

[james]

On 07/08/2016 11:51 AM, Michał Górny wrote:
> On Fri, 8 Jul 2016 18:33:35 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
>
>> On Fri, 8 Jul 2016 10:11:45 -0500 William Hubbs wrote:
>>> I'm starting a new thread so this will be a completely separate
>>> discussion.
>>>
>>> On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>>>> On Fri, 8 Jul 2016 10:42:14 -0400 Rich Freeman wrote:
>>>>> On Fri, Jul 8, 2016 at 10:30 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
>>>>>>
>>>>>> Also there's some debate in IRC about whether or not these packages
>>>>>> should be lastrited or dropped to maintainer-needed.  These forks are
>>>>>> not in good shape upstream, so I think it makes better sense to
>>>>>> p.mask/lastrite and then move them to the graveyard overlay when I
>>>>>> remove them from the tree in 30 days.
>>>>>>
>>>>>
>>>>> IMO the criteria should be whether they work or not.  Not whether
>>>>> upstream is more or less active.
>>>>>
>>>>> If they're blockers on other work, by all means cull them.  However,
>>>>> if the biggest problem with them is that they're using a few inodes in
>>>>> the repo, then they should probably stay.
>>>>
>>>> +1
>>>>
>>>> Best regards,
>>>> Andrew Savchenko
>>>
>>> There is also an overlay for packages that are removed from the official
>>> tree [1], and imo that is where old software should go if it doesn't
>>> have an active maintainer.
>>>
>>> I don't know why we haven't been using this, but using it more than we
>>> have makes a lot of sense.
>>
>> When software is in the main tree, it is a subject of tree-wide
>> changes like GLEP 67 update, package moves and so on. In a
>> separated overlay it will be completely abandoned and it may create
>> inter-overlay dependencies issues (e.g. when A is an old
>> package from the tree and package B from some overlay depends on A,
>> so if A will move to graveyard, B will be broken).
>
> And that is the exact problem. As long as it's in Gentoo, it creates
> work for others. Others who have enough work already without your help,
> thank you.
>
> There is a major difference between doing a global changes involving
> few dozen or hundred packages when they are maintained, and having few
> dozen more unmaintained packages to care for. The changes already
> require a lot of effort, you know.
>
> Now, there's a significant difference between lastriting unmaintained
> packages at treecleaner's leisure and having a clean tree to work on,
> and having to figure out how many of the packages blocking some global
> change are unmaintained and if they can be cleaned, and cleaning them
> while doing something completely different, then checking again,
> then...
>
>> I completely do not understand why having "old" software in tree is
>> a problem, if such software have no serious issues and is not
>> blocking major progress. If software _is_ sufficiently broken, then
>> indeed move it to graveyard.
>
> Right now, we have over 1500 unmaintained packages. Please tell me, how
> that speaks of Gentoo? Because as far as I can see, we have 1500
> packages which nobody looks after unless forced to.
>
> You say that there are no bugs in those packages. How do you know? You
> don't know unless you test it, and no maintainer means nobody is known
> to test it regularly. The package can be pretty much completely broken
> and we'll not know unless someone tests it.
>
> Now, as long as package is in ::gentoo it is somehow officially
> supported. Which means it pops up for user when he is looking for
> something, and he assumes it's going to solve his problem. This is good
> if it works. But considering it's unmaintained, nobody's testing it.
>
> So the package might work, or it might not. It might have major
> problems but nobody may notice them before user learns about them
> the hard way. If he reports a bug, the bug goes to /dev/null, unless
> some developer accidentally notices it and decides to fix it. Or just
> lastrite the package.

More often a given package does not work for a noob-to-gentoo but and 
old hack can make it work. I've seen tons of evidence for this on 
gentoo-user over the last 12 years....

>
> Let me summarize the user experience. User was looking for a good tool.
> Instead, he found a well-advertised unmaintained old piece of software
> that promised to solve his problems and instead created more problems
> for him. Nevertheless, he decided to be a good user and reported
> the bug. Then he learned that nobody cares to fix the bug, the package
> is long forgotten, and all his effort was for nothing.

 From a tree-cleaner prospective, you might be correct. As a gentoo-user,
and one that responses to many of the ML queries, there is a full 
spectrum of experiences you are not accurately characterizing, imho.

>
> Then he has to look for an alternative... and he starts to wonder how
> to filter out those unmaintained packages because he'd rather use
> something that somebody has really tested in the last 5 years.
>

Don't stop there. Users soon learn about overlays and a wide variety of 
packages contained in those, as well as sabayon, calculate, coreos and 
surely more to come (flatpak?). After a while you learn (after some 
pain) which overlays  and ebuild sources you can trust and use software 
from. Just look at the tragic status of 'java' on gentoo, despite the 
heroic efforts of many devs and users. Granted much pain with java is 
directly attributed to Oracle, but many of us need java. Personally, I 
add overlays (now about 10 currently) from devs I follow && trust. The 
current gentoo tooling does not work for discovery details on overlays 
run by devs. Example try 'equery m <package>' for an overlay you are not 
familiar with.

So you are correct in what you have detailed above. Still, I do not see 
that as reason to take a chainsaw to the gentoo portage tree. Be 
aggressive on new tools development/integration to solve these problems 
and not aggressive on removing packages from the tree. Those removals 
should be on a conservative basis. Label the packages with these 
non-critical deficiencies but not removal, unless severely broken
or serious security issues exist.

I for one run lots of (security) broken codes on closed networks with 
vendor products to test. You think our gentoo tree has issues? You 
should start probing and popping industrial products, some on 2.2 linux 
kernels and insufficient hardware resources. Old codes on gentoo match 
up very will with many currently selling products, just so you know.

I like old codes, that is for sure, and the graying of codes does not 
mean they are useless. Far from it. ymmv.


hth,
James

Re: [gentoo-dev] the graveyard overlay

[Chí-Thanh Christopher Nguyễn]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

Rich Freeman schrieb:
>> You say that there are no bugs in those packages. How do you know? You
>> don't know unless you test it, and no maintainer means nobody is known
>> to test it regularly. The package can be pretty much completely broken
>> and we'll not know unless someone tests it.
>>
>
> This sounds like a tree falling in the forest with nobody around...
>
> If a package is in the tree, and it has no known bugs, and no users,
> who cares?
>
> If somebody tries to use it, and it doesn't work, then they can file a
> bug, and then we can treeclean it.

One might add here that toralf is doing a great job at building all packages 
and reporting those that fail. So at least we see build failures.

> Having a graveyard that ONLY contains broken stuff as an overlay just
> doesn't make sense to me.  Why would you install packages directly
> from it without fixing them first?  Certainly for build failures you'd
> be forced to do this.  I guess for security issues you could decide
> that you don't care, or that your host is in a locked room with no
> network access or something.  However, these seem like such minor use
> cases that somebody could just stick the ebuilds in their own overlay
> if they needed them.

I think the point of a graveyard repository is that discovering and 
extracting deleted ebuilds from git is more cumbersome than from CVS attic.

It would be even better if the graveyard repository preserved the commit 
history, but I don't see any easy solution for that.


Best regards,
Chí-Thanh Christopher Nguyễn



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: the graveyard overlay

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

On Fri, Jul 08, 2016 at 07:17:44PM +0300, Andrew Savchenko wrote:

*snip*

> But the problem is that this way overlay will become completely
> broken in terms of both QA and security.

Once it is in an overlay, we don't care about qa or security any longer,
so this isn't a problem just a fact of dealing with an overlay.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Rich Freeman]

On Fri, Jul 8, 2016 at 2:22 PM, Chí-Thanh Christopher Nguyễn
<chithanh@gentoo.org> wrote:
> I think the point of a graveyard repository is that discovering and
> extracting deleted ebuilds from git is more cumbersome than from CVS attic.
>
> It would be even better if the graveyard repository preserved the commit
> history, but I don't see any easy solution for that.
>

Like I said.  If the only use case is helping people who don't know
how to use git find deleted ebuilds, then just create a directory tree
with everything that was ever in the Gentoo repo.  That would be
pretty easy to script.  QA doesn't need to have anything to do with
it.

-- 
Rich

Re: [gentoo-dev] the graveyard overlay

[Chí-Thanh Christopher Nguyễn]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

Rich Freeman schrieb:
> On Fri, Jul 8, 2016 at 2:22 PM, Chí-Thanh Christopher Nguyễn
> <chithanh@gentoo.org> wrote:
>> I think the point of a graveyard repository is that discovering and
>> extracting deleted ebuilds from git is more cumbersome than from CVS attic.
>>
>> It would be even better if the graveyard repository preserved the commit
>> history, but I don't see any easy solution for that.
>>
>
> Like I said.  If the only use case is helping people who don't know
> how to use git find deleted ebuilds, then just create a directory tree
> with everything that was ever in the Gentoo repo.  That would be
> pretty easy to script.  QA doesn't need to have anything to do with
> it.

I'm sorry for harping on that topic again, but if we had used grobian's 
initial proposal for git migration[0] - one repository per package, and the 
portage tree would be an aggregation of those - then we could have such a 
thing basically for free now.

But that's how it is now. Getting ebuilds from CVS attic could be done via 
the sources.g.o web interface even, no local checkout needed.


Best regards,
Chí-Thanh Christopher Nguyễn

[0] 
https://archives.gentoo.org/gentoo-dev/message/753620a99ab88b9525a253590617db3c


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Rich Freeman]

On Fri, Jul 8, 2016 at 2:51 PM, Chí-Thanh Christopher Nguyễn
<chithanh@gentoo.org> wrote:
>
> I'm sorry for harping on that topic again, but if we had used grobian's
> initial proposal for git migration[0] - one repository per package, and the
> portage tree would be an aggregation of those - then we could have such a
> thing basically for free now.

Not really.  Unless you planned to never delete old versions of
packages from the tree.  Also, if you remove a package from the tree
you'd need to ensure that its repository didn't later disappear, or
that people could actually find it.

That design also has other problems, like a lack of consistency across
the tree.  If one package syncs and another one doesn't, maybe you get
things that don't work.  And heaven help the guy trying to do a
tree-wide change.  Just as with cvs there would be no association with
a change in one package with a change in another.

It wasn't a bad idea, but in the end the pros didn't seem worth the
cons.  I definitely wouldn't do it just so that I didn't have to run
git log to find a deleted ebuild.

-- 
Rich

Re: [gentoo-dev] the graveyard overlay

[Philip Webb]

160708 William Hubbs wrote:
> On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>> IMO the criteria should be whether they work or not,
>> not whether upstream is more or less active.
>> If they're blockers on other work, by all means cull them.
>> However, if the biggest problem with them is
>> that they're using a few inodes in the repo, they should probably stay.
> There is an overlay for packages that are removed from the official tree
> -- https://github.com/gentoo/graveyard --
> and that is where old software should go,
> if it doesn't have an active maintainer.

A lot of this lengthy discussion is missing some basic points,
though a few people have mentioned them in passing.
As someone who has used Gentoo exclusively since 2003
& who raised the objections to removal of Xcdroast + Nethack,
let me try to get you all to focus on the real-life issues.

(1) The fact that a pkg has little or no upstream support
or that it doesn't have an active Gentoo maintainer
is not a reason for removing it from the regular tree.

One basic reason some software is no longer being actively developed
is simply that they work perfectly well as they now are,
eg the file manager Krusader & the desktop manager Fluxbox :
both of these are very useful & have no drop-in replacements,
but very little development has occurred for several years.
The same is true of Xcdroast & Nethack, which have been threatened,
but which have been rescued after some small patches have been applied.
This is likely to be true of more + more pkgs, as time passes :
even changes in the kernel these days rarely affect desktop users.

(2) There are  3  basic categories of Gentoo user :
(a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
Each of these have different security concerns :
(a) need to be alert to the many threats from all over the Internet ;
(b) need (among other things) to prevent privilege escalation ;
(c) are largely immune to those types of threat,
though a few of the Internet variety can affect them.

The security objections raised against Xcdroast + Nethack
were both problems which would arise only on multi-user systems,
yet single-users were also to be deprived of access to them.
Perhaps part of the problem is that many Gentoo developers
also earn their livings as sysadmins with many users or many servers :
the simpler happier world of single-users escapes their attention.

(3) Users generally don't want to be developers : they're too busy or too old.
Asking them "Are you willing to maintain it yourself ?" is a silly excuse ;
offering them the chance to dig around in a graveyard is even worse ;
even maintaining an overlay is a nuisance : I tried it with KDE Sunset.
Neither Xcdroast nor Nethack belong in a graveyard of any kind :
once the obscure security problems have been fixed,
they belong in the regular tree marked 'stable',
like many other pkgs whose development has been completed.

Users all do -- or should -- appreciate the unpaid work of the developers,
but developers also need to realise that without non-developer users
Gentoo would very quickly die & their justified pride + satisfaction die too.

(4) I have  3  simple recommendations to fix the everyday problems.

(a) the justification for tree-cleaning should be explicitly
that a pkg either (i) won't compile, (ii) crashes when run
or (iii) has a serious security hole which affects all  3  types of user.

(b) there needs to be a developer role 'General Maintainer',
who should be available to look at pkgs which have no regular maintainer,
but which compile, run properly & are generally secure :
their job would be to step in, like Mr Savchenko -- thanks again -- ,
to fix small problems which would otherwise be neglected ;
less formally, all developers might see it as part of their role
to help out occasionally with such small problems.

(c) Gentoo's rules + policies need explicitly to reflect the fact
that there are  3  types of user, as described :
eg some pkgs might be marked as 'not safe for multi-user systems' ;
that would recognise real distinctions which are now being ignored.

HTH & thanks as always to all of you for making Gentoo work since 2003.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-dev] Re: the graveyard overlay

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

On Fri, 8 Jul 2016 19:17:44 +0300, Andrew Savchenko wrote:

> > Completely aside from the question of criteria for removing stuff from
> > the main tree, it would make a lot of users happy if every package
> > which *is* removed were added to the graveyard overlay.  
> 
> Good idea, since we have attic no more, and users need this [1].
> While any removed package can be extracted from git, this is not
> a trivial operation and is definitely harder then fetching filesv
> from overlay.
> 
> But the problem is that this way overlay will become completely
> broken in terms of both QA and security.

That's no different from ebuilds in the CVS attic. They were no longer
maintained and may even have been removed for security reasons, or even
failure to build, but they were still made available through the attic.


-- 
Neil Bothwick

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 6973 bytes --]

On Fri, Jul 8, 2016 at 1:21 PM, Philip Webb <purslow@ca.inter.net> wrote:

> 160708 William Hubbs wrote:
> > On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
> >> IMO the criteria should be whether they work or not,
> >> not whether upstream is more or less active.
> >> If they're blockers on other work, by all means cull them.
> >> However, if the biggest problem with them is
> >> that they're using a few inodes in the repo, they should probably stay.
> > There is an overlay for packages that are removed from the official tree
> > -- https://github.com/gentoo/graveyard --
> > and that is where old software should go,
> > if it doesn't have an active maintainer.
>
> A lot of this lengthy discussion is missing some basic points,
> though a few people have mentioned them in passing.
> As someone who has used Gentoo exclusively since 2003
> & who raised the objections to removal of Xcdroast + Nethack,
> let me try to get you all to focus on the real-life issues.
>
> (1) The fact that a pkg has little or no upstream support
> or that it doesn't have an active Gentoo maintainer
> is not a reason for removing it from the regular tree.
>

So basically what you are advocating for is:

"Having completely unmaintained packages in the tree is OK."

And honestly, I do not buy that premise.


>
> One basic reason some software is no longer being actively developed
> is simply that they work perfectly well as they now are,
> eg the file manager Krusader & the desktop manager Fluxbox :
> both of these are very useful & have no drop-in replacements,
> but very little development has occurred for several years.
> The same is true of Xcdroast & Nethack, which have been threatened,
> but which have been rescued after some small patches have been applied.
> This is likely to be true of more + more pkgs, as time passes :
> even changes in the kernel these days rarely affect desktop users.
>

No one is trying to remove flubox (which had a release in 2015 and had
activity in its git repo as recently as last week.)

Xcdroast for example, hasn't had a release in 8 years and I can't even find
its source tracker in sourceforge. These are the sorts of packages that I
think are not great to have in the tree and for Xcdroast, if I were
treecleaner lead i would probably advocate for working around the security
bug (dropped SUID) instead of removal. I do not necessarily want to remove
software that people are using.

That being said, I do not want unmaintained software in the tree either.


>
> (2) There are  3  basic categories of Gentoo user :
> (a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
> Each of these have different security concerns :
> (a) need to be alert to the many threats from all over the Internet ;
> (b) need (among other things) to prevent privilege escalation ;
> (c) are largely immune to those types of threat,
> though a few of the Internet variety can affect them.
>

I appreciate the argument you are trying to make; but i do not think it
really drives Gentoo Security Policy (nor should it.)

As my security manager used to say "security is not a race to the bottom."


>
> The security objections raised against Xcdroast + Nethack
> were both problems which would arise only on multi-user systems,
> yet single-users were also to be deprived of access to them.
> Perhaps part of the problem is that many Gentoo developers
> also earn their livings as sysadmins with many users or many servers :
> the simpler happier world of single-users escapes their attention.
>
> (3) Users generally don't want to be developers : they're too busy or too
> old.
> Asking them "Are you willing to maintain it yourself ?" is a silly excuse ;
> offering them the chance to dig around in a graveyard is even worse ;
> even maintaining an overlay is a nuisance : I tried it with KDE Sunset.
> Neither Xcdroast nor Nethack belong in a graveyard of any kind :
> once the obscure security problems have been fixed,
> they belong in the regular tree marked 'stable',
> like many other pkgs whose development has been completed.
>
> Users all do -- or should -- appreciate the unpaid work of the developers,
> but developers also need to realise that without non-developer users
> Gentoo would very quickly die & their justified pride + satisfaction die
> too.
>

I'm a bit confused by this argument.

1) It appears that no Gentoo developers want to maintain a software package.
2) The software package has no active upstream.
3) The software has no bugs.
4) We mask the software because it has bugs and no active maintianer, for
years.
5) No one volunteers to proxy-maintain the software.

But you advocate we keep such software in the tree, because users are "too
busy" or "too old" to maintain it themselves?


>
> (4) I have  3  simple recommendations to fix the everyday problems.
>
> (a) the justification for tree-cleaning should be explicitly
> that a pkg either (i) won't compile, (ii) crashes when run
> or (iii) has a serious security hole which affects all  3  types of user.
>

> (b) there needs to be a developer role 'General Maintainer',
> who should be available to look at pkgs which have no regular maintainer,
> but which compile, run properly & are generally secure :
> their job would be to step in, like Mr Savchenko -- thanks again -- ,
> to fix small problems which would otherwise be neglected ;
> less formally, all developers might see it as part of their role
> to help out occasionally with such small problems.
>

In an ideal world, the tree would be full of properly maintained packages.

There are over 1500 packages in the tree in the 'maintainer-needed'
state[1].

Even if we allocated 100 packages per developer, this "General Maintainer"
team would be 15 developers strong and one of the largest projects in
Gentoo. To compare the Treecleaner project is 7 people; the Security
project is 10 people.

This is in fact part of the rationale of the Treecleaner project itself.
Ebuilds require maintenance (eclass updates, new EAPIs, etc) and someone
has to do this work (or we end up with 6 supports EAPIs in the tree.) This
is one reason why packages that are unmaintained are removed; we do not
have 15 spare humans to clean up the unmaintained packages, so we remove
them when it is feasible to do so.


> (c) Gentoo's rules + policies need explicitly to reflect the fact
> that there are  3  types of user, as described :
> eg some pkgs might be marked as 'not safe for multi-user systems' ;
> that would recognise real distinctions which are now being ignored.
>
> HTH & thanks as always to all of you for making Gentoo work since 2003.
>


[1] https://qa-reports.gentoo.org/output/maintainer-needed.html


>
> --
> ========================,,============================================
> SUPPORT     ___________//___,   Philip Webb
> ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
> TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca
>
>
>

[-- Attachment #2: Type: text/html, Size: 9809 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 7331 bytes --]

On Fri, Jul 8, 2016 at 2:50 PM, Alec Warner <antarus@gentoo.org> wrote:

>
>
> On Fri, Jul 8, 2016 at 1:21 PM, Philip Webb <purslow@ca.inter.net> wrote:
>
>> 160708 William Hubbs wrote:
>> > On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>> >> IMO the criteria should be whether they work or not,
>> >> not whether upstream is more or less active.
>> >> If they're blockers on other work, by all means cull them.
>> >> However, if the biggest problem with them is
>> >> that they're using a few inodes in the repo, they should probably stay.
>> > There is an overlay for packages that are removed from the official tree
>> > -- https://github.com/gentoo/graveyard --
>> > and that is where old software should go,
>> > if it doesn't have an active maintainer.
>>
>> A lot of this lengthy discussion is missing some basic points,
>> though a few people have mentioned them in passing.
>> As someone who has used Gentoo exclusively since 2003
>> & who raised the objections to removal of Xcdroast + Nethack,
>> let me try to get you all to focus on the real-life issues.
>>
>> (1) The fact that a pkg has little or no upstream support
>> or that it doesn't have an active Gentoo maintainer
>> is not a reason for removing it from the regular tree.
>>
>
> So basically what you are advocating for is:
>
> "Having completely unmaintained packages in the tree is OK."
>
> And honestly, I do not buy that premise.
>
>
>>
>> One basic reason some software is no longer being actively developed
>> is simply that they work perfectly well as they now are,
>> eg the file manager Krusader & the desktop manager Fluxbox :
>> both of these are very useful & have no drop-in replacements,
>> but very little development has occurred for several years.
>> The same is true of Xcdroast & Nethack, which have been threatened,
>> but which have been rescued after some small patches have been applied.
>> This is likely to be true of more + more pkgs, as time passes :
>> even changes in the kernel these days rarely affect desktop users.
>>
>
> No one is trying to remove flubox (which had a release in 2015 and had
> activity in its git repo as recently as last week.)
>
> Xcdroast for example, hasn't had a release in 8 years and I can't even
> find its source tracker in sourceforge. These are the sorts of packages
> that I think are not great to have in the tree and for Xcdroast, if I were
> treecleaner lead i would probably advocate for working around the security
> bug (dropped SUID) instead of removal. I do not necessarily want to remove
> software that people are using.
>
> That being said, I do not want unmaintained software in the tree either.
>
>
>>
>> (2) There are  3  basic categories of Gentoo user :
>> (a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
>> Each of these have different security concerns :
>> (a) need to be alert to the many threats from all over the Internet ;
>> (b) need (among other things) to prevent privilege escalation ;
>> (c) are largely immune to those types of threat,
>> though a few of the Internet variety can affect them.
>>
>
> I appreciate the argument you are trying to make; but i do not think it
> really drives Gentoo Security Policy (nor should it.)
>
> As my security manager used to say "security is not a race to the bottom."
>
>
>>
>> The security objections raised against Xcdroast + Nethack
>> were both problems which would arise only on multi-user systems,
>> yet single-users were also to be deprived of access to them.
>> Perhaps part of the problem is that many Gentoo developers
>> also earn their livings as sysadmins with many users or many servers :
>> the simpler happier world of single-users escapes their attention.
>>
>> (3) Users generally don't want to be developers : they're too busy or too
>> old.
>> Asking them "Are you willing to maintain it yourself ?" is a silly excuse
>> ;
>> offering them the chance to dig around in a graveyard is even worse ;
>> even maintaining an overlay is a nuisance : I tried it with KDE Sunset.
>> Neither Xcdroast nor Nethack belong in a graveyard of any kind :
>> once the obscure security problems have been fixed,
>> they belong in the regular tree marked 'stable',
>> like many other pkgs whose development has been completed.
>>
>> Users all do -- or should -- appreciate the unpaid work of the developers,
>> but developers also need to realise that without non-developer users
>> Gentoo would very quickly die & their justified pride + satisfaction die
>> too.
>>
>
> I'm a bit confused by this argument.
>
> 1) It appears that no Gentoo developers want to maintain a software
> package.
> 2) The software package has no active upstream.
> 3) The software has no bugs.
>

Sorry, in my argument the package has open bugs, I mis-typed ;)


> 4) We mask the software because it has bugs and no active maintianer, for
> years.
> 5) No one volunteers to proxy-maintain the software.
>
> But you advocate we keep such software in the tree, because users are "too
> busy" or "too old" to maintain it themselves?
>
>
>>
>> (4) I have  3  simple recommendations to fix the everyday problems.
>>
>> (a) the justification for tree-cleaning should be explicitly
>> that a pkg either (i) won't compile, (ii) crashes when run
>> or (iii) has a serious security hole which affects all  3  types of user.
>>
>
>> (b) there needs to be a developer role 'General Maintainer',
>> who should be available to look at pkgs which have no regular maintainer,
>> but which compile, run properly & are generally secure :
>> their job would be to step in, like Mr Savchenko -- thanks again -- ,
>> to fix small problems which would otherwise be neglected ;
>> less formally, all developers might see it as part of their role
>> to help out occasionally with such small problems.
>>
>
> In an ideal world, the tree would be full of properly maintained packages.
>
> There are over 1500 packages in the tree in the 'maintainer-needed'
> state[1].
>
> Even if we allocated 100 packages per developer, this "General Maintainer"
> team would be 15 developers strong and one of the largest projects in
> Gentoo. To compare the Treecleaner project is 7 people; the Security
> project is 10 people.
>
> This is in fact part of the rationale of the Treecleaner project itself.
> Ebuilds require maintenance (eclass updates, new EAPIs, etc) and someone
> has to do this work (or we end up with 6 supports EAPIs in the tree.) This
> is one reason why packages that are unmaintained are removed; we do not
> have 15 spare humans to clean up the unmaintained packages, so we remove
> them when it is feasible to do so.
>
>
>> (c) Gentoo's rules + policies need explicitly to reflect the fact
>> that there are  3  types of user, as described :
>> eg some pkgs might be marked as 'not safe for multi-user systems' ;
>> that would recognise real distinctions which are now being ignored.
>>
>> HTH & thanks as always to all of you for making Gentoo work since 2003.
>>
>
>
> [1] https://qa-reports.gentoo.org/output/maintainer-needed.html
>
>
>>
>> --
>> ========================,,============================================
>> SUPPORT     ___________//___,   Philip Webb
>> ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
>> TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 10651 bytes --]

Re: [gentoo-dev] the graveyard overlay

[Philip Webb]

160708 Alec Warner wrote:
> On Fri, Jul 8, 2016 at 1:21 PM, Philip Webb <purslow@ca.inter.net> wrote:
>> (1) The fact that a pkg has little or no upstream support
>> or that it doesn't have an active Gentoo maintainer
>> is not a reason for removing it from the regular tree.
> So basically what you are advocating for is:
> "Having completely unmaintained packages in the tree is OK".
> And honestly, I do not buy that premise.

I went on to point out :
>> One basic reason some software is no longer being actively developed
>> is simply that they work perfectly well as they now are,
>> eg the file manager Krusader & the desktop manager Fluxbox :
>> both of these are very useful & have no drop-in replacements,
>> but very little development has occurred for several years.
>> The same is true of Xcdroast & Nethack, which have been threatened,
>> but which have been rescued after some small patches have been applied.
>> This is likely to be true of more + more pkgs, as time passes :
>> even changes in the kernel these days rarely affect desktop users.

My point here is that lack of upstream development doesn't necessarily mean
that an app is "dead", but may simply result from it's being completed.
Xcdroast simply works & had  1  obscure security problem, now fixed.
The problem with other burners is that they demand sound software,
which I have no need for on my system, so I want to go on using
the simple reliable app which has always got the job done.

> No one is trying to remove Flubox, which had a release in 2015
> and had activity in its git repo as recently as last week.

Not yet, they're not, but changes have been minimal for a long time.
Krusader had an update in Git 24 hours ago,
but the latest version is 3 years old.
It's an excellent file manager, which I rely on regularly,
but it's only "semi-alive" upstream.

> Xcdroast for example, hasn't had a release in 8 years
> and I can't even find its source tracker in Sourceforge.
> These are the sorts of packages I think are not great to have in the tree
> and for Xcdroast, if I were treecleaner lead, I would probably advocate
> for working around the security bug (dropped SUID) instead of removal.
> I do not necessarily want to remove software that people are using.

So you are saying -- perhaps correctly -- that the problem here
was not bad tree-cleaning policy, but incompetent tree-cleaning
(I don't mean to criticise whoever did it : I make mistakes too).

> That being said, I do not want unmaintained software in the tree either.

This is not black vs white : a package can be 'lightly maintained',
ie there's no regular maintainer, but equally there are no real problems
& those which exist could be fixed fairly easily, if need be.
That was the case with Xcdroast & earlier with Nethack.

So another suggestion from me for Gentoo policy
-- like recognising different categories of user --
is to create a new class of pkg called 'lightly maintained',
which would include older but still useable software,
which is no longer being actively developed, as it is largely complete.

>> (2) There are  3  basic categories of Gentoo user :
>> (a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
>> Each of these have different security concerns :
>> (a) need to be alert to the many threats from all over the Internet ;
>> (b) need (among other things) to prevent privilege escalation ;
>> (c) are largely immune to those types of threat,
>> though a few of the Internet variety can affect them.
> I appreciate the argument you are trying to make,
> but I do not think it should drive Gentoo Security Policy.

Surely, it's very relevant for the reasons I have listed :
eg I don't have to worry re privilege escalation,
as I can escalate my privileges anytime I want by opening a root terminal
(no-one else has physical access to my machine).

> As my security manager used to say "security is not a race to the bottom".

Obviously true, but that's not in question here.

> Suppose :
> 1) It appears that no Gentoo developers want to maintain a package.
> 2) The software package has no active upstream.
> 3) The software has open bugs.
> 4) We mask it for years, because it has bugs and no active maintainer.
> 5) No one volunteers to proxy-maintain the software.
> You advocate we keep such software in the tree,
> because users are "too busy" or "too old" to maintain it themselves ?

Yes, I do, depending on how serious the bugs are : in the cases
of Xcdroast + Nethack, they were not serious on single-user systems.
Nor do I accept your scare quotes : most users are too busy
to be able to become developers nor should they be asked to ;
the average age of Gentoo developers seems to be around 30 years,
so anyone 50 years old or more would find the job that much more challenging.

>> (b) there needs to be a developer role 'General Maintainer',
> In an ideal world, the tree would be full of properly maintained packages.
> There are > 1500 packages in the tree in the 'maintainer-needed' state :
> see  https://qa-reports.gentoo.org/output/maintainer-needed.html
> Even if we allocated 100 packages per developer,
> this "General Maintainer" team would be 15 developers strong
> and one of the largest projects in Gentoo.  To compare,
> the Treecleaner project is 7 people, the Security project is 10 people.

If there were a class of packages which were 'lightly maintained',
a developer could keep an occasional eye on > 100
&  7  devs could cover nearly half of the total.
Xcdroast hasn't needed attention since the bug report in 2010 :
it was only over-hasty behaviour by a tree-cleaner which created a problem
(again, I'm not targeting anyone for criticism) ;
Krusader hasn't had a new version since 2013 .
That will be true of more + more pkgs as the free-software galaxy matures.

> This is in fact part of the rationale of the Treecleaner project itself.
> Ebuilds require maintenance (eclass updates, new EAPIs, etc)
> and someone has to do this work
> or we end up with 6 supports EAPIs in the tree.
> This is one reason why packages that are unmaintained are removed :
> we do not have 15 spare humans to clean up the unmaintained packages,
> so we remove them when it is feasible to do so.

No-one has been doing any such maintenance on Xcdroast or Krusader,
but they go on doing their useful jobs for a number of Gentoo users.
A 'lightly maintained' pkg might get attention every year or two typically.

Thanks for your prompt + intelligent response.
I hope my thoughts from user-land will help to improve Gentoo policies.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-dev] the graveyard overlay

[james]

On 07/08/2016 04:50 PM, Alec Warner wrote:
>
>
> On Fri, Jul 8, 2016 at 1:21 PM, Philip Webb <purslow@ca.inter.net
> <mailto:purslow@ca.inter.net>> wrote:
>
>     160708 William Hubbs wrote:
>     > On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>      >> IMO the criteria should be whether they work or not,
>      >> not whether upstream is more or less active.
>     >> If they're blockers on other work, by all means cull them.
>     >> However, if the biggest problem with them is
>      >> that they're using a few inodes in the repo, they should
>     probably stay.
>      > There is an overlay for packages that are removed from the
>     official tree
>      > -- https://github.com/gentoo/graveyard --
>      > and that is where old software should go,
>     > if it doesn't have an active maintainer.
>
>     A lot of this lengthy discussion is missing some basic points,
>     though a few people have mentioned them in passing.
>     As someone who has used Gentoo exclusively since 2003
>     & who raised the objections to removal of Xcdroast + Nethack,
>     let me try to get you all to focus on the real-life issues.
>
>     (1) The fact that a pkg has little or no upstream support
>     or that it doesn't have an active Gentoo maintainer
>     is not a reason for removing it from the regular tree.

+1

>
> So basically what you are advocating for is:
> "Having completely unmaintained packages in the tree is OK."
> And honestly, I do not buy that premise.

I think what Phillip has outlined is that different use cases have 
different prospective on use, stability and security.  Take the ancient 
serial (rs232-C) code based net-dialup/minicom for example. It has long 
outlived it's usefulness for most users. Serial ports that have TCP/IP 
laid on top of them use to form the backbone of phone line modem access.

"Upstream:    None specified"

Yet it is still extraordinarily useful to embedded systems work, and 
industrial uses, such as modbusTCP/ppp/rs232. To a "go hacker" it needs 
to be cleaned out, but to an older or embedded hacker it is fundamental 
to raw hardware access. Granted the "Maintainer:  embedded@gentoo.org 
(Embedded Gentoo)" has stepped up. So, my point is there are (probably)
lots of other example codes in the 1500 unmaintained packages that have 
not yet been scrutinized by the larger gentoo user community. I've got 
over a dozen deprecated packages in my /usr/local/portage/ tree, as I 
try to keep up with the tree cleaners on old useful codes.

"no maintainer" should not be part of the criteria for removal, at least 
not for a year or so. WE need to document the methods and procedures for 
proxy-maint, to a robust level, before such actions are warranted. Then 
a campaign to recruit proxy maintainers for a while.
Right now the devManual, quizzes, repo structures(github, sunrise etc), 
eapi, and many other aspects of gentoo are in a phase of rapid change. 
It's going to take a while to get things stabilized and documented and 
then a period of recruitment. I know, I for one have been on this path
and things are time consuming. Just look at the dev-wrangling over many 
of these issues. That is the reason we have 1500 orphaned packages.

For now, I suggest we just label these packages as deficient, 
no-maintainer, security, inactive-upstream and any other relevant 
statuses. While some parse the proxy-maint correspondences to first 
create FAQs and then a guide, with some basic examples. Then aggressive 
removal would be viable and community supported.

This effort would also bode well for corporations to train and maintain
their linux sources with lots of folks participating in a full-spectrum
structure, that these discussions illuminate, should we want to attract 
new and younger hackers to gentoo. A well defined proxy-maint program 
would do more for gentoo recruiting (of both devs and users) than most
any thing else, imho. Teach a user how to create and maintain their own 
code, and you'll have a user for life.


>     One basic reason some software is no longer being actively developed
>     is simply that they work perfectly well as they now are,
>     eg the file manager Krusader & the desktop manager Fluxbox :
>     both of these are very useful & have no drop-in replacements,
>     but very little development has occurred for several years.
>     The same is true of Xcdroast & Nethack, which have been threatened,
>     but which have been rescued after some small patches have been applied.
>     This is likely to be true of more + more pkgs, as time passes :
>     even changes in the kernel these days rarely affect desktop users.

!1.

>
> No one is trying to remove flubox (which had a release in 2015 and had
> activity in its git repo as recently as last week.)
>
> Xcdroast for example, hasn't had a release in 8 years and I can't even
> find its source tracker in sourceforge. These are the sorts of packages
> that I think are not great to have in the tree and for Xcdroast, if I
> were treecleaner lead i would probably advocate for working around the
> security bug (dropped SUID) instead of removal. I do not necessarily
> want to remove software that people are using.
>
> That being said, I do not want unmaintained software in the tree either.

Funny, I used xcdroast to burn a couple of systemrescue images, just 
yesterday, and I never used it before yesterday.


>     (2) There are  3  basic categories of Gentoo user :
>     (a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
>     Each of these have different security concerns :
>     (a) need to be alert to the many threats from all over the Internet ;
>     (b) need (among other things) to prevent privilege escalation ;
>     (c) are largely immune to those types of threat,
>     though a few of the Internet variety can affect them.

great perspective. I'd add a forth category of user. 
Embedded/close-network users of gentoo. For products, you do not need to 
use the internet, until the very end of product development or product 
testing.
Old codes really shine in this forth category, and gentoo is definitely 
one of the champions in this stand-alone space. We are the only major 
distro to support both systemd and legacy init. It's not clear how many 
embedded projects will even use systemd. (think new arm boards that we 
all are anticipating).


> I appreciate the argument you are trying to make; but i do not think it
> really drives Gentoo Security Policy (nor should it.)

Basically the current gentoo security policy is a "greedy algorithm" 
that too aggressively makes too many mistakes (on what to remove) by
a limited scope of how linux is used in the wider world. Not every linux 
system is a multi-user internet connected risk. Granted Rf interfaces 
may change that, but most responsible designers have a way to 'turn 
down' RF interfaces. Surely a hacker can identify the single trace that 
is the antennae connection and ground it out, should the manufacture not 
provide a software switch. That is a vendor choice of Rf turndown is a 
problem in the new IoT family of products, but you can blame the NSA for 
that sort of crap. Sot if a system is not connect to the outside world, 
99% of the security risks disappear.

Please stop assuming the worst case environment that a code is used
for. If you want that, then create a tree within a tree, such as what 
the good folks of the 'hardened' project do. Work to assist them to add
codes to the musl platform, rather that working the fringes of the 
existing (rich) portage tree.

If you did that, then we could all (easily) routinely secure our 
connections to the internet, and keep other projects alive that do not
need a 'security onion' [1] level of security. Perhaps the security team 
needs to mimic the security onion project, but build everything on 
hardened gentoo? That effort would do vastly more to secure gentoo 
users, than fuzzing codes, one at time. We have Pentoo for penetration, 
but where is the equivalent for defense?

[1] https://securityonion.net/


> As my security manager used to say "security is not a race to the bottom."

Anecdotal limericks are mostly useless in real security. Real security, 
starts at the overall schema. Where was the security project on the 
recent malaise of check sums not matching for minimal iso? It appears to 
many that the security team, needs to be "refocused" and more amenable 
to valid suggestions and a  new set of priorities, imho. ymmv.

>     The security objections raised against Xcdroast + Nethack
>     were both problems which would arise only on multi-user systems,
>     yet single-users were also to be deprived of access to them.
>     Perhaps part of the problem is that many Gentoo developers
>     also earn their livings as sysadmins with many users or many servers :
>     the simpler happier world of single-users escapes their attention.

Excellent point!

>     (3) Users generally don't want to be developers : they're too busy
>     or too old.
>     Asking them "Are you willing to maintain it yourself ?" is a silly
>     excuse ;
>     offering them the chance to dig around in a graveyard is even worse ;
>     even maintaining an overlay is a nuisance : I tried it with KDE Sunset.
>     Neither Xcdroast nor Nethack belong in a graveyard of any kind :
>     once the obscure security problems have been fixed,
>     they belong in the regular tree marked 'stable',
>     like many other pkgs whose development has been completed.

Any effort to separate packages from the tree, should *first* be meet 
with a robust way to maintain and retrieve those codes. Destruction of a 
rich legacy of codes, many simple and tightly focus, robs gentoo or
a very rich part of it's heritage, needlessly.


>     Users all do -- or should -- appreciate the unpaid work of the
>     developers,
>     but developers also need to realise that without non-developer users
>     Gentoo would very quickly die & their justified pride + satisfaction
>     die too.
>
>
> I'm a bit confused by this argument.
>
> 1) It appears that no Gentoo developers want to maintain a software package.

(um, you missed the keyword that is missing, "currently")

> 2) The software package has no active upstream.

One fork fixes the no active upstream.

> 3) The software has no bugs.
> 4) We mask the software because it has bugs and no active maintianer,
> for years.

The word "bugs" means many things to different folks. It's a 
full-spectrum word that you use as an inaccurate 'monolithic' metric.

> 5) No one volunteers to proxy-maintain the software.

This is because of numerous, aforementioned changes, including but not 
limited to the roll out of github, and the lack of sufficient mechanisms 
for package retrieval, compared to what we had with the attic. If there 
is a robust method via git(hub), then just document those methods, 
because git is not an intuitively obvious collect of methodologies to 
replace what we had. That and the aggressive tree cleansing is unfair to 
the wider user communities, many of which only drop in occasionally.

>
> But you advocate we keep such software in the tree, because users are
> "too busy" or "too old" to maintain it themselves?

Not a consensus viewpoint but anecdotal, see above comments.


>     (4) I have  3  simple recommendations to fix the everyday problems.
>
>     (a) the justification for tree-cleaning should be explicitly
>     that a pkg either (i) won't compile, (ii) crashes when run
>     or (iii) has a serious security hole which affects all  3  types of
>     user.
>
>
>     (b) there needs to be a developer role 'General Maintainer',
>     who should be available to look at pkgs which have no regular
>     maintainer,
>     but which compile, run properly & are generally secure :
>     their job would be to step in, like Mr Savchenko -- thanks again -- ,
>     to fix small problems which would otherwise be neglected ;
>     less formally, all developers might see it as part of their role
>     to help out occasionally with such small problems.

Good points (+1). It takes time to grow up a proxy team. It takes time 
to develop documents that are github eapi 6/7 centric and a 
proxy-dev-guide. It takes time, so stop the aggressive tree pruning, in 
the absence of a fully functional, well documented system like the attic 
that is user friendly to learn and use.

> In an ideal world, the tree would be full of properly maintained packages.

Yes we agree on this, but, the security and tree-cleaners are being too 
aggressive as the proxy workflow is new and still not fully functional 
atm. Once this is fixed, it going to take a while (6 months to a year).
How long did the github migration planning and execution take?


> There are over 1500 packages in the tree in the 'maintainer-needed'
> state[1].

So what? Dont use them (gentoo is about choice right?). Modify a 
existing, common portage tool (eix?) to label your issues with these codes.

> Even if we allocated 100 packages per developer, this "General
> Maintainer" team would be 15 developers strong and one of the largest
> projects in Gentoo. To compare the Treecleaner project is 7 people; the
> Security project is 10 people.

How about a refocus on a security onion, gentoo centric, for the 
security team. Many of us would appreciate that sort of security much 
more than removal of old packages.


> This is in fact part of the rationale of the Treecleaner project itself.
> Ebuilds require maintenance (eclass updates, new EAPIs, etc) and someone
> has to do this work (or we end up with 6 supports EAPIs in the tree.)
> This is one reason why packages that are unmaintained are removed; we do
> not have 15 spare humans to clean up the unmaintained packages, so we
> remove them when it is feasible to do so.

Correct. What you are not fairly considering is the rapid changes and 
your time allowances for the user community to adapt to all of these 
changes. Are we nixos or gentoo? At least nixos has documentation [2].

[2] https://nixos.org/wiki/Development_Environments

On Gentoo's proxy project, there is scant documentation on a myriad of 
core issues. The proxy-maint ML is still not functioning for user 
questions and FAQ parsing.


>     (c) Gentoo's rules + policies need explicitly to reflect the fact
>     that there are  3  types of user, as described :
>     eg some pkgs might be marked as 'not safe for multi-user systems' ;
>     that would recognise real distinctions which are now being ignored.
>
>     HTH & thanks as always to all of you for making Gentoo work since 2003.

Likewise. I appreciated the devs and the gentoo distro, immensely. But, 
we need a well documented pathway from start to finish on the 
proxy-maint (regardless of what the details are) before such radical 
pruning, and that includes robust archiving of what is necessary to
personally restore old packages. Use the old attic as your basis of 
reasonable functionality.


James

> [1] https://qa-reports.gentoo.org/output/maintainer-needed.html
>
>
>     --
>     ========================,,============================================
>     SUPPORT     ___________//___,   Philip Webb
>     ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
>     TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca
>
>
>