[gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Starting from a question by Markos in #gentoo-portage about whether to
remove entries in profiles/updates for tree-cleaned packages...

I propose that we say, once a year, schedule a tree-cleaning of old
updates files.  These updates files could be added to a tarball made
available for download.  That way if they are needed to update a system
older than what the main tree has been tree-cleaned to. They can then be
manually downloaded, extracted to the normal location and then run the
"fixpackages" command.

The main question here is what is a reasonable length of time to keep
the updates actively in-tree?

 -- From my experience in the forums, I think any updates older than 
    4 years should be subject to tree-cleaning.  

 -- Most old systems that have been updated tend to be less than that,
    probably about 2 years.


This should allow a reasonable amount of time for a user to update an
old system.

Thoughts?

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

On 12/9/12 1:17 PM, Brian Dolbec wrote:
> Starting from a question by Markos in #gentoo-portage about whether to
> remove entries in profiles/updates for tree-cleaned packages...

What's the advantage of doing that?

> I propose that we say, once a year, schedule a tree-cleaning of old
> updates files.  These updates files could be added to a tarball made
> available for download.  That way if they are needed to update a system
> older than what the main tree has been tree-cleaned to. They can then be
> manually downloaded, extracted to the normal location and then run the
> "fixpackages" command.

I think that complicates the process. :-/ But maybe the advantages
outweigh that.

> The main question here is what is a reasonable length of time to keep
> the updates actively in-tree?
> 
>  -- From my experience in the forums, I think any updates older than 
>     4 years should be subject to tree-cleaning.  

Yeah, 4 years is ancient and would probably be non-trivial to update anyway.

>  -- Most old systems that have been updated tend to be less than that,
>     probably about 2 years.

2 years seem reasonable.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2221 bytes --]

On Sun, 2012-12-09 at 15:10 -0800, "Paweł Hajdan, Jr." wrote:
> On 12/9/12 1:17 PM, Brian Dolbec wrote:
> > Starting from a question by Markos in #gentoo-portage about whether to
> > remove entries in profiles/updates for tree-cleaned packages...
> 
> What's the advantage of doing that?

None, it actually could make it more difficult for a user to know why
his old installed pkg isn't available.  It was just what started the
discussion about cleaning the old updates.  Zac suggested this thread
for opinions...

...
[12:46] <zmedico> dol-sen: you should take a poll on the gentoo-dev ml
to see how long people think we should keep them
...
[12:47] <zmedico> yeah, seems like it's good to end-of-life them at some
point

> 
> > I propose that we say, once a year, schedule a tree-cleaning of old
> > updates files.  These updates files could be added to a tarball made
> > available for download.  That way if they are needed to update a system
> > older than what the main tree has been tree-cleaned to. They can then be
> > manually downloaded, extracted to the normal location and then run the
> > "fixpackages" command.
> 
> I think that complicates the process. :-/ But maybe the advantages
> outweigh that.
> 

It does make updating an ancient system slightly more difficult. But
that would be the least of the user's troubles compared to some of the
pkg updates, tinderbox downloads and manual unpacks that have been
needed to be done.

But on the other hand how long should we keep that stale info in the
tree?  See below :)

> > The main question here is what is a reasonable length of time to keep
> > the updates actively in-tree?
> > 
> >  -- From my experience in the forums, I think any updates older than 
> >     4 years should be subject to tree-cleaning.  
> 
> Yeah, 4 years is ancient and would probably be non-trivial to update anyway.

yup, they are

> 
> >  -- Most old systems that have been updated tend to be less than that,
> >     probably about 2 years.
> 
> 2 years seem reasonable.
> 

 That works too.  

FYI... Currently there are updates files in profiles/updates/ dating
back to 2004
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

Brian Dolbec wrote:
> > > remove entries in profiles/updates for tree-cleaned packages...
> > 
> > What's the advantage of doing that?
> 
> None
..
> FYI... Currently there are updates files in profiles/updates/
> dating back to 2004

Do they take up significant storage space or transfer time, compared
to the rest of the tree?

If not, I can't think of a reason to remove them.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1245 bytes --]

On Mon, 2012-12-10 at 01:52 +0100, Peter Stuge wrote:
> Brian Dolbec wrote:
> > > > remove entries in profiles/updates for tree-cleaned packages...
> > > 
> > > What's the advantage of doing that?
> > 
> > None
> ..
> > FYI... Currently there are updates files in profiles/updates/
> > dating back to 2004
> 
> Do they take up significant storage space or transfer time, compared
> to the rest of the tree?
> 
> If not, I can't think of a reason to remove them.
> 
> 
> //Peter


No, once they are downloaded, they don't change ever after the quarterly
rollover which starts a new updates file.  Nor do they take up
significant storage space.  They probably take up a higher percentage of
your fs's inodes than % diskspace.

But they do take time to process and verify every time emerge initiates
a fixpackages call after every sync.  That will save time more than it
will space.

Also, (taking it a bit to extremes ;) ) why would a new user with a 2
month old system and up-to-date tree want 8 year old update cruft
lingering about their filesystem...taking time to check they've been
processed? 
  Do you?

Let's just do an annual tree-cleaning and be done with them.
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

On Sun, 09 Dec 2012 21:15:37 -0800
Brian Dolbec <dolsen@gentoo.org> wrote:

> No, once they are downloaded, they don't change ever after the quarterly
> rollover which starts a new updates file.

gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time

4Q-2012
3Q-2012
1Q-2008
2Q-2012
4Q-2011
1Q-2012
3Q-2011
2Q-2011
1Q-2011
2Q-2005
3Q-2008
4Q-2010
1Q-2005
2Q-2008
3Q-2010
2Q-2010
1Q-2010
3Q-2004
4Q-2009
3Q-2009
2Q-2009
1Q-2009
4Q-2008
3Q-2005
3Q-2007
4Q-2007
2Q-2007
1Q-2007
4Q-2006
2Q-2006
3Q-2006
1Q-2006
4Q-2005
4Q-2004
2Q-2004
1Q-2004

old entries are done in different context (comparing to 2012):

- some packages change names 2 or 3 times
- slots have different meaning

moreover:

-  if you set your PORTDIR to different
   directory you'll get all that full update.
   And will break the system. Old profile entries
   used to break eclass-manpages and latex-base
   (due to double renaming)

Random example:

Let's loot at why 2Q-2005 was changed in 2010:
    https://bugs.gentoo.org/show_bug.cgi?id=351760

Thus the reason for removal is simple:
old entries are potentially buggy as nobody
verifies them.

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Zac Medico]

On 12/09/2012 09:15 PM, Brian Dolbec wrote:
> No, once they are downloaded, they don't change ever after the quarterly
> rollover which starts a new updates file.  Nor do they take up
> significant storage space.  They probably take up a higher percentage of
> your fs's inodes than % diskspace.
> 
> But they do take time to process and verify every time emerge initiates
> a fixpackages call after every sync.  That will save time more than it
> will space.

It skips the older files if their timestamp hasn't changed since it was
recorded in /var/cache/edb/mtimedb. So, there's minimal overhead from
keeping them, unless you lose /var/cache/edb/mtimedb somehow.
-- 
Thanks,
Zac

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Ulrich Mueller]

>>>>> On Mon, 10 Dec 2012, Sergei Trofimovich wrote:

> gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time
> [long list omitted]

> old entries are done in different context (comparing to 2012):

> - some packages change names 2 or 3 times
> - slots have different meaning

> moreover:

> -  if you set your PORTDIR to different directory you'll get all
>    that full update. And will break the system. Old profile entries
>    used to break eclass-manpages and latex-base (due to double
>    renaming)

It's worse: Bad entries in the old files may go unnoticed for a long
time. But if such a file is updated for whatever reason, it will be
reprocessed on users' systems, including any bad entries contained in
it.

> Thus the reason for removal is simple: old entries are potentially
> buggy as nobody verifies them.

I wouldn't even know how to verify them.

Let's remove that cruft. We can be extra conservative and keep five
years of backlog (i.e. everything from before 2008 would be removed
now).

Ulrich

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Michael Haubenwallner]

On 12/10/2012 12:10 AM, "Paweł Hajdan, Jr." wrote:
>> I propose that we say, once a year, schedule a tree-cleaning of old
>> updates files.  These updates files could be added to a tarball made
>> available for download.  That way if they are needed to update a system
>> older than what the main tree has been tree-cleaned to. They can then be
>> manually downloaded, extracted to the normal location and then run the
>> "fixpackages" command.
> 
> I think that complicates the process. :-/ But maybe the advantages
> outweigh that.
> 
>> The main question here is what is a reasonable length of time to keep
>> the updates actively in-tree?
>>
>>  -- From my experience in the forums, I think any updates older than 
>>     4 years should be subject to tree-cleaning.  
> 
> Yeah, 4 years is ancient and would probably be non-trivial to update anyway.
> 
>>  -- Most old systems that have been updated tend to be less than that,
>>     probably about 2 years.
> 
> 2 years seem reasonable.

For the records:
We do have some Gentoo box serving as VirtualBox host here, installed in early 2010,
not updated since then, with an uptime of 836 days right now. It is subject to upgrade,
but there may come another year until that to happen ("never change a running system").
Although I do not expect the update to be trivial, keeping things like pkgmove for at
least 4 years sounds reasonable.

/haubi/

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]

On Mon, 2012-12-10 at 08:59 +0100, Ulrich Mueller wrote:
> >>>>> On Mon, 10 Dec 2012, Sergei Trofimovich wrote:
> 
> > gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time
> > [long list omitted]
> 
> > old entries are done in different context (comparing to 2012):
> 
> > - some packages change names 2 or 3 times
> > - slots have different meaning
> 
> > moreover:
> 
> > -  if you set your PORTDIR to different directory you'll get all
> >    that full update. And will break the system. Old profile entries
> >    used to break eclass-manpages and latex-base (due to double
> >    renaming)
> 
> It's worse: Bad entries in the old files may go unnoticed for a long
> time. But if such a file is updated for whatever reason, it will be
> reprocessed on users' systems, including any bad entries contained in
> it.
> 
> > Thus the reason for removal is simple: old entries are potentially
> > buggy as nobody verifies them.
> 
> I wouldn't even know how to verify them.
> 
> Let's remove that cruft. We can be extra conservative and keep five
> years of backlog (i.e. everything from before 2008 would be removed
> now).
> 
> Ulrich
> 

OK, that seems to be some very good reasons to tree-clean them.

What's our next step?  
Tree-cleaners, does this fall into your department?  
Or should I prepare a list of files and/or updates to clean?
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Zac Medico]

On 01/02/2013 02:46 PM, Brian Dolbec wrote:
> On Mon, 2012-12-10 at 08:59 +0100, Ulrich Mueller wrote:
>>>>>>> On Mon, 10 Dec 2012, Sergei Trofimovich wrote:
>>
>>> gentoo-x86/profiles/updates $ LANG=C ls -1 --sort=time
>>> [long list omitted]
>>
>>> old entries are done in different context (comparing to 2012):
>>
>>> - some packages change names 2 or 3 times
>>> - slots have different meaning
>>
>>> moreover:
>>
>>> -  if you set your PORTDIR to different directory you'll get all
>>>    that full update. And will break the system. Old profile entries
>>>    used to break eclass-manpages and latex-base (due to double
>>>    renaming)
>>
>> It's worse: Bad entries in the old files may go unnoticed for a long
>> time. But if such a file is updated for whatever reason, it will be
>> reprocessed on users' systems, including any bad entries contained in
>> it.
>>
>>> Thus the reason for removal is simple: old entries are potentially
>>> buggy as nobody verifies them.
>>
>> I wouldn't even know how to verify them.
>>
>> Let's remove that cruft. We can be extra conservative and keep five
>> years of backlog (i.e. everything from before 2008 would be removed
>> now).
>>
>> Ulrich
>>
> 
> OK, that seems to be some very good reasons to tree-clean them.
> 
> What's our next step?

It might be nice to document the removal policy in the developer
handbook, devmanual, or something.

> Tree-cleaners, does this fall into your department?

That seems fitting.

> Or should I prepare a list of files and/or updates to clean?

This command seems to do the trick:

$ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$'
1Q-2004
1Q-2005
1Q-2006
1Q-2007
2Q-2004
2Q-2005
2Q-2006
2Q-2007
3Q-2004
3Q-2005
3Q-2006
3Q-2007
4Q-2004
4Q-2005
4Q-2006
4Q-2007


-- 
Thanks,
Zac

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

On Wed, 2013-01-02 at 17:05 -0800, Zac Medico wrote:

> This command seems to do the trick:
> 
> $ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$'
> 1Q-2004
> 1Q-2005
> 1Q-2006
> 1Q-2007
> 2Q-2004
> 2Q-2005
> 2Q-2006
> 2Q-2007
> 3Q-2004
> 3Q-2005
> 3Q-2006
> 3Q-2007
> 4Q-2004
> 4Q-2005
> 4Q-2006
> 4Q-2007
> 
> 

OK, these old updates have now been cleaned from the tree.

They have been tar'd and can be downloaded at:
http://dev.gentoo.org/~dolsen/cleaned-updates-20130111.tbz2

If we are going to archive them to be available. It can be moved to
there, otherwise I can remove it once we know it is not needed.

Now, where to document this procedure for future reference?
 
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal to end-of-life tree-clean old profiles/updates/ files

[Zac Medico]

On 01/11/2013 01:10 AM, Brian Dolbec wrote:
> On Wed, 2013-01-02 at 17:05 -0800, Zac Medico wrote:
> 
>> This command seems to do the trick:
>>
>> $ ls -1 /usr/portage/profiles/updates/ | grep -Ev '(08|09|10|11|12|13)$'
>> 1Q-2004
>> 1Q-2005
>> 1Q-2006
>> 1Q-2007
>> 2Q-2004
>> 2Q-2005
>> 2Q-2006
>> 2Q-2007
>> 3Q-2004
>> 3Q-2005
>> 3Q-2006
>> 3Q-2007
>> 4Q-2004
>> 4Q-2005
>> 4Q-2006
>> 4Q-2007
>>
>>
> 
> OK, these old updates have now been cleaned from the tree.
> 
> They have been tar'd and can be downloaded at:
> http://dev.gentoo.org/~dolsen/cleaned-updates-20130111.tbz2
> 
> If we are going to archive them to be available. It can be moved to
> there, otherwise I can remove it once we know it is not needed.
> 
> Now, where to document this procedure for future reference?
>  

Maybe here:

  http://www.gentoo.org/proj/en/qa/treecleaners/policy.xml
-- 
Thanks,
Zac