[gentoo-dev] ChangeLogs and rsync time

[gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

We have currently 10371 ChangeLog files, > 25 MB totally .
1365 == 13% with size >= 4096 B, 12 MB totally

rsync from "emerge --sync" has "--whole-file" between its options, that
mean transfer the whole file if changed.

To make things worse the bigger ChangeLogs are (oh, surprise) those that
change frequently (also the age count), following there is the list of
first 25:

110062  ./x11-base/xorg-x11/ChangeLog
96906   ./sys-devel/gcc/ChangeLog
86916   ./sys-libs/glibc/ChangeLog
72429   ./net-www/apache/ChangeLog
65331   ./sys-apps/baselayout/ChangeLog
61801   ./media-video/mplayer/ChangeLog
57688   ./dev-db/mysql/ChangeLog
53938   ./sys-kernel/gentoo-sources/ChangeLog
53810   ./net-im/gaim/ChangeLog
53173   ./www-client/mozilla/ChangeLog
51891   ./dev-php/mod_php/ChangeLog
48127   ./dev-db/postgresql/ChangeLog
47014   ./sys-devel/binutils/ChangeLog
46742   ./kde-base/kdelibs/ChangeLog
45370   ./dev-lang/perl/ChangeLog
44998   ./sys-kernel/mm-sources/ChangeLog
41010   ./kde-base/kdebase/ChangeLog
37644   ./www-client/mozilla-firefox/ChangeLog
37524   ./net-fs/samba/ChangeLog
36411   ./mail-mta/postfix/ChangeLog
35269   ./app-office/openoffice-ximian/ChangeLog
34890   ./app-office/openoffice/ChangeLog
34773   ./sys-kernel/mips-sources/ChangeLog
33245   ./media-sound/xmms/ChangeLog
32769   ./dev-util/subversion/ChangeLog


The information contained in the ChangeLogs is essential, and it must be
kept, but, force the users to download all that data it's not optimal.

That said I can see only two ways to reduce the ChangeLog files (a
centralized one is obviously not viable)

1) bzip2 them in some way.

   Pros:
   - whole story is avaiable
   - much lower file transfer size
   Cons:
   - grepping them need dedicated tools
   - never ending cvs issues, may be needed an alternate non cvs tree
   - vim already has a wrapper to read bz2 files, other editors does not
   - also 247 bytes file will need to be zipped ?

2) "rotate" Changelogs, keeping only the last changes, until a size
   of 4000 or [choose a preferred size here] bytes.
   This would save only about 7Mb of data (max size < 4096).

   Pros:
   - still easily readable and parseable
   - save download of data in the right points
   - affect only 13% of the actual tree
   Cons:
   - need changes in repoman/echangelog to cut the ChangeLog in the
     right position
   - grepping of ChangeLog impossible (for cutted data)
   - ChangeLog cutted at _no_ definite point in time (maybe yesterday)
   - whole history only on viewcvs or with acks like a
     sys-apps/ChangeLogs package

Thoughts ? It's doable in some way ?
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Grobian]

On 01-01-2006 21:35:34 +0100, Francesco Riosa wrote with possible deletions:
> The information contained in the ChangeLogs is essential, and it must be
> kept, but, force the users to download all that data it's not optimal.
> 
> That said I can see only two ways to reduce the ChangeLog files (a
> centralized one is obviously not viable)
> 
> 1) bzip2 them in some way.
> 2) "rotate" Changelogs, keeping only the last changes, until a size

or
3) remove entries for non-existing ebuilds
   This may, or may not be a good idea, but it is founded on the
   following observation: currently old or redundant ebuilds are removed
   from the tree.  Once that happens, they don't show up in the rsync
   tree and are only available through the (centralised) CVS Attic.  One
   can argue that Changelog entries for non-existing ebuilds are not of
   any use, since the files they refer to aren't present.
   This method would clean up the Changelog entries, in the same way
   ebuilds are removed, and CVS keeps the history around.
4) compress Changelog entries where possible
   Often, a package is marked testing or stable on request via a bug.
   This usually results in having as much as new Changelog entries as
   there are arch teams involved.  These kind of entries, that all more
   or less report 'Marked arch' could be merged into one, given that the
   information itself is Changelog-worthy anyway.  (This is arguable
   IMHO.)
   This method may involve a lot of fuzzy matching to perform it
   automatically, with its related risks.  The win for the large
   Changelog files is probably minimal.

-- 
Fabian Groffen
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: ChangeLogs and rsync time

[Duncan]

Grobian posted <20060101214818.GB17018@gentoo.org>, excerpted below,  on
Sun, 01 Jan 2006 22:48:18 +0100:

> On 01-01-2006 21:35:34 +0100, Francesco Riosa wrote with possible deletions:
>> The information contained in the ChangeLogs is essential, and it must be
>> kept, but, force the users to download all that data it's not optimal.
>> 
>> That said I can see only two ways to reduce the ChangeLog files (a
>> centralized one is obviously not viable)
>> 
>> 1) bzip2 them in some way.
>> 2) "rotate" Changelogs, keeping only the last changes, until a size
> 
> or
> 3) remove entries for non-existing ebuilds

> 4) compress Changelog entries where possible
>    Often, a package is marked testing or stable on request via a bug.
>    This usually results in having as much as new Changelog entries as
>    there are arch teams involved.

I'd say:

* Remove keywording entries for ebuilds no longer in the tree.

* Snip/rotate changelogs, but only manually, and where it makes sense, for
the largest changelogs.

For the largest, xorg, keep only from the earliest in-tree minor version,
forward.  Currently, we have 6.8.2-rX in the tree, so 6.8 forward, keeping
older info available by CVS only.  When the last 6.8 monolithic is
removed, that will leave only 7.x modular, and that particular changelog
won't grow so fast, as it'll be split into the component packages (the
changelogs of which, unfortunately, will grow faster, when the effect is
combined, due to entries in multiple changelogs that used too be just one).

gcc and glibc are #2,3. They will be tougher, due to the number of
versions retained in-tree.  Still, keyword bump entry removal for ebuilds
no longer in the tree will help some. 

* Standardize keyword bump entries.  Single description, with optional
bug number/reporter, followed by keywords in standardized (alpha?) order
per entry.  This will aid in compression, if decided upon, AND in
automated removal of keywording entries upon ebuild removal.

* Do /not/ remove other than keyword bump (only) entries for intermediate
ebuilds, those no longer in the tree, but bracketed by versions still in
the tree.  Much of this data will remain valid and useful, as it will
often continue to pertain to ebuilds still in-tree.  An example would be
the record of changes necessary to update an ebuild from one upstream
versiion to the next, after a security bump to -rX.  We would't
want to lose the ebuild version change documentation just because revision
zero was removed after the security bump.  (Keywording entries for ebuilds
no longer in-tree, however, aren't very useful, so they can be removed as
suggested above.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

On Sun, 01 Jan 2006 21:35:34 +0100 Francesco Riosa <vivo@gentoo.org>
wrote:
| That said I can see only two ways to reduce the ChangeLog files (a
| centralized one is obviously not viable)

5) Anyone who really cares can use the excludes list, and check the
ChangeLogs on the web when they need them.

-- 
Ciaran McCreesh : Gentoo Developer (King of all Londinium)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Andrej Kacian]

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

On Sun, 01 Jan 2006 21:35:34 +0100
Francesco Riosa <vivo@gentoo.org> wrote:

> Thoughts ? It's doable in some way ?

Whatever way will get chosen, a full copy of every ChangeLog should be made
available somewhere (somewhere other than CVS Attic), for future reference.

Kind regards,
-- 
Andrej "Ticho" Kacian <ticho at gentoo dot org>
Gentoo Linux Developer - net-mail, antivirus, sound, x86

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Andrej Kacian]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On Sun, 1 Jan 2006 22:55:19 +0000
Ciaran McCreesh <ciaranm@gentoo.org> wrote:

> On Sun, 01 Jan 2006 21:35:34 +0100 Francesco Riosa <vivo@gentoo.org>
> wrote:
> | That said I can see only two ways to reduce the ChangeLog files (a
> | centralized one is obviously not viable)
> 
> 5) Anyone who really cares can use the excludes list, and check the
> ChangeLogs on the web when they need them.

Most users care, they just don't know about it.

Kind regards,
-- 
Andrej "Ticho" Kacian <ticho at gentoo dot org>
Gentoo Linux Developer - net-mail, antivirus, sound, x86

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

Andrej Kacian wrote:
> On Sun, 01 Jan 2006 21:35:34 +0100
> Francesco Riosa <vivo@gentoo.org> wrote:
> 
>> Thoughts ? It's doable in some way ?
> 
> Whatever way will get chosen, a full copy of every ChangeLog should be made
> available somewhere (somewhere other than CVS Attic), for future reference.
> 
> Kind regards,

I was thinking in a package that install in /usr/share/changelogs and/or
a web access, infra would be happy to add changelogs.gentoo.org domain .

/me runs
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

Grobian wrote:
> On 01-01-2006 21:35:34 +0100, Francesco Riosa wrote with possible deletions:
>> The information contained in the ChangeLogs is essential, and it must be
>> kept, but, force the users to download all that data it's not optimal.
>>
>> That said I can see only two ways to reduce the ChangeLog files (a
>> centralized one is obviously not viable)
>>
>> 1) bzip2 them in some way.
>> 2) "rotate" Changelogs, keeping only the last changes, until a size
> 
> or
> 3) remove entries for non-existing ebuilds
> 4) compress Changelog entries where possible

A combination of 2) and 3) looks good.
Maybe with some tool to automate it but that left to developer the final
decision on what cut or not.


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

Ciaran McCreesh wrote:
> On Sun, 01 Jan 2006 21:35:34 +0100 Francesco Riosa <vivo@gentoo.org>
> wrote:
> | That said I can see only two ways to reduce the ChangeLog files (a
> | centralized one is obviously not viable)
> 
> 5) Anyone who really cares can use the excludes list, and check the
> ChangeLogs on the web when they need them.
> 
yes but this excludes totally the ChangeLogs. Having the latest changes
handy may be useful.

Simply trying to equilibrate benefits and disadvantages,
for desktop user that care on changelogs only when emerge fail
and the network administrator that has a local rsync mirror and could
not care less of some additional MB transferred.


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Paweł Madej]

Francesco Riosa wrote:
> yes but this excludes totally the ChangeLogs. Having the latest changes
> handy may be useful.
> 
> Simply trying to equilibrate benefits and disadvantages,
> for desktop user that care on changelogs only when emerge fail
> and the network administrator that has a local rsync mirror and could
> not care less of some additional MB transferred.

As a common user with ADSL 256kbps all additional data downloaded via 
rsync causes longer wait for syncing. If I want to see a changelog i go 
to packages.gentoo.org and read it. As i got Gentoo for about 10 months 
I don't remember if I read Changelog via bash.

It is quite important thing at this time as Gentoo Dev's plan to add 
news messages adding to portage which also will be synced (as I recon).

Greets
Pawel

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Pawe?? Madej <linux@quanteam.info> wrote:
> As a common user with ADSL 256kbps all additional data downloaded via
> rsync causes longer wait for syncing. If I want to see a changelog i
> go to packages.gentoo.org and read it. As i got Gentoo for about 10
> months I don't remember if I read Changelog via bash.

I strongly disagree. Reading changelogs on the web means browsing for
them. This is much slower for me than a 'less
/usr/portage/foo/bar/Changelog'.

I rather go with ciaranm and tell users to use rsync_exclude if they
don't want changelogs. Taking away choice without an option is a bad
thing (tm).

Regards and a happy new year,
Matti
-- 
Marriage is the only adventure open to the cowardly.
		-- Voltaire

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

Matti Bickel wrote:
> Pawe?? Madej <linux@quanteam.info> wrote:
>> As a common user with ADSL 256kbps all additional data downloaded via
>> rsync causes longer wait for syncing. If I want to see a changelog i
>> go to packages.gentoo.org and read it. As i got Gentoo for about 10
>> months I don't remember if I read Changelog via bash.
> 
> I strongly disagree. Reading changelogs on the web means browsing for
> them. This is much slower for me than a 'less
> /usr/portage/foo/bar/Changelog'.
> 
> I rather go with ciaranm and tell users to use rsync_exclude if they
> don't want changelogs. Taking away choice without an option is a bad
> thing (tm).
> 
> Regards and a happy new year,
> Matti

A (rethoric) question:
Do you read every time the whole ChangeLog ?

If not, generally, how much of it ?

Regards,
Francesco R.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

Francesco Riosa <vivo@gentoo.org> wrote:
> Matti Bickel wrote:
> > Pawe?? Madej <linux@quanteam.info> wrote:
> >> As a common user with ADSL 256kbps all additional data downloaded via
> >> rsync causes longer wait for syncing. If I want to see a changelog i
> >> go to packages.gentoo.org and read it. As i got Gentoo for about 10
> >> months I don't remember if I read Changelog via bash.
> > 
> > I strongly disagree. Reading changelogs on the web means browsing for
> > them. This is much slower for me than a 'less
> > /usr/portage/foo/bar/Changelog'.
> > 
> > I rather go with ciaranm and tell users to use rsync_exclude if they
> > don't want changelogs. Taking away choice without an option is a bad
> > thing (tm).
> 
> A (rethoric) question:
> Do you read every time the whole ChangeLog ?
> If not, generally, how much of it ?

About 20%. I don't care about changelogs most of the time because
diffing the ebuilds is easier and you get more info.
However if there's a bug or some major upgrade, i tend to kind of read
the thing. For bug# and stuff.

Personally it's also interesting who commited a fix.

Regards,
Matti
-- 

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Henrik Brix Andersen]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Mon, Jan 02, 2006 at 05:37:48PM +0100, Francesco Riosa wrote:
> Do you read every time the whole ChangeLog ?
> 
> If not, generally, how much of it ?

I often find myself grepping through the entire ChangeLog too see when
and who introduced a particular change in an ebuild - but I normally
do this in my CVS check-out.

I'm also for telling the users to rsync exclude the ChangeLogs if they
don't want them instead of getting rid of them or crippling them.

Regards,
Brix
-- 
Henrik Brix Andersen <brix@gentoo.org>
Gentoo Metadistribution | Mobile computing herd

[-- Attachment #2: Type: application/pgp-signature, Size: 211 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Lance Albertson]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

Henrik Brix Andersen wrote:
> On Mon, Jan 02, 2006 at 05:37:48PM +0100, Francesco Riosa wrote:
> 
>>Do you read every time the whole ChangeLog ?
>>
>>If not, generally, how much of it ?
> 
> 
> I often find myself grepping through the entire ChangeLog too see when
> and who introduced a particular change in an ebuild - but I normally
> do this in my CVS check-out.
> 
> I'm also for telling the users to rsync exclude the ChangeLogs if they
> don't want them instead of getting rid of them or crippling them.

I don't think that's really a solution. That's just a way to minimize
what they get. If you really want to look at a solution, you should
reduce whats actually in the changelog. All the changelogs are in CVS,
and if someone wanted to go look back at any changes, they can either
look at viewcvs or (hopefully soon), grab it via anoncvs. I bet you >80%
of things in the changelogs are for things that are in the attic. I say
we need a way to either have echangelog (or another script) to clean out
changelog entries for things that are in the attic (and make sense to
take out). Maybe another option would be to remove any 'version bump'
type entries that are old as well.

I just don't see the point of keeping changelog entries for stuff that
isn't even viewable in the tree anymore. We have CVS, the record will be
there, lets use it. We can't cater to every user out there, but come up
with a solid solution that makes sense and still gives them some form of
ability to look back.

-- 
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key:  <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1  4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

On Mon, 2006-01-02 at 11:25 -0600, Lance Albertson wrote: 
> > I'm also for telling the users to rsync exclude the ChangeLogs if they
> > don't want them instead of getting rid of them or crippling them.
> 
> I don't think that's really a solution. That's just a way to minimize
> what they get. If you really want to look at a solution, you should
> reduce whats actually in the changelog. All the changelogs are in CVS,
> and if someone wanted to go look back at any changes, they can either
> look at viewcvs or (hopefully soon), grab it via anoncvs. I bet you >80%
> of things in the changelogs are for things that are in the attic. I say
> we need a way to either have echangelog (or another script) to clean out
> changelog entries for things that are in the attic (and make sense to
> take out). Maybe another option would be to remove any 'version bump'
> type entries that are old as well.

OK.  I keep seeing this argument about ChangeLog stuff for ebuilds in
the Attic and I just think people might not be thinking it totally
trough.  For example, I made changes to the vmware-workstation ebuilds
to force group membership a while back because of a security bug.
However, there's been another security bug since, so those changes were
made on ebuilds now in Attic, but the change is still valid in the
current ebuilds.

I don't see a problem with removing version bump and stabilization
messages, but everything else should stay in the ChangeLog for as long
as the package is still around.

> I just don't see the point of keeping changelog entries for stuff that
> isn't even viewable in the tree anymore. We have CVS, the record will be
> there, lets use it. We can't cater to every user out there, but come up
> with a solid solution that makes sense and still gives them some form of
> ability to look back.

Because as I stated before, there are many times where the ChangeLog
entry for an older ebuild applies to the newer ones.  This is especially
true when ebuilds are simply copied for version bumps.  Removing this
information removes a lot of data.  Forcing users/developers to use CVS
makes it more of a hassle than the gains will give us.

Again, I think this is another case of us doing things that aren't
necessary rather than educating users.

I wouldn't have a problem with seeing ChangeLog in a default
RSYNC_EXCLUDES with a nice comment explaining how to get the ChangeLog
files.  This way we are removing the problem by default, educating
users, and still not removing any data or options for our users and
developers.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Lance Albertson]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

Chris Gianelloni wrote:
> On Mon, 2006-01-02 at 11:25 -0600, Lance Albertson wrote: 
> 
>>>I'm also for telling the users to rsync exclude the ChangeLogs if they
>>>don't want them instead of getting rid of them or crippling them.
>>
>>I don't think that's really a solution. That's just a way to minimize
>>what they get. If you really want to look at a solution, you should
>>reduce whats actually in the changelog. All the changelogs are in CVS,
>>and if someone wanted to go look back at any changes, they can either
>>look at viewcvs or (hopefully soon), grab it via anoncvs. I bet you >80%
>>of things in the changelogs are for things that are in the attic. I say
>>we need a way to either have echangelog (or another script) to clean out
>>changelog entries for things that are in the attic (and make sense to
>>take out). Maybe another option would be to remove any 'version bump'
>>type entries that are old as well.
> 
> 
> OK.  I keep seeing this argument about ChangeLog stuff for ebuilds in
> the Attic and I just think people might not be thinking it totally
> trough.  For example, I made changes to the vmware-workstation ebuilds
> to force group membership a while back because of a security bug.
> However, there's been another security bug since, so those changes were
> made on ebuilds now in Attic, but the change is still valid in the
> current ebuilds.
> 
> I don't see a problem with removing version bump and stabilization
> messages, but everything else should stay in the ChangeLog for as long
> as the package is still around.

See the part that I said "make sense to take out". What you described is
a perfect example of something that needs to stay. All I'm getting at is
that minor things that the package maintainer thinks isn't important to
keep in the active changelog, should be able to be removed. I'm not
implying to blindly remove changelog entries solely based on the fact
that those ebuilds are in the attic. Common sense would play here.

-- 
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key:  <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1  4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Peter Volkov (pva)]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Вск, 2006-01-01 at 21:35 +0100, Francesco Riosa wrote:
> We have currently 10371 ChangeLog files, > 25 MB totally .
> 1365 == 13% with size >= 4096 B, 12 MB totally
> 
> rsync from "emerge --sync" has "--whole-file" between its options, that
> mean transfer the whole file if changed.

> 2) "rotate" Changelogs, keeping only the last changes, until a size
>    of 4000 or [choose a preferred size here] bytes.
>    This would save only about 7Mb of data (max size < 4096).

ChangeLog is the only source of information on upgrade reasons and IMO
users are interested only in recent information there. Another
suggestion is to combine rotate like and exclude list solutions.

When ChangeLog size reaches above maximum allowed size echangelog should
create ChangeLog.old file and copy tail of ChangeLog there, in order to
keep ChangeLog size. Then using excludes list one may drop really old
entries in ChangeLog.old, whereas keeping recent information in portage
tree.

Peter.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Paweł Madej]

Ok then i suggest to add such input to Gentoo Manual that users not 
interested in reading changelogs could mask them by rsync_excludes 
*/*/ChangeLog

Greets
Pawel Madej
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 3418 bytes --]

Ok, last shoot, then let put this stuff to sleep.
Description of the attachment at the end:

Chris Gianelloni wrote:
> On Mon, 2006-01-02 at 11:25 -0600, Lance Albertson wrote: 
>>> I'm also for telling the users to rsync exclude the ChangeLogs if they
>>> don't want them instead of getting rid of them or crippling them.
>> I don't think that's really a solution. That's just a way to minimize
>> what they get. If you really want to look at a solution, you should
>> reduce whats actually in the changelog. All the changelogs are in CVS,
>> and if someone wanted to go look back at any changes, they can either
>> look at viewcvs or (hopefully soon), grab it via anoncvs. I bet you >80%
>> of things in the changelogs are for things that are in the attic. I say
>> we need a way to either have echangelog (or another script) to clean out
>> changelog entries for things that are in the attic (and make sense to
>> take out). Maybe another option would be to remove any 'version bump'
>> type entries that are old as well.
> 
> OK.  I keep seeing this argument about ChangeLog stuff for ebuilds in
> the Attic and I just think people might not be thinking it totally
> trough.  For example, I made changes to the vmware-workstation ebuilds
> to force group membership a while back because of a security bug.
> However, there's been another security bug since, so those changes were
> made on ebuilds now in Attic, but the change is still valid in the
> current ebuilds.

Could ignore the revision (-r*) address this issue ?

> 
> I don't see a problem with removing version bump and stabilization
> messages, but everything else should stay in the ChangeLog for as long
> as the package is still around.
> 
>> I just don't see the point of keeping changelog entries for stuff that
>> isn't even viewable in the tree anymore. We have CVS, the record will be
>> there, lets use it. We can't cater to every user out there, but come up
>> with a solid solution that makes sense and still gives them some form of
>> ability to look back.
> 
> Because as I stated before, there are many times where the ChangeLog
> entry for an older ebuild applies to the newer ones.  This is especially
> true when ebuilds are simply copied for version bumps.  Removing this
> information removes a lot of data.  Forcing users/developers to use CVS
> makes it more of a hassle than the gains will give us.

same as before

> 
> Again, I think this is another case of us doing things that aren't
> necessary rather than educating users.
> 
> I wouldn't have a problem with seeing ChangeLog in a default
> RSYNC_EXCLUDES with a nice comment explaining how to get the ChangeLog
> files.  This way we are removing the problem by default, educating
> users, and still not removing any data or options for our users and
> developers.
> 

IMHO RSYNC_EXCLUDES it's a bad option, in it's extended version, exclude
foo/bar could bring to every sort of problems, difficult to address (but
I've never tryed that ;-).

The attached bash script attempt to purge the ChangeLog (it broke a bit
but could be solved)

It clean out all entries without corresponding files (*.ebuild files/*)
It left in place date of addition of euilds and developer that changed
something

The test is moved in two new files Changelog.{new,old}

be careful, it leave a ".tmp__revisions" in place

The size of the resulting ChangeLog may be less han 50% than the original.


[-- Attachment #2: elogcleaner --]
[-- Type: text/plain, Size: 2955 bytes --]

#! /bin/bash

function exit_no_changelog() {
    echo "No ChangeLog found"
    exit 1
}


function main() {

    local ChangeLog="ChangeLog"
    local ChangeLogNew="ChangeLog.new"
    local ChangeLogOld="ChangeLog.old"
    local inside=0
    local buffer

    CWD="."
    pushd "${CVD}" &> /dev/null

    [[ -f "${ChangeLog}" ]] || exit_no_changelog

	# create a list of each version, remove gentoo specific revisions
    #grep ^\* ChangeLog \
    #    | sed -e 's/^*//;s/ .*//;s/-r[[:digit:]][[:digit:]]*//' \
    #    | sort \
    #    | uniq \
    #    > .tmp__versions

	# create a list of each version
    grep ^\* "${ChangeLog}" \
        | sed -e 's/^*//;s/ .*//' \
        | sort \
        | uniq \
        > .tmp__revisions

    # keep track of each modify and author
    # FIXME this is a sed job
    local entry_pattern='[[:digit:]][[:digit:]]\ [[:alnum:]][[:alnum:]][[:alnum:]]\ 20[[:digit:]][[:digit:]]; '
    echo -n "" > .tmp__revisions
    while read line ; do
    case "${line}" in
            ${entry_pattern}*:*)
            echo "${line}" >> .tmp__revisions
            ;;
        ${entry_pattern}*)
            echo -n ${line} >> .tmp__revisions
            inside=1
            ;;
        *:*)
            if [[ inside -eq 1 ]] ; then
                echo ${line} >> .tmp__revisions
                inside=0
            fi
            ;;
        *)
            [[ inside -eq 1 ]] && echo -n ${line} >> .tmp__revisions
            ;;
        esac
    done < "${ChangeLog}"
    sed -i -e 's/:.*$/:/' .tmp__revisions

    # mark revisions to keep
    local sed_prg f
	    for e in *.ebuild files/* ; do
        buffer="${e%.ebuild}"
        buffer="${buffer%-r*}"
        sed_prg="${sed_prg};s!^\(.*${buffer}.*$\)!K\1!"
    done

    sed -i -e "${sed_prg}" \
        -e 's/^KK*/K/' \
        -e 's/^\([^K]\)/D\1/' \
        .tmp__revisions

    # attempt to write the purged ChangeLog
    echo -n "" > "${ChangeLogNew}"
    echo -n "" > "${ChangeLogOld}"
    local cl="${ChangeLogNew}"
    inside=0
    while read line ; do
        case "${line}" in
            \#*)
                echo "${line}" >> "${ChangeLogNew}"
                ;;
            \**)
                echo "${line}" >> "${ChangeLogNew}"
                echo "${line}" >> "${ChangeLogOld}"
                ;;
            ${entry_pattern}*)
                if $(grep -q "D${line%%:*}" .tmp__revisions) ; then
                    cl="${ChangeLogOld}"
                    echo "  ${line%%>*}>:" >> "${ChangeLogNew}"
                else
                    cl="${ChangeLogNew}"
                fi
                echo "  ${line}" >> "${cl}"
                ;;
            *)
                echo "  ${line}" >> "${cl}"
                ;;
        esac
    done < "${ChangeLog}"

    # further cleaning
    sed -i -e 's/[ \t]*$//' "${ChangeLogNew}"
    sed -i -e '$!N; /^  .. ... ....; \(.*\)\n  .. ... ....; \1$/!P; D' "${ChangeLogNew}"


    popd &> /dev/null

}


main

Re: [gentoo-dev] ChangeLogs and rsync time

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lance Albertson wrote:
| See the part that I said "make sense to take out". What you described is
| a perfect example of something that needs to stay. All I'm getting at is
| that minor things that the package maintainer thinks isn't important to
| keep in the active changelog, should be able to be removed. I'm not
| implying to blindly remove changelog entries solely based on the fact
| that those ebuilds are in the attic. Common sense would play here.

This is a nice idea in theory, but that means maintainers will have to
sit around reading the whole ChangeLog and editing it. That directly
takes away from real development time, for how much benefit?

The tradeoff isn't worth it to me.

Thanks,
Donnie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDuguVXVaO67S1rtsRAsiBAKD0aLpFFyDmv26vLbEA9Hh4X8UGGgCg6GIT
EPVU6pfW2IiFPDNGrtJX45g=
=zVFO
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On Mon, 2006-01-02 at 13:20 -0600, Lance Albertson wrote:
> > I don't see a problem with removing version bump and stabilization
> > messages, but everything else should stay in the ChangeLog for as long
> > as the package is still around.
> 
> See the part that I said "make sense to take out". What you described is
> a perfect example of something that needs to stay. All I'm getting at is
> that minor things that the package maintainer thinks isn't important to
> keep in the active changelog, should be able to be removed. I'm not
> implying to blindly remove changelog entries solely based on the fact
> that those ebuilds are in the attic. Common sense would play here.

So you have now taken what was described as an automatic solution to
reduce size into a manual process, reducing usability of the ChangeLog
and increasing workload for every ebuild developer.

I'm sorry, but I still think the idea of simply RSYNC_EXCLUDEing the
ChangeLog by default would be a much better solution.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1569 bytes --]

On Tue, 2006-01-03 at 00:35 +0300, Peter Volkov (pva) wrote:
> On Вск, 2006-01-01 at 21:35 +0100, Francesco Riosa wrote:
> > We have currently 10371 ChangeLog files, > 25 MB totally .
> > 1365 == 13% with size >= 4096 B, 12 MB totally
> > 
> > rsync from "emerge --sync" has "--whole-file" between its options, that
> > mean transfer the whole file if changed.
> 
> > 2) "rotate" Changelogs, keeping only the last changes, until a size
> >    of 4000 or [choose a preferred size here] bytes.
> >    This would save only about 7Mb of data (max size < 4096).
> 
> ChangeLog is the only source of information on upgrade reasons and IMO
> users are interested only in recent information there. Another
> suggestion is to combine rotate like and exclude list solutions.
> 
> When ChangeLog size reaches above maximum allowed size echangelog should
> create ChangeLog.old file and copy tail of ChangeLog there, in order to
> keep ChangeLog size. Then using excludes list one may drop really old
> entries in ChangeLog.old, whereas keeping recent information in portage
> tree.

I could see this working well without removing the information from CVS
or the tree.  We would then add RSYNC_EXCLUDE on ChangeLog.old instead
of ChangeLog.  I wouldn't have a problem with this, and it would still
be benificial.  The only question left to be answered is how much space
would actually be saved to make this extra load on CVS worth it.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re[2]: [gentoo-dev] ChangeLogs and rsync time

[Jakub Moc]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]


3.1.2006, 12:47:27, Chris Gianelloni wrote:


> So you have now taken what was described as an automatic solution to
> reduce size into a manual process, reducing usability of the ChangeLog
> and increasing workload for every ebuild developer.

> I'm sorry, but I still think the idea of simply RSYNC_EXCLUDEing the
> ChangeLog by default would be a much better solution.

+1; I frequently find even pretty old changelog entries useful (even the
'stable on foo' ones are useful, if you need to find out who keyworded the
thing stable despite the fact that it's severely broken).

As for RSYNC_EXCLUDE by default, that's not something that should be
considered until GLEP 42 or an equivalent solution gets implemented.


-- 
Best regards,

 Jakub Moc
 mailto:jakub@gentoo.org
 GPG signature: http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
 Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95  B30F 8717 D5FD CEBA 3D9E

 ... still no signature ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 183 bytes --]

Re: [gentoo-dev] ChangeLogs and rsync time

[Paweł Madej]

Chris Gianelloni wrote:
> On Mon, 2006-01-02 at 13:20 -0600, Lance Albertson wrote:
> 
> I'm sorry, but I still think the idea of simply RSYNC_EXCLUDEing the
> ChangeLog by default would be a much better solution.
> 

I didn't know before that it is possible, but after some reading man 
rsync I've added */*/ChangeLog line to my RSYNC_EXCLUDEs file and it 
resolves problems with big downloads. I think that for common users it 
will be good idea, but you will have to inform them that sth like this 
is set and how to make full syncing availible.


Greets
Pawel Madej
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Mike Frysinger]

On Tuesday 03 January 2006 09:29, Paweł Madej wrote:
> Chris Gianelloni wrote:
> > On Mon, 2006-01-02 at 13:20 -0600, Lance Albertson wrote:
> >
> > I'm sorry, but I still think the idea of simply RSYNC_EXCLUDEing the
> > ChangeLog by default would be a much better solution.
>
> I didn't know before that it is possible, but after some reading man
> rsync I've added */*/ChangeLog line to my RSYNC_EXCLUDEs file

you could just do 'ChangeLog'
-mike

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Francesco Riosa]

Chris Gianelloni wrote:
> On Tue, 2006-01-03 at 00:35 +0300, Peter Volkov (pva) wrote:
>> On Вск, 2006-01-01 at 21:35 +0100, Francesco Riosa wrote:
>>> We have currently 10371 ChangeLog files, > 25 MB totally .
>>> 1365 == 13% with size >= 4096 B, 12 MB totally
[...]
> be benificial.  The only question left to be answered is how much space
> would actually be saved to make this extra load on CVS worth it.
> 

At the extreme of all ChangeLog moved in old there are 10371 more files
in the tree, 25 MB less to check and to transfer (in case all the
Changelogs are modified at the same time).

In a more general scenario, only about 1300 Changelogs need to be
transferred in ChangeLog.old, Leaving The total size of the Changelogs
to about 18 MB instead of 25, with about 6 MB in ChangeLog.old (assuming
no duplicated data).
The Total number of ChangeLog* raise to 10371[norm] + 1365[.old] = 11736

There is a overhead for user that not use rsync exclude and a noticeable
gain for user that instead exclude ChangeLog.old .


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] ChangeLogs and rsync time

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Sun, Jan 01, 2006 at 10:48:18PM +0100, Grobian wrote:
> On 01-01-2006 21:35:34 +0100, Francesco Riosa wrote with possible deletions:
> > 1) bzip2 them in some way.
> 4) compress Changelog entries where possible

Anyone gathered transfer stats for rsync without --whole-file?
compression won't play nice with rsync's delta compression...
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]