[gentoo-dev] [Council] ChangeLog generation within Gentoo

[gentoo-dev] [Council] ChangeLog generation within Gentoo

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 2251 bytes --]

All,

As decided by the Council in its last meeting at the beginning of this
month, ChangeLogs will not be generated from VCS commit messages,
because the Council wants to keep the ability to edit generated
ChangeLog messages [1].

As a result, the Council took this up with the Portage team, to make
repoman update the ChangeLog file upon `repoman commit'.

Because some of the reasons put forward for autogenerating ChangeLogs
was the actual speed of making a commit, repoman does not call
echangelog, but instead uses its information about the current state of
the VCS directory it already collected to do its QA checks to write a
ChangeLog entry directly from Python (in which repoman is written).

To facilitate those developers who have echangelog, or their own scripts
that do the same, hardwired into their systems, repoman commit will by
default *NOT* update the ChangeLog if it is already modified.  This
allows to run echangelog followed by repoman commit, without getting a
double ChangeLog entry.

However, this also allows to do all kinds of other actions to the
ChangeLog file, without actually adding an entry for the current change
being committed, as we've already seen in practice.
The Council would like to remind developers that it is still a
requirement that all actions are documented in the ChangeLog and that it
is hence the responsibility of the committing developer to make sure
this requirement is met.

Repoman in the most recent Portage versions in the tree, has an
--echangelog flag, that takes the argument `y', `n' or `force'.  The
default for this option can be set to yes or no, and is controlled by
metadata/layout.conf.  For the `gentoo' (gentoo-x86) repository, this is
set to true, which results in the previously described behaviour.

When the ChangeLog is modified, but a new entry should still be written
by repoman, --echangelog=force will accomplish this.  Obviously, passing
--echangelog=n makes repoman skip writing a ChangeLog entry at all.


The Council hopes to have informed all developers sufficiently through
this message.


[1] http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Bruno]

On Wed, 26 October 2011 Fabian Groffen <grobian@gentoo.org> wrote:
> However, this also allows to do all kinds of other actions to the
> ChangeLog file, without actually adding an entry for the current change
> being committed, as we've already seen in practice.
> The Council would like to remind developers that it is still a
> requirement that all actions are documented in the ChangeLog and that it
> is hence the responsibility of the committing developer to make sure
> this requirement is met.

Is there some guideline about old entries in the ChangeLog?

Over the past months ChangeLogs represent a big part of the tree, some
of them being pretty big and going back many changes (hundreds of them)
and years (even for actively maintained ebuilds).

In order to not bloat the tree I would like to see old entries purged
when there are more than 25-50 of them, especially if they refer to
ebuilds gone since more than 3-6 months.
Someone in need for long gone ebuild would have to look at VCS anyhow, so
looking at ChangeLog/history over there would seem logical.

On a compressed tree (squashfs) dropping all ChangeLogs reduces size from
~55MiB to around 35MiB which is quite a lot!

Bruno

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On 27 October 2011 06:33, Bruno <bonbons67@internet.lu> wrote:

>
> Is there some guideline about old entries in the ChangeLog?
>
> Over the past months ChangeLogs represent a big part of the tree, some
> of them being pretty big and going back many changes (hundreds of them)
> and years (even for actively maintained ebuilds).
>
>
A lot of the older changelog entries also tend to be less normal, and some
long changelogs are so a-typical its virtually impossible to parse them with
machine code.

I know its probably not a big priority for most people, but if changelogs
can retain consistency so that the only part which is inconsistent is the
messages themselves.

I saw the --echangelog parameter turn up sometime this week and I haven't
played with it for fear of subtle inconsistencies with the form produced by
the echangelog client.

( I've been hacking on various tools to parse/normalise changelogs myself,
but it got overwhelming and I got sidetracked )

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 1569 bytes --]

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Rich Freeman]

On Wed, Oct 26, 2011 at 1:56 PM, Kent Fredric <kentfredric@gmail.com> wrote:
> On 27 October 2011 06:33, Bruno <bonbons67@internet.lu> wrote:
>>
>> Is there some guideline about old entries in the ChangeLog?
>>
>> Over the past months ChangeLogs represent a big part of the tree, some
>> of them being pretty big and going back many changes (hundreds of them)
>> and years (even for actively maintained ebuilds).
>>
>
> A lot of the older changelog entries also tend to be less normal, and some
> long changelogs are so a-typical its virtually impossible to parse them with
> machine code.

Well, if the desire to trim changelogs is generally agreed upon we
could always just count the lines and post a top-100 list or something
and let package maintainers go in and truncate things as seems bet to
them, with the guideline to keep the file intact up to a year before
the last commit.  Eventually the files will be cleaned up.

Rich

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1706 bytes --]

El mié, 26-10-2011 a las 19:33 +0200, Bruno escribió:
> On Wed, 26 October 2011 Fabian Groffen <grobian@gentoo.org> wrote:
> > However, this also allows to do all kinds of other actions to the
> > ChangeLog file, without actually adding an entry for the current change
> > being committed, as we've already seen in practice.
> > The Council would like to remind developers that it is still a
> > requirement that all actions are documented in the ChangeLog and that it
> > is hence the responsibility of the committing developer to make sure
> > this requirement is met.
> 
> Is there some guideline about old entries in the ChangeLog?
> 
> Over the past months ChangeLogs represent a big part of the tree, some
> of them being pretty big and going back many changes (hundreds of them)
> and years (even for actively maintained ebuilds).
> 
> In order to not bloat the tree I would like to see old entries purged
> when there are more than 25-50 of them, especially if they refer to
> ebuilds gone since more than 3-6 months.
> Someone in need for long gone ebuild would have to look at VCS anyhow, so
> looking at ChangeLog/history over there would seem logical.
> 
> On a compressed tree (squashfs) dropping all ChangeLogs reduces size from
> ~55MiB to around 35MiB which is quite a lot!
> 
> Bruno
> 
> 

Personally, I want to have full ChangeLog easily accessible, I remember
to need to look for really old entries when becoming a new maintainer
for an old package previously maintained by others.

What I don't know is the reasons for not compressing ChangeLogs by
default (well, I don't have a compressed tree, this probably won't be a
gain for people using it)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Bruno]

On Wed, 26 October 2011 Pacho Ramos <pacho@gentoo.org> wrote:
> El mié, 26-10-2011 a las 19:33 +0200, Bruno escribió:
> > On Wed, 26 October 2011 Fabian Groffen <grobian@gentoo.org> wrote:
> > > However, this also allows to do all kinds of other actions to the
> > > ChangeLog file, without actually adding an entry for the current change
> > > being committed, as we've already seen in practice.
> > > The Council would like to remind developers that it is still a
> > > requirement that all actions are documented in the ChangeLog and that it
> > > is hence the responsibility of the committing developer to make sure
> > > this requirement is met.
> > 
> > Is there some guideline about old entries in the ChangeLog?
> > 
> > Over the past months ChangeLogs represent a big part of the tree, some
> > of them being pretty big and going back many changes (hundreds of them)
> > and years (even for actively maintained ebuilds).
> > 
> > In order to not bloat the tree I would like to see old entries purged
> > when there are more than 25-50 of them, especially if they refer to
> > ebuilds gone since more than 3-6 months.
> > Someone in need for long gone ebuild would have to look at VCS anyhow, so
> > looking at ChangeLog/history over there would seem logical.
> > 
> > On a compressed tree (squashfs) dropping all ChangeLogs reduces size from
> > ~55MiB to around 35MiB which is quite a lot!
> 
> Personally, I want to have full ChangeLog easily accessible, I remember
> to need to look for really old entries when becoming a new maintainer
> for an old package previously maintained by others.

That seems fair, though I guess old ebuilds are needed as well in that case
so the not-trimmed ChangeLog probably rather should be a feature of VCS GUI
that would take all the changes to ChangeLog and assemble all the additions
ignoring the removed lines, possibly connecting that to the history of the
individual ebuilds.

> What I don't know is the reasons for not compressing ChangeLogs by
> default (well, I don't have a compressed tree, this probably won't be a
> gain for people using it)

Compressing the ChangeLogs in the tree would not help rsync as the beginning
of the file changes and thus the compressed binary result is all but a stream
to which data got appended.

Bruno

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On 26-10-2011 14:02:12 -0400, Rich Freeman wrote:
> Well, if the desire to trim changelogs is generally agreed upon we
> could always just count the lines and post a top-100 list or something
> and let package maintainers go in and truncate things as seems bet to
> them, with the guideline to keep the file intact up to a year before
> the last commit.  Eventually the files will be cleaned up.

Don't you think it's much more sensical to remove all entries for
ebuilds that are no longer in the tree then?


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: [Council] ChangeLog generation within Gentoo

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

On Wed, 26 Oct 2011 19:33:56 +0200
Bruno <bonbons67@internet.lu> wrote:

> Is there some guideline about old entries in the ChangeLog?
> 
> Over the past months ChangeLogs represent a big part of the tree, some
> of them being pretty big and going back many changes (hundreds of them)
> and years (even for actively maintained ebuilds).
> 
> In order to not bloat the tree I would like to see old entries purged
> when there are more than 25-50 of them, especially if they refer to
> ebuilds gone since more than 3-6 months.
> Someone in need for long gone ebuild would have to look at VCS anyhow, so
> looking at ChangeLog/history over there would seem logical.

No, it's a pain in the ass to have to go pull up that info through cvs and
impossible when you're offline.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [Council] ChangeLog generation within Gentoo

[Duncan]

Fabian Groffen posted on Wed, 26 Oct 2011 23:00:22 +0200 as excerpted:

> On 26-10-2011 14:02:12 -0400, Rich Freeman wrote:
>> Well, if the desire to trim changelogs is generally agreed upon we
>> could always just count the lines and post a top-100 list or something
>> and let package maintainers go in and truncate things as seems bet to
>> them, with the guideline to keep the file intact up to a year before
>> the last commit.  Eventually the files will be cleaned up.
> 
> Don't you think it's much more sensical to remove all entries for
> ebuilds that are no longer in the tree then?

1) Given the irregularity of older entries, that could be difficult to 
automate, tho it could be done going forward, once a log has been 
manually trimmed once.

2) I'd argue for keeping upstream version commits and removals, so at 
minimum, the dates they were in the tree can be tracked.  (FWIW, I often 
find myself checking this information when helping someone try to build 
something half-modern on a stale distro like CentOS 5, for instance.)  It 
could be argued that this is a reasonably important minimal historical 
record.

But all stabilizations, -rX bumps, and changes other than upstream 
version addition and removal, indeed, removing those entries say three 
months minimum after the ebuilds are no longer in the tree does make 
sense.  (And as a bonus, such removal would make the historical record 
above, addition and removal of older upstream versions, far easier to 
read, since it'd be all that's left for versions already out-of-tree. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: [Council] ChangeLog generation within Gentoo

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

On 27-10-2011 03:28:33 +0000, Duncan wrote:
> Fabian Groffen posted on Wed, 26 Oct 2011 23:00:22 +0200 as excerpted:
> > On 26-10-2011 14:02:12 -0400, Rich Freeman wrote:
> >> Well, if the desire to trim changelogs is generally agreed upon we
> >> could always just count the lines and post a top-100 list or something
> >> and let package maintainers go in and truncate things as seems bet to
> >> them, with the guideline to keep the file intact up to a year before
> >> the last commit.  Eventually the files will be cleaned up.
> > 
> > Don't you think it's much more sensical to remove all entries for
> > ebuilds that are no longer in the tree then?
> 
> 1) Given the irregularity of older entries, that could be difficult to 
> automate, tho it could be done going forward, once a log has been 
> manually trimmed once.

a) take the set of available ebuilds
b) forward scan through the ChangeLog for entries that affect any of the
   files
c) copy those entries to a new ChangeLog

Technically, you could do it on the machine that generates the rsync
image, but that brings the problem that the Manifest file gets broken,
hence an update + resign is necessary.  E.g. all developer signs are
replaced with a generic one.  Same issue when generating the ChangeLogs
from VCS with the current Manifests.

Upside of doing it on rsync0 is that the full log is stored in
sources.g.o, and the short/most relevant one is on rsync to the users.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [Council] ChangeLog generation within Gentoo

[Michael Haubenwallner]

On 10/26/11 19:33, Bruno wrote:
> In order to not bloat the tree I would like to see old entries purged
> when there are more than 25-50 of them, especially if they refer to
> ebuilds gone since more than 3-6 months.

One thing to remember:

Even if old ebuilds are gone in the tree already, they still may be
installed on users systems. As a result, 'emerge --changelog' searches
for their addition-entries in ChangeLog.

So when purging ChangeLog's really becomes necessary, we might need to
keep the addition-entries back to until a once-been-stable ebuild was
superseded by another stable ebuild more than 1 year ago - or sth. similar.

my 0.02 [€$?]

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level