[gentoo-dev] Portage repo usage survey and change evaluation

[gentoo-dev] Portage repo usage survey and change evaluation

[Robin H. Johnson]

http://goo.gl/forms/5riWkN8VMK

I've put together a quick survey about a number of potential changes to
the rsync distribution that have been bike-shedded about in various IRC
channels and the mailing lists for some time now.

Questions:
(answer them on the query, input via email will not be considered or
bikeshedded).
- do you use changelogs
- (rsync) exclude changelogs to save local disk space
- (rsync) order of changelog entries?
- Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata 


I'll report the results not sooner than 2 weeks from now, but I might
put together some intermediate results if I see anything interesting
happening.

P.S. For the crowd that wants to claim I'm depending on some non-open
infrastructure, I'm not going to go through all of the hoops to set up
some open source questionnaire infrastructure at this point. If don't
like that fact that I've used Google, simply don't answer the survey,
and feel free to complain about any results that I do end up collecting.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Rich Freeman]

On Mon, Feb 1, 2016 at 1:11 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> - Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata
>

You're missing an option to replace rsync with a git repo that has
metadata, but not the other stuff.

Of course, this already exists and no doubt many are using it.  It
just isn't official.

I'm fine with Changelogs going away entirely, in any case.  That's
what git is for.  If people want to spend time dynamically generating
them, it isn't harmful to do so.

-- 
Rich

[gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

On Mon, 1 Feb 2016 06:46:26 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Mon, Feb 1, 2016 at 1:11 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> > - Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata
> >  
> 
> You're missing an option to replace rsync with a git repo that has
> metadata, but not the other stuff.
> 
> Of course, this already exists and no doubt many are using it.  It
> just isn't official.

It *is* official. It just isn't hosted on Gentoo Infra :-P. It's
officially maintained by an official Gentoo developers who happens to
be officially on Infra too, and officially is more reliable than
Infra-provided distributions channels were. Can it be more official?!

But seriously, I don't mind moving either the scripts or the mirrors to
Gentoo infrastructure. But...

...as for the scripts, Infra so far didn't offer me hardware better
than the one donated for the scripts now. There were proposals about
distributing it, etc. But I simply don't have the time to work on
making the scripts very complex in order to make them run efficiently
in a complex environment, when simple scripts work very well so far.

I've already spent some effort cleaning them up and making
configurable. I haven't had the time to prepare proper
deployment/install script though. It's still in todo, and sometimes I
regret I don't have a backup server ready to take over whenever this
one fails (which is still more rare than Gentoo infra problems).

...and as for the mirrors, nobody has offered to host them so far.
While GitHub provides quite a lot bandwidth, for free, without having
to ask and with API that makes it trivial to create and delete
repositories as needed. If someone can provide something like this, I
don't mind mirroring to 2+ hosts.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

>>>>> On Mon, 1 Feb 2016, Robin H Johnson wrote:

> http://goo.gl/forms/5riWkN8VMK
> I've put together a quick survey about a number of potential changes
> to the rsync distribution that have been bike-shedded about in
> various IRC channels and the mailing lists for some time now.

> Questions:
> (answer them on the query, input via email will not be considered or
> bikeshedded).
> - do you use changelogs
> - (rsync) exclude changelogs to save local disk space
> - (rsync) order of changelog entries?
> - Augment/replace rsync with git repo that has thick-Manifests,
> changelogs, metadata  

> I'll report the results not sooner than 2 weeks from now, but I
> might put together some intermediate results if I see anything
> interesting happening.

Have I missed your posting the results of this? Especially, what is
the preferred ordering of ChangeLog entries?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On Mon, Feb 29, 2016 at 09:01:19AM +0100, Ulrich Mueller wrote:
> Have I missed your posting the results of this? Especially, what is
> the preferred ordering of ChangeLog entries?

I just hadn't finished putting the results into a long-term format quite
yet, but did so this afternoon:
http://dev.gentoo.org/~robbat2/201602-portage-survey/

I have included a CSV of the public answers, excludes only the last
question about contact info.

Some remarks about question #2 and #3:

Q2: Reduce local disk usage by excluding ChangeLogs?
----------------------------------------------------
It was unfortunately pointed out to me very late that my question #2 had
some confusing text:
- "No, but only if were optional (I do NOT want it, but others might)"
- "Yes, but only if it were optional (I want it, but others might NOT)"

The bracket portion of each answer was interpreted as meaning the
opposite as the start of each answer :-(.

Either way, ~60% are in favour of getting rid of changelogs.

IMO this is a BETTER goal than continuing to generate them for rsync,
and bike-shedding about what the order should be; and it provides a huge
benefit by reducing the size of rsync by 155MiB.

Q3: What order should ChangeLog entries be in?
----------------------------------------------
- 85.3% of responses either preferred newest first OR didn't care (incl
  so as long as the tools work).
- 2.9% wanted oldest first.
- NOBODY selected "I'd prefer oldest entries first, but do what is best
  for distribution"
- 11.8% said get rid of changelogs.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Patrick Lauer]

On 03/02/2016 02:32 AM, Robin H. Johnson wrote:
> On Mon, Feb 29, 2016 at 09:01:19AM +0100, Ulrich Mueller wrote:
>> Have I missed your posting the results of this? Especially, what is
>> the preferred ordering of ChangeLog entries?
> I just hadn't finished putting the results into a long-term format quite
> yet, but did so this afternoon:
> http://dev.gentoo.org/~robbat2/201602-portage-survey/
>
> I have included a CSV of the public answers, excludes only the last
> question about contact info.
>
> Some remarks about question #2 and #3:
>
> Q2: Reduce local disk usage by excluding ChangeLogs?
> ----------------------------------------------------
> It was unfortunately pointed out to me very late that my question #2 had
> some confusing text:
> - "No, but only if were optional (I do NOT want it, but others might)"
> - "Yes, but only if it were optional (I want it, but others might NOT)"
>
> The bracket portion of each answer was interpreted as meaning the
> opposite as the start of each answer :-(.
>
> Either way, ~60% are in favour of getting rid of changelogs.
Well, with those confusing answers I'd interpret it differently:

~15% are in favour of removal (see Q3)
~45% are in favour of available-but-not-default ("No but optional")
~40% are in favour of available and default

That'd be, like, 85% in favour of keeping changelogs, and about half the
people would want an option to remove them.
>
> IMO this is a BETTER goal than continuing to generate them for rsync,
> and bike-shedding about what the order should be; and it provides a huge
> benefit by reducing the size of rsync by 155MiB.
There's no bikeshedding about order (see below), and if most people are
in favour of keeping or providing optionally I don't see how removal is
in the interest of the majority - which was the reason you did this survey.
>
> Q3: What order should ChangeLog entries be in?
> ----------------------------------------------
> - 85.3% of responses either preferred newest first OR didn't care (incl
>   so as long as the tools work).
> - 2.9% wanted oldest first.
> - NOBODY selected "I'd prefer oldest entries first, but do what is best
>   for distribution"
> - 11.8% said get rid of changelogs.
>
So people want ChangeLogs in ChangeLog order. An important, but
unexpected result :)

The obvious thing to do is to continue providing ChangeLogs, in the
obvious order, and possibly document a way for users to exclude them
without breaking Manifest. I'm not sure if there's a simple/clean way to
do that except maybe providing two rsync trees ...

[gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

>>>>> On Wed, 2 Mar 2016, Robin H Johnson wrote:

> I just hadn't finished putting the results into a long-term format
> quite yet, but did so this afternoon:
> http://dev.gentoo.org/~robbat2/201602-portage-survey/

Thank you.

> Some remarks about question #2 and #3:

> Q2: Reduce local disk usage by excluding ChangeLogs?
> ----------------------------------------------------
> It was unfortunately pointed out to me very late that my question #2
> had some confusing text:
> - "No, but only if were optional (I do NOT want it, but others might)"
> - "Yes, but only if it were optional (I want it, but others might NOT)"

> The bracket portion of each answer was interpreted as meaning the
> opposite as the start of each answer :-(.

> Either way, ~60% are in favour of getting rid of changelogs.

Not sure if it can be interpreted this way. This would contradict the
results of both Q1 and Q3.

For Q1, 45 responses read ChangeLogs in some way (A1.2 to A1.5 or a
combination of them), whereas only 17 responses don't read ChangeLogs
at all (A1.1 or some combination including it). Disregarding the two
responses who at the same time read them and don't read them at all.

> IMO this is a BETTER goal than continuing to generate them for
> rsync, and bike-shedding about what the order should be; and it
> provides a huge benefit by reducing the size of rsync by 155MiB.

Hm, that's almost 40% of the total size of the tree.

   $ find /usr/portage/ -type f -name 'ChangeLog-20*' -printf '%s\n' | awk '{ s+=$1 } END { print s/1024^2 }'
   102.961

That's the old ones from CVS.

   $ find /usr/portage/ -type f -name ChangeLog -printf '%s\n' | awk '{ s+=$1 } END { print s/1024^2 }'
   52.0908

That's the new ones autogenerated from git.

How is it possible that we have 52 MiB of ChangeLog entries generated
in the 0.5 years since the git conversion, whereas we had only a total
of 103 MiB in the 13.5 years since ChangeLogs were introduced in 2002?
Certainly our commit rate hasn't increased by more than an order of
magnitude in the last half year?

> Q3: What order should ChangeLog entries be in?
> ----------------------------------------------
> - 85.3% of responses either preferred newest first OR didn't care
>   (incl so as long as the tools work).
> - 2.9% wanted oldest first.
> - NOBODY selected "I'd prefer oldest entries first, but do what is
>   best for distribution"
> - 11.8% said get rid of changelogs.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/03/16 03:50 AM, Ulrich Mueller wrote:
> How is it possible that we have 52 MiB of ChangeLog entries
> generated in the 0.5 years since the git conversion, whereas we
> had only a total of 103 MiB in the 13.5 years since ChangeLogs
> were introduced in 2002? Certainly our commit rate hasn't
> increased by more than an order of magnitude in the last half
> year?
> 

The content of a changelog entry from git is a lot bigger than it
was just from echangelog, isn't it?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlbXI4kACgkQAJxUfCtlWe2gJwEA6EDRDBa94PuopiPc7lP/GAyw
cTyWHzPznQpUGyMPXHsBAMQi+EluVkEkf6ilttjXw+XMqi//C0QyaT1jRhvRAprL
=nIHW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

>>>>> On Wed, 2 Mar 2016, Ian Stakenvicius wrote:

> On 02/03/16 03:50 AM, Ulrich Mueller wrote:
>> How is it possible that we have 52 MiB of ChangeLog entries
>> generated in the 0.5 years since the git conversion, whereas we had
>> only a total of 103 MiB in the 13.5 years since ChangeLogs were
>> introduced in 2002? Certainly our commit rate hasn't increased by
>> more than an order of magnitude in the last half year?

> The content of a changelog entry from git is a lot bigger than it
> was just from echangelog, isn't it?

Not by a factor of ten.

I've investigated a bit, and the main problem seems to be that for git
commits that extend over several directories, the commit message is
duplicated into many ChangeLog entries.

For example, the message of the initial commit 56bd759 appears in some
18000 files, which accounts for 25 MiB. Then there is commit eaaface
and its revert 1bfb585, again appearing in almost all ChangeLog files
in the tree. These account for another 9 MiB. Last example, commit
8849b09, another 2 MiB.

So about 70% of the size is caused by these 4 tree-wide commits alone.
However, there are many more examples of duplication on a smaller
scale.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[malc]

I still fail to understand the bikeshedding here - you really don't
need a git checkout to get something akin to a changelog. Use the
github API directly...

The following 1-liner could be trivially productised (maybe even parse
$PWD to set the path argument...)

curl https://api.github.com/repos/gentoo/gentoo/commits?path=app-admin/eselect
| perl -MJSON -e 'foreach $i (@{decode_json(join("",@lines=<STDIN>))})
{ print "$i->{commit}->{author}->{name} -
$i->{commit}->{author}->{date}\n\n $i->{commit}->{message}\n"; }'

Yeah - it's not quite as pretty as our current Changelog, but date,
author/committer, commit-msg etc. are all there and you can filter by
path just the same as you would with native git log...
You could parse the local $PORTDIR/metadata/timestamp* and add an
'until' param to the URL to filter commits beyond where a user has
rsync'd up to...

Cheers,
malc.


On Wed, Mar 2, 2016 at 6:14 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Wed, 2 Mar 2016, Ian Stakenvicius wrote:
>
>> On 02/03/16 03:50 AM, Ulrich Mueller wrote:
>>> How is it possible that we have 52 MiB of ChangeLog entries
>>> generated in the 0.5 years since the git conversion, whereas we had
>>> only a total of 103 MiB in the 13.5 years since ChangeLogs were
>>> introduced in 2002? Certainly our commit rate hasn't increased by
>>> more than an order of magnitude in the last half year?
>
>> The content of a changelog entry from git is a lot bigger than it
>> was just from echangelog, isn't it?
>
> Not by a factor of ten.
>
> I've investigated a bit, and the main problem seems to be that for git
> commits that extend over several directories, the commit message is
> duplicated into many ChangeLog entries.
>
> For example, the message of the initial commit 56bd759 appears in some
> 18000 files, which accounts for 25 MiB. Then there is commit eaaface
> and its revert 1bfb585, again appearing in almost all ChangeLog files
> in the tree. These account for another 9 MiB. Last example, commit
> 8849b09, another 2 MiB.
>
> So about 70% of the size is caused by these 4 tree-wide commits alone.
> However, there are many more examples of duplication on a smaller
> scale.
>
> Ulrich

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Rich Freeman]

On Wed, Mar 2, 2016 at 2:48 PM, malc <mlashley@gmail.com> wrote:
> I still fail to understand the bikeshedding here - you really don't
> need a git checkout to get something akin to a changelog. Use the
> github API directly...
>

The main downside to using github would be that you don't get a
combined history pre/post-migration, but it certainly works.  Github
doesn't work with git replace.  I'm not sure if anongit does or if it
has a useful API like this.  I think you can push git replace
references, but whether the web viewer ignores them or not is another
matter.  They aren't cloned by default I believe (which makes sense
since they're references - an explicit fetch does work).

Somebody could create one big combined repo without using git replace,
but the hashes won't match and that sounds like a recipe for mass
confusion.  You couldn't directly sync it via pull/push either, since
the hashes will never match.

-- 
Rich

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Patrick Lauer]

On 03/02/2016 08:48 PM, malc wrote:
> I still fail to understand the bikeshedding here - you really don't
> need a git checkout to get something akin to a changelog. Use the
> github API directly...
>
> The following 1-liner could be trivially productised (maybe even parse
> $PWD to set the path argument...)
>
> curl https://api.github.com/repos/gentoo/gentoo/commits?path=app-admin/eselect
> | perl -MJSON -e 'foreach $i (@{decode_json(join("",@lines=<STDIN>))})
> { print "$i->{commit}->{author}->{name} -
> $i->{commit}->{author}->{date}\n\n $i->{commit}->{message}\n"; }'
Requires you to be online, can't grep over multiple packages.

This version relies on an unreliable thirdparty service and is thus more
of an intellectual curiosity.
> Yeah - it's not quite as pretty as our current Changelog, but date,
> author/committer, commit-msg etc. are all there and you can filter by
> path just the same as you would with native git log...
> You could parse the local $PORTDIR/metadata/timestamp* and add an
> 'until' param to the URL to filter commits beyond where a user has
> rsync'd up to...
>
It is almost, but not completely unlike it. A simple ChangeLog is a lot
easier ...


(Why are people now trying to add middleware layers to indirect the
problem to become invisible in a huge machinery? This is wonderfully
insane ...)

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

On Thu, Mar 3, 2016 at 1:20 AM, Patrick Lauer <patrick@gentoo.org> wrote:

> It is almost, but not completely unlike it. A simple ChangeLog is a lot
> easier ...
>
>
> (Why are people now trying to add middleware layers to indirect the
> problem to become invisible in a huge machinery? This is wonderfully
> insane ...)


Exactly... The SCM is a lot easier, and you avoid any middleware layers and
indirection :)

Regarding all the previous size arguments: my system doesn't agree. A
recent rsync sync gave me an 856 MB directory (of which 223 MB is those
precious ChangeLog and ChangeLog-2015 files). A checkout from git, plus
news, glsa, etc. (using hasufell's convenient portage-gentoo-git-config on
GitHub) is only 782M, and I can get more detailed change information than
those ChangeLog* files could ever give. Then there's the option of putting
it on btrfs with compression if I still want the working tree smaller. You
can argue about squashfs reducing the size, but you can do that with a tree
from git just like you could with a tree from rsync, so it's irrelevant.

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

Re: [gentoo-dev] Re: [gentoo-project] Portage repo usage survey and change evaluation

[Rich Freeman]

On Wed, Mar 2, 2016 at 1:14 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>
> For example, the message of the initial commit 56bd759 appears in some
> 18000 files, which accounts for 25 MiB.

Not discounting the general issue, I wouldn't count the initial
commit.  All that space will get taken up the first time something
gets committed to all the packages.

-- 
Rich