[gentoo-project] Portage repo usage survey and change evaluation

[gentoo-project] Portage repo usage survey and change evaluation

[Robin H. Johnson]

http://goo.gl/forms/5riWkN8VMK

I've put together a quick survey about a number of potential changes to
the rsync distribution that have been bike-shedded about in various IRC
channels and the mailing lists for some time now.

Questions:
(answer them on the query, input via email will not be considered or
bikeshedded).
- do you use changelogs
- (rsync) exclude changelogs to save local disk space
- (rsync) order of changelog entries?
- Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata 


I'll report the results not sooner than 2 weeks from now, but I might
put together some intermediate results if I see anything interesting
happening.

P.S. For the crowd that wants to claim I'm depending on some non-open
infrastructure, I'm not going to go through all of the hoops to set up
some open source questionnaire infrastructure at this point. If don't
like that fact that I've used Google, simply don't answer the survey,
and feel free to complain about any results that I do end up collecting.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Rich Freeman]

On Mon, Feb 1, 2016 at 1:11 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> - Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata
>

You're missing an option to replace rsync with a git repo that has
metadata, but not the other stuff.

Of course, this already exists and no doubt many are using it.  It
just isn't official.

I'm fine with Changelogs going away entirely, in any case.  That's
what git is for.  If people want to spend time dynamically generating
them, it isn't harmful to do so.

-- 
Rich

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

You should probably ask a broader audience than just gentoo-project.
Maybe post it in the forums and send an email to the users list.

BTW, I guess my efforts to get devs to call it the Gentoo tree instead
of the Portage repo or whatever (which would be the source code
repository of Portage the software) were completely wasted...
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWr0/QAAoJENQqWdRUGk8BtREQAO3dvBxqeYLRtjK+WAc5bsiz
BJRAlQvWwX8XZnUjamJ6IwABFgFH+8HWKqd874GuIOCC30nMkoYRr0EghIjIVVfx
K8yeRR0sAuaGufMXIiJoupmasnGzlqgpbNn4dAWBZU6WPYdcMArmqCavAtLMYW9B
qdMD7Uc8ZuW3784JLwH1tB7c9hoLaWhuypasuxjySrAtZ7duwYB/+LsYtCAVWMLv
5ieG0ZEAdFmLdZ/5A54OXH5QhBPAA/PmDwFGYScvyM+tXUdQnoJOpxx/XEPhZ8i6
7z+5Ck+AebsS9VLVagAj+H4e2Rr21ctjGYd2IVKl8edi5og71lG2DVXEtFOjs2YC
T05K6vwOGKY9rk4rzSJoiYraYqsj0kI1rHgQVx8uiUiLs76gKvtATgDw+mYY6zru
OFNhKqYJ9pk6TfzAiIGq12WVQ5Go2RK8OsJkqSHDgG8HS0OVfFB7mf8HEJ9EUpnA
I4IF1tYdBg5z/jah3QbiaDhu69eY7fzY0EzI8bIzr80RMRP2ED/EeQ7WNQIJPaXz
Mwzf8GDyLj9fMsWZCxTbNdlOkAWzTzrbAEiP2ZxR5oCe7f3J5pYRpDED7raNatXt
j5yJIJu9WhddkJ0UvpVXNB+/NchrdR+ftvc0egBRmU+8uEyhU42NGntZU7IZla37
DaMAtAA2AcYTIr1AAi4u
=i3Ib
-----END PGP SIGNATURE-----

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

On Mon, 1 Feb 2016 06:46:26 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Mon, Feb 1, 2016 at 1:11 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> > - Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata
> >  
> 
> You're missing an option to replace rsync with a git repo that has
> metadata, but not the other stuff.
> 
> Of course, this already exists and no doubt many are using it.  It
> just isn't official.

It *is* official. It just isn't hosted on Gentoo Infra :-P. It's
officially maintained by an official Gentoo developers who happens to
be officially on Infra too, and officially is more reliable than
Infra-provided distributions channels were. Can it be more official?!

But seriously, I don't mind moving either the scripts or the mirrors to
Gentoo infrastructure. But...

...as for the scripts, Infra so far didn't offer me hardware better
than the one donated for the scripts now. There were proposals about
distributing it, etc. But I simply don't have the time to work on
making the scripts very complex in order to make them run efficiently
in a complex environment, when simple scripts work very well so far.

I've already spent some effort cleaning them up and making
configurable. I haven't had the time to prepare proper
deployment/install script though. It's still in todo, and sometimes I
regret I don't have a backup server ready to take over whenever this
one fails (which is still more rare than Gentoo infra problems).

...and as for the mirrors, nobody has offered to host them so far.
While GitHub provides quite a lot bandwidth, for free, without having
to ask and with API that makes it trivial to create and delete
repositories as needed. If someone can provide something like this, I
don't mind mirroring to 2+ hosts.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Patrick Lauer]

On 02/01/2016 07:11 AM, Robin H. Johnson wrote:
> http://goo.gl/forms/5riWkN8VMK
>
> I've put together a quick survey about a number of potential changes to
> the rsync distribution that have been bike-shedded about in various IRC
> channels and the mailing lists for some time now.
>
> Questions:
> (answer them on the query, input via email will not be considered or
> bikeshedded).
> - do you use changelogs
yes
> - (rsync) exclude changelogs to save local disk space
no
> - (rsync) order of changelog entries?
the right order (reverse chronological)
> - Augment/replace rsync with git repo that has thick-Manifests, changelogs, metadata 
No.
>
>
> I'll report the results not sooner than 2 weeks from now, but I might
> put together some intermediate results if I see anything interesting
> happening.
>
> P.S. For the crowd that wants to claim I'm depending on some non-open
> infrastructure, I'm not going to go through all of the hoops to set up
> some open source questionnaire infrastructure at this point. If don't
> like that fact that I've used Google, simply don't answer the survey,
> and feel free to complain about any results that I do end up collecting.
>
Eh, don't expect me to figure out how to operate such things :)

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

>>>>> On Mon, 1 Feb 2016, Robin H Johnson wrote:

> http://goo.gl/forms/5riWkN8VMK
> I've put together a quick survey about a number of potential changes
> to the rsync distribution that have been bike-shedded about in
> various IRC channels and the mailing lists for some time now.

> Questions:
> (answer them on the query, input via email will not be considered or
> bikeshedded).
> - do you use changelogs
> - (rsync) exclude changelogs to save local disk space
> - (rsync) order of changelog entries?
> - Augment/replace rsync with git repo that has thick-Manifests,
> changelogs, metadata  

> I'll report the results not sooner than 2 weeks from now, but I
> might put together some intermediate results if I see anything
> interesting happening.

Have I missed your posting the results of this? Especially, what is
the preferred ordering of ChangeLog entries?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On Mon, Feb 29, 2016 at 09:01:19AM +0100, Ulrich Mueller wrote:
> Have I missed your posting the results of this? Especially, what is
> the preferred ordering of ChangeLog entries?

I just hadn't finished putting the results into a long-term format quite
yet, but did so this afternoon:
http://dev.gentoo.org/~robbat2/201602-portage-survey/

I have included a CSV of the public answers, excludes only the last
question about contact info.

Some remarks about question #2 and #3:

Q2: Reduce local disk usage by excluding ChangeLogs?
----------------------------------------------------
It was unfortunately pointed out to me very late that my question #2 had
some confusing text:
- "No, but only if were optional (I do NOT want it, but others might)"
- "Yes, but only if it were optional (I want it, but others might NOT)"

The bracket portion of each answer was interpreted as meaning the
opposite as the start of each answer :-(.

Either way, ~60% are in favour of getting rid of changelogs.

IMO this is a BETTER goal than continuing to generate them for rsync,
and bike-shedding about what the order should be; and it provides a huge
benefit by reducing the size of rsync by 155MiB.

Q3: What order should ChangeLog entries be in?
----------------------------------------------
- 85.3% of responses either preferred newest first OR didn't care (incl
  so as long as the tools work).
- 2.9% wanted oldest first.
- NOBODY selected "I'd prefer oldest entries first, but do what is best
  for distribution"
- 11.8% said get rid of changelogs.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-project] Portage repo usage survey and change evaluation

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

>>>>> On Wed, 2 Mar 2016, Robin H Johnson wrote:

> I just hadn't finished putting the results into a long-term format
> quite yet, but did so this afternoon:
> http://dev.gentoo.org/~robbat2/201602-portage-survey/

Thank you.

> Some remarks about question #2 and #3:

> Q2: Reduce local disk usage by excluding ChangeLogs?
> ----------------------------------------------------
> It was unfortunately pointed out to me very late that my question #2
> had some confusing text:
> - "No, but only if were optional (I do NOT want it, but others might)"
> - "Yes, but only if it were optional (I want it, but others might NOT)"

> The bracket portion of each answer was interpreted as meaning the
> opposite as the start of each answer :-(.

> Either way, ~60% are in favour of getting rid of changelogs.

Not sure if it can be interpreted this way. This would contradict the
results of both Q1 and Q3.

For Q1, 45 responses read ChangeLogs in some way (A1.2 to A1.5 or a
combination of them), whereas only 17 responses don't read ChangeLogs
at all (A1.1 or some combination including it). Disregarding the two
responses who at the same time read them and don't read them at all.

> IMO this is a BETTER goal than continuing to generate them for
> rsync, and bike-shedding about what the order should be; and it
> provides a huge benefit by reducing the size of rsync by 155MiB.

Hm, that's almost 40% of the total size of the tree.

   $ find /usr/portage/ -type f -name 'ChangeLog-20*' -printf '%s\n' | awk '{ s+=$1 } END { print s/1024^2 }'
   102.961

That's the old ones from CVS.

   $ find /usr/portage/ -type f -name ChangeLog -printf '%s\n' | awk '{ s+=$1 } END { print s/1024^2 }'
   52.0908

That's the new ones autogenerated from git.

How is it possible that we have 52 MiB of ChangeLog entries generated
in the 0.5 years since the git conversion, whereas we had only a total
of 103 MiB in the 13.5 years since ChangeLogs were introduced in 2002?
Certainly our commit rate hasn't increased by more than an order of
magnitude in the last half year?

> Q3: What order should ChangeLog entries be in?
> ----------------------------------------------
> - 85.3% of responses either preferred newest first OR didn't care
>   (incl so as long as the tools work).
> - 2.9% wanted oldest first.
> - NOBODY selected "I'd prefer oldest entries first, but do what is
>   best for distribution"
> - 11.8% said get rid of changelogs.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]