[gentoo-project] Call for agenda items - Council meeting 2016-04-10

[gentoo-project] Call for agenda items - Council meeting 2016-04-10

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

In two weeks from now, the council will meet again. This is the time
to raise and prepare items that the council should put on the agenda
to discuss or vote on.

Please respond to this message with agenda items. Do not hesitate to
repeat your agenda item here with a pointer if you previously
suggested one (since the last meeting).

The agenda for the meeting will be sent out on Sunday 2016-04-03.

Please reply to the gentoo-project list.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-project] Call for agenda items - Council meeting 2016-04-10

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Sun, 27 Mar 2016 22:00:39 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:

> In two weeks from now, the council will meet again. This is the time
> to raise and prepare items that the council should put on the agenda
> to discuss or vote on.
> 
> Please respond to this message with agenda items. Do not hesitate to
> repeat your agenda item here with a pointer if you previously
> suggested one (since the last meeting).
> 
> The agenda for the meeting will be sent out on Sunday 2016-04-03.

While I still remember about it, I would like to get the Council's
stamp of approval for GLEP 68 [1,2].

[1]:https://bugs.gentoo.org/show_bug.cgi?id=glep68
[2]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:68

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

> In two weeks from now, the council will meet again. This is the time
> to raise and prepare items that the council should put on the agenda
> to discuss or vote on.

I would like the council to follow up on the results of robbat2's
portage repo usage survey:
https://archives.gentoo.org/gentoo-project/message/c2ffa62837fd4cbdd42945bf57b09b25

The following two points should be discussed and possibly be voted on:

1. Should we continue providing ChangeLog files in the rsync
distribution?

Some facts: As of today, ChangeLog* files take up 156 MiB out of a
total size of 390 MiB in the rsync tree, or 40 %. The files propagated
from CVS (i.e. ChangeLog-20*) account for 103 MiB, whereas the
ChangeLog files generated from Git commit messages account for 53 MiB.

Even when ignoring the initial Git commit (whose message is duplicated
in all ChangeLogs and accounts for 25 MiB), we still have a growth
rate of 28 MiB in 239 days (2015-08-08 to 2016-04-03) which is 43 MiB
per year.

2. If we continue providing Changelogs, then what should be their
order of entries?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Rich Freeman]

On Sun, Apr 3, 2016 at 2:07 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>> In two weeks from now, the council will meet again. This is the time
>> to raise and prepare items that the council should put on the agenda
>> to discuss or vote on.
>
> I would like the council to follow up on the results of robbat2's
> portage repo usage survey:
> https://archives.gentoo.org/gentoo-project/message/c2ffa62837fd4cbdd42945bf57b09b25
>
> The following two points should be discussed and possibly be voted on:
>
> 1. Should we continue providing ChangeLog files in the rsync
> distribution?
>
> Some facts: As of today, ChangeLog* files take up 156 MiB out of a
> total size of 390 MiB in the rsync tree, or 40 %. The files propagated
> from CVS (i.e. ChangeLog-20*) account for 103 MiB, whereas the
> ChangeLog files generated from Git commit messages account for 53 MiB.
>
> Even when ignoring the initial Git commit (whose message is duplicated
> in all ChangeLogs and accounts for 25 MiB), we still have a growth
> rate of 28 MiB in 239 days (2015-08-08 to 2016-04-03) which is 43 MiB
> per year.
>
> 2. If we continue providing Changelogs, then what should be their
> order of entries?

What project (if any) is officially responsible for the creation or
non-creation of Changelogs in the rsync mirrors?  Do they have an
opinion on this matter?  Would they prefer that the Council make a
decision?

I bring this up because this seems like the sort of thing the Council
typically doesn't interfere with.

Right now I'm personally inclined to vote against any resolution
requiring anybody to do anything simply because I don't see a pressing
need to impose a policy on them.  I'd encourage anybody who wants a
repo with different/absent Changelogs to just create one and let
others sync it as they desire.

I can certainly see the pros and cons but they don't really seem all
that dramatic to me.

-- 
Rich

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Matthew Thode]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

On 04/03/2016 01:40 PM, Rich Freeman wrote:
> On Sun, Apr 3, 2016 at 2:07 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>> In two weeks from now, the council will meet again. This is the time
>>> to raise and prepare items that the council should put on the agenda
>>> to discuss or vote on.
>>
>> I would like the council to follow up on the results of robbat2's
>> portage repo usage survey:
>> https://archives.gentoo.org/gentoo-project/message/c2ffa62837fd4cbdd42945bf57b09b25
>>
>> The following two points should be discussed and possibly be voted on:
>>
>> 1. Should we continue providing ChangeLog files in the rsync
>> distribution?
>>
>> Some facts: As of today, ChangeLog* files take up 156 MiB out of a
>> total size of 390 MiB in the rsync tree, or 40 %. The files propagated
>> from CVS (i.e. ChangeLog-20*) account for 103 MiB, whereas the
>> ChangeLog files generated from Git commit messages account for 53 MiB.
>>
>> Even when ignoring the initial Git commit (whose message is duplicated
>> in all ChangeLogs and accounts for 25 MiB), we still have a growth
>> rate of 28 MiB in 239 days (2015-08-08 to 2016-04-03) which is 43 MiB
>> per year.
>>
>> 2. If we continue providing Changelogs, then what should be their
>> order of entries?
> 
> What project (if any) is officially responsible for the creation or
> non-creation of Changelogs in the rsync mirrors?  Do they have an
> opinion on this matter?  Would they prefer that the Council make a
> decision?
> 
> I bring this up because this seems like the sort of thing the Council
> typically doesn't interfere with.
> 
> Right now I'm personally inclined to vote against any resolution
> requiring anybody to do anything simply because I don't see a pressing
> need to impose a policy on them.  I'd encourage anybody who wants a
> repo with different/absent Changelogs to just create one and let
> others sync it as they desire.
> 
> I can certainly see the pros and cons but they don't really seem all
> that dramatic to me.
> 
Infra deals with it and it's a pain to continuously generate.  It'd be
nice to have people just check/use git instead if they want history, but
people seem resistant to change.  (this is my opinion)

-- 
-- Matthew Thode (prometheanfire)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

>>>>> On Sun, 3 Apr 2016, Rich Freeman wrote:

> What project (if any) is officially responsible for the creation or
> non-creation of Changelogs in the rsync mirrors?  Do they have an
> opinion on this matter?  Would they prefer that the Council make a
> decision?

> I bring this up because this seems like the sort of thing the
> Council typically doesn't interfere with.

This concerns the Portage tree as a whole, as it is seen by a large
fraction of users. Therefore I think it is a global issue, which is
genuine council territory. Also the council has discussed this topic
previously, most recently in the 20141014 and 20151108 meetings.

From the 20141014 meeting summary:

   "do we need to continue to create new ChangeLog entries once we're
   operating in git?"
   No: blueness, creffett (proxy for ulm), dberkholz, dilfridge,
   radhermit, rich0, williamh

Furthermore, quoting robbat2's message from March 2nd:

| Either way, ~60% are in favour of getting rid of changelogs.

| IMO this is a BETTER goal than continuing to generate them for
| rsync, and bike-shedding about what the order should be; and it
| provides a huge benefit by reducing the size of rsync by 155MiB.

To me this sounds more like an open question than as a notice that
infra is going to drop ChangeLogs. If the council thinks that such
a decision is at infra's discretion then presumably we should make
a statement to that effect.

> Right now I'm personally inclined to vote against any resolution
> requiring anybody to do anything simply because I don't see a
> pressing need to impose a policy on them.  I'd encourage anybody who
> wants a repo with different/absent Changelogs to just create one and
> let others sync it as they desire.

Presumably, this would imply duplicating the rsync mirror system?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Rich Freeman]

On Sun, Apr 3, 2016 at 4:32 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Sun, 3 Apr 2016, Rich Freeman wrote:
>
>> What project (if any) is officially responsible for the creation or
>> non-creation of Changelogs in the rsync mirrors?  Do they have an
>> opinion on this matter?  Would they prefer that the Council make a
>> decision?
>
>> I bring this up because this seems like the sort of thing the
>> Council typically doesn't interfere with.
>
> This concerns the Portage tree as a whole, as it is seen by a large
> fraction of users. Therefore I think it is a global issue, which is
> genuine council territory. Also the council has discussed this topic
> previously, most recently in the 20141014 and 20151108 meetings.
>
> From the 20141014 meeting summary:
>
>    "do we need to continue to create new ChangeLog entries once we're
>    operating in git?"
>    No: blueness, creffett (proxy for ulm), dberkholz, dilfridge,
>    radhermit, rich0, williamh

Sure, but the whole point of our vote was to not create a requirement.
Per the previous Council decision there is no requirement for
Changelogs to be present, but there is also no prohibition on them
being present.

I'm not really advocating for changing this.

>
> Furthermore, quoting robbat2's message from March 2nd:
>
> | Either way, ~60% are in favour of getting rid of changelogs.
>
> | IMO this is a BETTER goal than continuing to generate them for
> | rsync, and bike-shedding about what the order should be; and it
> | provides a huge benefit by reducing the size of rsync by 155MiB.
>
> To me this sounds more like an open question than as a notice that
> infra is going to drop ChangeLogs. If the council thinks that such
> a decision is at infra's discretion then presumably we should make
> a statement to that effect.

That works fine for me, and was basically my intent.

>
>> Right now I'm personally inclined to vote against any resolution
>> requiring anybody to do anything simply because I don't see a
>> pressing need to impose a policy on them.  I'd encourage anybody who
>> wants a repo with different/absent Changelogs to just create one and
>> let others sync it as they desire.
>
> Presumably, this would imply duplicating the rsync mirror system?

Sure.  It has already been done once with git:
https://github.com/gentoo-mirror/gentoo.git

I don't see why others couldn't do the same if they wished.

In any case, I'm fine with leaving it completely up to infra.  I
consider Changelogs to be nice to have, and I'm not going to force
devs to maintain them.  I'm not sure how I could even force a dev to
maintain them if I wanted to.  At most I could volunteer to maintain
them myself, and I don't intend to do that.  However, if somebody else
wants to maintain them and feels there is a barrier standing in their
way I don't have a problem with removing that barrier if it is
reasonable to do so.

-- 
Rich

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[NP-Hardass]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 04/03/2016 06:55 PM, Rich Freeman wrote:
> On Sun, Apr 3, 2016 at 4:32 PM, Ulrich Mueller <ulm@gentoo.org>
> wrote:
>>>>>>> On Sun, 3 Apr 2016, Rich Freeman wrote:
>> 
>>> What project (if any) is officially responsible for the
>>> creation or non-creation of Changelogs in the rsync mirrors?
>>> Do they have an opinion on this matter?  Would they prefer that
>>> the Council make a decision?
>> 
>>> I bring this up because this seems like the sort of thing the 
>>> Council typically doesn't interfere with.
>> 
>> This concerns the Portage tree as a whole, as it is seen by a
>> large fraction of users. Therefore I think it is a global issue,
>> which is genuine council territory. Also the council has
>> discussed this topic previously, most recently in the 20141014
>> and 20151108 meetings.
>> 
>> From the 20141014 meeting summary:
>> 
>> "do we need to continue to create new ChangeLog entries once
>> we're operating in git?" No: blueness, creffett (proxy for ulm),
>> dberkholz, dilfridge, radhermit, rich0, williamh
> 
> Sure, but the whole point of our vote was to not create a
> requirement. Per the previous Council decision there is no
> requirement for Changelogs to be present, but there is also no
> prohibition on them being present.
> 
> I'm not really advocating for changing this.
> 
>> 
>> Furthermore, quoting robbat2's message from March 2nd:
>> 
>> | Either way, ~60% are in favour of getting rid of changelogs.
>> 
>> | IMO this is a BETTER goal than continuing to generate them for 
>> | rsync, and bike-shedding about what the order should be; and
>> it | provides a huge benefit by reducing the size of rsync by
>> 155MiB.
>> 
>> To me this sounds more like an open question than as a notice
>> that infra is going to drop ChangeLogs. If the council thinks
>> that such a decision is at infra's discretion then presumably we
>> should make a statement to that effect.
> 
> That works fine for me, and was basically my intent.
> 
>> 
>>> Right now I'm personally inclined to vote against any
>>> resolution requiring anybody to do anything simply because I
>>> don't see a pressing need to impose a policy on them.  I'd
>>> encourage anybody who wants a repo with different/absent
>>> Changelogs to just create one and let others sync it as they
>>> desire.
>> 
>> Presumably, this would imply duplicating the rsync mirror
>> system?
> 
> Sure.  It has already been done once with git: 
> https://github.com/gentoo-mirror/gentoo.git
> 
> I don't see why others couldn't do the same if they wished.
> 
> In any case, I'm fine with leaving it completely up to infra.  I 
> consider Changelogs to be nice to have, and I'm not going to force 
> devs to maintain them.  I'm not sure how I could even force a dev
> to maintain them if I wanted to.  At most I could volunteer to
> maintain them myself, and I don't intend to do that.  However, if
> somebody else wants to maintain them and feels there is a barrier
> standing in their way I don't have a problem with removing that
> barrier if it is reasonable to do so.
> 

An aside: Part of the git migration was converting the cvs history to
git to be available as a graft point [1][2]. Regardless of whether
this is to ultimately become the canonical means of looking up a
package's changelog, can we document how the graft is supposed to be
done/configure the repo accordingly?  I'm completely unfamiliar with
git grafting, but [3] suggests that there should be an entry in
.git/info/grafts.  Since I'm unfamiliar with grafting, is there a
reason why we don't ship the repo with this file populated to begin with
?

- -- 
NP-Hardass

[1]
https://wiki.gentoo.org/wiki/Project:Infrastructure/Git_migration#Steps
[2] https://gitweb.gentoo.org/repo/gentoo/historical.git
[3] https://git.wiki.kernel.org/index.php/GraftPoint
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJXAdEdAAoJEBzZQR2yrxj7SroP/j7V3r7hKC/TxrxkZNwsMq5E
97Cjg0qUpjbDjzxTlCVKQHL2tc5ff3dWI9PfXPpol6mo/DMmPdiPClXGKg6ukmmh
5zGjDQ3DZHw6F1r4jINjZ6/Dp+w5k2qeLFCJnKpD25RvGldYxwlltoxVp4D7dhCn
sztYEksHYqmeBx+TQTxlvfx2BCaiPk/AE2VQI4KDaETcgZv5hVrg9tMqljG45S+/
WVrB7jv4s68As6yqCbOyQkFDAi4IyoKNhJXREpV56lbS0ujvCL5fKXz2QyRGAPwi
/jtTISbTMYu9xo6cCJCJwRMJxSacpip7Hv28tfUkGFKs0Taeib2q+vr1Fd29073q
RfM2S0coVjZWJnlIdE0wclfDYhkv6Y2rGwBgEWxz0xatFAsQKHK0wqD8mb0q9VY8
X2Xoh6kzJ6fivoxfhHS8GkC7Wx6ir9bo9SnfaT7kV59OvL9iUbysV7MxTNMRtl9c
9NP7ozIOD5FTePyvlQGLGQChJ6px4520oGDZxxJQR/ZrENSTR29SibjMjhj6MM+T
igdNnx844NP7/SJCEtRolxcwExN090Vw6pAxrYvTRBj/rJ/RkWwG+hry2OhJ717x
I2ZytlrV6pe2osJXtl4adCImINnFixcLx6/rKtV+QNnrlyGSIeFqH7gkk6kki9rd
mpFerByvoaMy+M1m+mWY
=/1DF
-----END PGP SIGNATURE-----

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1774 bytes --]

Hi,

On Sun, 3 Apr 2016 15:06:59 -0500 Matthew Thode wrote:
> On 04/03/2016 01:40 PM, Rich Freeman wrote:
[...]
> > What project (if any) is officially responsible for the creation or
> > non-creation of Changelogs in the rsync mirrors?  Do they have an
> > opinion on this matter?  Would they prefer that the Council make a
> > decision?
> > 
> > I bring this up because this seems like the sort of thing the Council
> > typically doesn't interfere with.
> > 
> > Right now I'm personally inclined to vote against any resolution
> > requiring anybody to do anything simply because I don't see a pressing
> > need to impose a policy on them.  I'd encourage anybody who wants a
> > repo with different/absent Changelogs to just create one and let
> > others sync it as they desire.
> > 
> > I can certainly see the pros and cons but they don't really seem all
> > that dramatic to me.
> > 
> Infra deals with it and it's a pain to continuously generate.  It'd be
> nice to have people just check/use git instead if they want history, but
> people seem resistant to change.  (this is my opinion)
 
Git is not a suitable replacement for rsync in some legitimate
cases. From memory I remember two (of course, other cases are
possible):

1) Git works poorly on slow/unstable connections: it can't resume
truncated connection and starts over again from the beginning.

2) While git pull is much faster on a decent connection than rsync,
metadata generation is damn slow, especially on older systems. On
my Atom it takes several hours which is unacceptable. Please note
I'm not talking about ChangeLogs generation, I meant metadata
directory requires for portage and many tools (like eix) to work
properly.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Rich Freeman]

On Sun, Apr 3, 2016 at 10:27 PM, NP-Hardass <NP-Hardass@gentoo.org> wrote:
>
> An aside: Part of the git migration was converting the cvs history to
> git to be available as a graft point [1][2]. Regardless of whether
> this is to ultimately become the canonical means of looking up a
> package's changelog, can we document how the graft is supposed to be
> done/configure the repo accordingly?  I'm completely unfamiliar with
> git grafting, but [3] suggests that there should be an entry in
> .git/info/grafts.

https://wiki.gentoo.org/wiki/Gentoo_git_workflow#Grafting_Gentoo_history_onto_the_active_repository

> Since I'm unfamiliar with grafting, is there a
> reason why we don't ship the repo with this file populated to begin with

It is a reference, and if you fetch/clone a git repo you only get the
references you explicitly ask for.  I've actually tried pushing the
reference and while that works you don't get it when you clone unless
you ask for it, which is the same effort as just re-creating it.

-- 
Rich

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Rich Freeman]

On Mon, Apr 4, 2016 at 3:01 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>
> 2) While git pull is much faster on a decent connection than rsync,
> metadata generation is damn slow, especially on older systems. On
> my Atom it takes several hours which is unacceptable. Please note
> I'm not talking about ChangeLogs generation, I meant metadata
> directory requires for portage and many tools (like eix) to work
> properly.

I don't think anybody has proposed ditching rsync metadata - just the
Changelogs at this point.

However, there is a git mirror that contains metadata:
https://github.com/gentoo-mirror/gentoo.git

I use this personally - it syncs really fast daily, and if I'm
tweaking ebuilds I'm typically doing it in /usr/portage before I'm
doing it in my actual committing tree.  By running git in /usr/portage
I can trivially extract patches/etc and confirm that I'm not missing
things when I make the actual commits.  It also makes changes
persistent across syncs.

In any case, it is just one more option at this point.

-- 
Rich

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[NP-Hardass]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 04/04/2016 05:24 AM, Rich Freeman wrote:
> On Sun, Apr 3, 2016 at 10:27 PM, NP-Hardass <NP-Hardass@gentoo.org>
> wrote:
>> 
>> An aside: Part of the git migration was converting the cvs
>> history to git to be available as a graft point [1][2].
>> Regardless of whether this is to ultimately become the canonical
>> means of looking up a package's changelog, can we document how
>> the graft is supposed to be done/configure the repo accordingly?
>> I'm completely unfamiliar with git grafting, but [3] suggests
>> that there should be an entry in .git/info/grafts.
> 
> https://wiki.gentoo.org/wiki/Gentoo_git_workflow#Grafting_Gentoo_histo
ry_onto_the_active_repository
>
> 
>> Since I'm unfamiliar with grafting, is there a reason why we
>> don't ship the repo with this file populated to begin with
> 
> It is a reference, and if you fetch/clone a git repo you only get
> the references you explicitly ask for.  I've actually tried pushing
> the reference and while that works you don't get it when you clone
> unless you ask for it, which is the same effort as just re-creating
> it.
> 

Ah.  Thanks so much.  Greatly appreciate the info/resources.  Didn't
realize that the info was on the wiki.

- -- 
NP-Hardass
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJXAmgRAAoJEBzZQR2yrxj7wHUQAJoQxieMDHbR0gdfv0Ejp/G3
qacdcLm8CjmJTbRM4jpP67uHtWuNalDJ++Do/7srMppFq++wI/z8epQiXlDT3EYS
LD4DRFwvZXMMFYsJXmTDR+0lQYSCx/85jA+IJHuKYKHiINVTT3lQ2DmeIguhgE7X
nleFSTKReqK6safHurDCSJ3/iU8RUSRfhdOf3Gq5YY3YE4wG3kHy15Ih0Gdo9c8H
ZkVYlTRYCahd+7oGbd7oef/YI7M+PRvz+McqSx7o6xIVpZg0lQSWe6WLn97OAvc2
G3yh/LyCByTveavGErgpZsVDZTbO74kETGF0UmmuQi35yqp8FAWRumqSypOxAUyn
NE3Kvp92M6thTDjonO8sSql4QtQ+mYfsZ1/yTCBy7K35Dj3gOWLLhTlbdwIJM/qb
KbicUg4Q1+ojyw31x+Hag4tAn9b1JNpIdxh6/vdJ1jauFpYXVG/S2AMQchtMxo15
DlW9BsMsxbXtlzdmjNNCela4/q9eWyT5ztb0M2r43V+/6FSknWF0SBBucwD9hqmM
7ZLks4rHexAiH1sdGTEWOniEFerIJTRb0GjYUryrNQlJh4OrEtfr5I4vHJ7DUQvk
nrnO/YCSLqJkF+3C0kiaUjavs6Uymt9qqW7G1GgiwkcrOUPE+NrHaScWjnZGFdAo
q5lHytItr4i84mZlrcH5
=okrV
-----END PGP SIGNATURE-----

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

Hi,

On Mon, 4 Apr 2016 05:29:07 -0400 Rich Freeman wrote:
> On Mon, Apr 4, 2016 at 3:01 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> >
> > 2) While git pull is much faster on a decent connection than rsync,
> > metadata generation is damn slow, especially on older systems. On
> > my Atom it takes several hours which is unacceptable. Please note
> > I'm not talking about ChangeLogs generation, I meant metadata
> > directory requires for portage and many tools (like eix) to work
> > properly.
> 
> I don't think anybody has proposed ditching rsync metadata - just the
> Changelogs at this point.

Please see Mattehew's e-mail I replied to:

> It'd be nice to have people just check/use git instead if they want
> history, but people seem resistant to change.

I explained my opinion why git can't fully replace rsync at this
moment.

> However, there is a git mirror that contains metadata:
> https://github.com/gentoo-mirror/gentoo.git

Thanks, I wasn't aware that it contains metadata. Can we have the
same on our infra?

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Patrick Lauer]

On 04/05/2016 08:59 AM, Andrew Savchenko wrote:
> Hi,
>
> On Mon, 4 Apr 2016 05:29:07 -0400 Rich Freeman wrote:
>> On Mon, Apr 4, 2016 at 3:01 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>>> 2) While git pull is much faster on a decent connection than rsync,
>>> metadata generation is damn slow, especially on older systems. On
>>> my Atom it takes several hours which is unacceptable. Please note
>>> I'm not talking about ChangeLogs generation, I meant metadata
>>> directory requires for portage and many tools (like eix) to work
>>> properly.
>> I don't think anybody has proposed ditching rsync metadata - just the
>> Changelogs at this point.
> Please see Mattehew's e-mail I replied to:
>
>> It'd be nice to have people just check/use git instead if they want
>> history, but people seem resistant to change.
> I explained my opinion why git can't fully replace rsync at this
> moment.
And you make the workflow painful:
- rsync and git are not synchronized, so I can't even be sure that I'm
looking at the right thing in git (e.g. if there were some commits in
between that affect the thing that appears to be broken, so works in git
head but is broken in current rsync checkout)

- Either I struggle with a frustratingly bad "Web UI" (both gitweb and
proprietary frontends like github are sad)
- Or I have to clone a huge repository locally to be able to look at it,
instead of looking at things on the machine where it broke

- Now I need both rsync *and* git checkout to see issues (see for
example https://bugs.gentoo.org/show_bug.cgi?id=577722 that only affects
rsync, and only some of the time)

But who cares about users ;)
(Yeah, I'm possibly a bit cynical because after 9 months we're still in
the middle of a git migration with no end in sight ...)

Re: [gentoo-project] Call for agenda items - Council meeting 2016-04-10

[Alexis Ballier]

On Sunday, April 3, 2016 8:07:13 PM CEST, Ulrich Mueller wrote:
>> In two weeks from now, the council will meet again. This is the time
>> to raise and prepare items that the council should put on the agenda
>> to discuss or vote on.
>
> I would like the council to follow up on the results of robbat2's
> portage repo usage survey:
> https://archives.gentoo.org/gentoo-project/message/c2ffa62837fd4cbdd42945bf57b09b25
>
> The following two points should be discussed and possibly be voted on:
>
> 1. Should we continue providing ChangeLog files in the rsync
> distribution?

If space is the sole consideration, then, changelogs can be removed from 
manifests, and $PM default rsync command can exclude '*/*/ChangeLog*' (or, 
at least, leave the possibility to do it). This solution seems to be what 
would please everyone since robbat2's survey results can be interpreted in 
many ways, but for Q2, more than 50% voted "something but only if it were 
optional".


However, I think I recall a nice recap of some infra guy on how much time 
it took to generate them. IIRC it was bearable at the moment (a few hours) 
but still slow. Assuming infra hardware stays the same, where will we be in 
1, 2 or 5 years wrt to generating changelogs ? Is there something that can 
be improved on the software side or are we just bound to have slower and 
slower rsync distribution generation ? If so, how much slower does it get 
over time ?
What I mean there is that if changelog generation takes 5 days then we 
don't have much of a choice but dropping them.

Alexis.

Re: [gentoo-project] Re: Call for agenda items - Council meeting 2016-04-10

[Rich Freeman]

On Tue, Apr 5, 2016 at 2:59 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>
> On Mon, 4 Apr 2016 05:29:07 -0400 Rich Freeman wrote:
>> On Mon, Apr 4, 2016 at 3:01 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
>> >
>> > 2) While git pull is much faster on a decent connection than rsync,
>> > metadata generation is damn slow, especially on older systems. On
>> > my Atom it takes several hours which is unacceptable. Please note
>> > I'm not talking about ChangeLogs generation, I meant metadata
>> > directory requires for portage and many tools (like eix) to work
>> > properly.
>>
>> I don't think anybody has proposed ditching rsync metadata - just the
>> Changelogs at this point.
>
> Please see Mattehew's e-mail I replied to:

There is no need to have a local copy if you don't want one.  Just use
one of the many available web-based front ends.

>
>> However, there is a git mirror that contains metadata:
>> https://github.com/gentoo-mirror/gentoo.git
>
> Thanks, I wasn't aware that it contains metadata. Can we have the
> same on our infra?

No objections from me - by all means go ahead!

-- 
Rich

Re: [gentoo-project] Call for agenda items - Council meeting 2016-04-10

[Patrick Lauer]

On 04/05/2016 11:37 AM, Alexis Ballier wrote:
> On Sunday, April 3, 2016 8:07:13 PM CEST, Ulrich Mueller wrote:
>>> In two weeks from now, the council will meet again. This is the time
>>> to raise and prepare items that the council should put on the agenda
>>> to discuss or vote on.
>>
>> I would like the council to follow up on the results of robbat2's
>> portage repo usage survey:
>> https://archives.gentoo.org/gentoo-project/message/c2ffa62837fd4cbdd42945bf57b09b25
>>
>>
>> The following two points should be discussed and possibly be voted on:
>>
>> 1. Should we continue providing ChangeLog files in the rsync
>> distribution?
>
> If space is the sole consideration, then, changelogs can be removed
> from manifests, and $PM default rsync command can exclude
> '*/*/ChangeLog*' (or, at least, leave the possibility to do it). This
> solution seems to be what would please everyone since robbat2's survey
> results can be interpreted in many ways, but for Q2, more than 50%
> voted "something but only if it were optional".
>
>
> However, I think I recall a nice recap of some infra guy on how much
> time it took to generate them. IIRC it was bearable at the moment (a
> few hours) but still slow. Assuming infra hardware stays the same,
> where will we be in 1, 2 or 5 years wrt to generating changelogs ? Is
> there something that can be improved on the software side or are we
> just bound to have slower and slower rsync distribution generation ?
> If so, how much slower does it get over time ?
If one had lots of time for fixing up things ... an incremental
thick-manifest update in lockstep with new commits would be possible. So
maybe each commit lags by a few seconds to a few minutes (if it's a
single commit touching lots of files).

But there would always be a thick-manifest copy available that would be
behind only by a small time window, and could be propagated to rsync
mirrors easily. The problem with that approach is that someone would
need to write such code, test it, etc.etc.


> What I mean there is that if changelog generation takes 5 days then we
> don't have much of a choice but dropping them.
>
> Alexis.
>
Pff. Worst case you throw more hardware at it ;)

Re: [gentoo-project] Call for agenda items - Council meeting 2016-04-10

[Alexis Ballier]

On Tuesday, April 5, 2016 11:59:40 AM CEST, Patrick Lauer wrote:
>> What I mean there is that if changelog generation takes 5 days then we
>> don't have much of a choice but dropping them.
>> 
>> Alexis.
>> 
> Pff. Worst case you throw more hardware at it ;)


Eeeeh, no :)
You're assuming linear increase in generation time, for which I'd like to 
see numbers first.

And anyway, this would mean it is an infra (& maybe trustees) issue, not a 
council one :p