[gentoo-dev] ChangeLog

[gentoo-dev] ChangeLog

[Patrick Lauer]

Ahoi,

I'm getting mildly very irritated with the lack of easily accessible
ChangeLogs for our packages.

Apparently updating them stopped some time in August, so now there are
some outdated ChangeLogs that don't really serve any purpose, and the
easiest way for users to figure out why something changed is to yell at
the clumsy gitweb.g.o interface. So instead of grep we now need lots of
patience.

This does not look reasonable to me.

Can we please either properly remove ChangeLogs and tell people to not
be curious about changes, or make them useful again?

Thanks,

A Gentoo User.

Re: [gentoo-dev] ChangeLog

[Anthony G. Basile]

On 11/1/15 7:16 AM, Patrick Lauer wrote:
> Ahoi,
>
> I'm getting mildly very irritated with the lack of easily accessible
> ChangeLogs for our packages.
>
> Apparently updating them stopped some time in August, so now there are
> some outdated ChangeLogs that don't really serve any purpose, and the
> easiest way for users to figure out why something changed is to yell at
> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
> patience.
>
> This does not look reasonable to me.
>
> Can we please either properly remove ChangeLogs and tell people to not
> be curious about changes, or make them useful again?
>
> Thanks,
>
> A Gentoo User.
>
I don't have strong feelings about this, but I'm happy with `git log` to 
tell me what's going on with packages.  I would be okay with the 
ChangeLog files just being archived and removed.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ChangeLog

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/01/2015 04:22 AM, Anthony G. Basile wrote:
> On 11/1/15 7:16 AM, Patrick Lauer wrote:
>> Ahoi,
>> 
>> I'm getting mildly very irritated with the lack of easily
>> accessible ChangeLogs for our packages.
>> 
>> Apparently updating them stopped some time in August, so now
>> there are some outdated ChangeLogs that don't really serve any
>> purpose, and the easiest way for users to figure out why
>> something changed is to yell at the clumsy gitweb.g.o interface.
>> So instead of grep we now need lots of patience.
>> 
>> This does not look reasonable to me.
>> 
>> Can we please either properly remove ChangeLogs and tell people
>> to not be curious about changes, or make them useful again?
>> 
>> Thanks,
>> 
>> A Gentoo User.
>> 
> I don't have strong feelings about this, but I'm happy with `git
> log` to tell me what's going on with packages.  I would be okay
> with the ChangeLog files just being archived and removed.
> 

Is there a way for us (devs and users both) to only get `git log`
results from a specific directory or package? I'm aware we could pipe
git log to grep, but that seems hackish, like git has a better way to
isolate log entries.

- -- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWN8ICAAoJEAEkDpRQOeFwa7UP/191KfADyGZfF/cPfBp5tpE8
SP9mYoZB4ZUbCV/UUE5/Oz5st2lBtK+LNI2C8Apea53bu+Qd48qP/Q2l4WtFTVR5
JTfT/EK10kE+hu2tie1Bq43lFdWB+yWSS7lmoHMWkQH5wSr+0jGi1jm5njTvbvSE
GxlNMqFjHcVg3zwkRXoGtPjqQqJqjbX81ZSxxIHYwpYbDJIXgZcgrEy6Hvf8nzTi
5G6GtSoW490rZm+HtRQ8NiC8mKnLqyzsVAQZSFomKu7lA0r9dsI2vzwc31pSBljs
K+Gh1oTho8YCV/+LFADgxJOWk2oI3AaCH5ar39oV8sWO8Hs1VuvAGVFY+0EqiMUL
nSZX/ghjjWP7yO6FrBF8eh32hmP3K/F+ZuMbCubx9pdL+OcuoGWVu3kf8sa64zyY
pJITSy0AqFrdMdTXnpRc4QWW+CmScQFrS8X5uYgQaceKtGHSy6vDzKRYyC6YJcDy
afMB4jHCaLiNFkOzjIkJ6/TnTaCXbnICy5HXb3IAWA+6vj6CZ/zJpSVMm+gdx8np
MDS4ROltElah7yVsauGqca93ux5TNCGkktw3zSJcvYiDZo/0LVsMwXFNOPaT1R2R
p53tW9juK2Tj2eY357DrAij770ru5LznJ00bVFx2HFp8j+fjakflZlZK1BTNFZHK
lZEiWbGPJa54QTcLah7J
=8NAr
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ChangeLog

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

Actually, git log understands if you specify path for it (especially
after --)...


03.11.2015 02:05, Daniel Campbell пишет:
> On 11/01/2015 04:22 AM, Anthony G. Basile wrote:
> > On 11/1/15 7:16 AM, Patrick Lauer wrote:
> >> Ahoi,
> >>
> >> I'm getting mildly very irritated with the lack of easily
> >> accessible ChangeLogs for our packages.
> >>
> >> Apparently updating them stopped some time in August, so now
> >> there are some outdated ChangeLogs that don't really serve any
> >> purpose, and the easiest way for users to figure out why
> >> something changed is to yell at the clumsy gitweb.g.o interface.
> >> So instead of grep we now need lots of patience.
> >>
> >> This does not look reasonable to me.
> >>
> >> Can we please either properly remove ChangeLogs and tell people
> >> to not be curious about changes, or make them useful again?
> >>
> >> Thanks,
> >>
> >> A Gentoo User.
> >>
> > I don't have strong feelings about this, but I'm happy with `git
> > log` to tell me what's going on with packages.  I would be okay
> > with the ChangeLog files just being archived and removed.
>
>
> Is there a way for us (devs and users both) to only get `git log`
> results from a specific directory or package? I'm aware we could pipe
> git log to grep, but that seems hackish, like git has a better way to
> isolate log entries.
>
>



[-- Attachment #2: Type: text/html, Size: 2105 bytes --]

Re: [gentoo-dev] ChangeLog

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

On 2015-11-03 02:22, Vadim A. Misbakh-Soloviov wrote:
> Actually, git log understands if you specify path for it (especially
> after --)...
> 
> 
> 03.11.2015 02:05, Daniel Campbell пишет:
> > On 11/01/2015 04:22 AM, Anthony G. Basile wrote:
> > > On 11/1/15 7:16 AM, Patrick Lauer wrote:
> > >> Ahoi,
> > >>
> > >> I'm getting mildly very irritated with the lack of easily
> > >> accessible ChangeLogs for our packages.
> > >>
> > >> Apparently updating them stopped some time in August, so now
> > >> there are some outdated ChangeLogs that don't really serve any
> > >> purpose, and the easiest way for users to figure out why
> > >> something changed is to yell at the clumsy gitweb.g.o interface.
> > >> So instead of grep we now need lots of patience.
> > >>
> > >> This does not look reasonable to me.
> > >>
> > >> Can we please either properly remove ChangeLogs and tell people
> > >> to not be curious about changes, or make them useful again?
> > >>
> > >> Thanks,
> > >>
> > >> A Gentoo User.
> > >>
> > > I don't have strong feelings about this, but I'm happy with `git
> > > log` to tell me what's going on with packages.  I would be okay
> > > with the ChangeLog files just being archived and removed.
> >
> >
> > Is there a way for us (devs and users both) to only get `git log`
> > results from a specific directory or package? I'm aware we could pipe
> > git log to grep, but that seems hackish, like git has a better way to
> > isolate log entries.
> >
> >
> 
> 

Vadim, please don't top post.

Daniel, yes, you just do:

    git log cat/pkg

Or:
    git log .

`git log' understands path arguments within the repo.

But, that doesn't really give you the full picture. You will be better
served by:

    git log --grep=<pattern>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 345 bytes --]

Re: [gentoo-dev] ChangeLog

[Jeroen Roovers]

On Mon, 2 Nov 2015 16:17:18 -0500
"Aaron W. Swenson" <titanofold@gentoo.org> wrote:

> Vadim, please don't top post.

But do quote some forty lines from the message you reply to. It
really helps in case someone lost the original, right? :-)


     jer

Re: [gentoo-dev] ChangeLog

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

On 2015-11-03 05:24, Jeroen Roovers wrote:
> On Mon, 2 Nov 2015 16:17:18 -0500
> "Aaron W. Swenson" <titanofold@gentoo.org> wrote:
> 
> > Vadim, please don't top post.
> 
> But do quote some forty lines from the message you reply to. It
> really helps in case someone lost the original, right? :-)
> 
> 
>      jer

Exactly! I'm glad somebody shares my views. (^_^)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 345 bytes --]

Re: [gentoo-dev] ChangeLog

[Мисбах-Соловьёв Вадим]

And why don't just only generate them on rsync mirrors, but remove them from git repo (like was planned initially, AFAIRC)?

01.11.2015, 18:17, "Patrick Lauer" <patrick@gentoo.org>:
> Ahoi,
>
> I'm getting mildly very irritated with the lack of easily accessible
> ChangeLogs for our packages.
>
> Apparently updating them stopped some time in August, so now there are
> some outdated ChangeLogs that don't really serve any purpose, and the
> easiest way for users to figure out why something changed is to yell at
> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
> patience.
>
> This does not look reasonable to me.
>
> Can we please either properly remove ChangeLogs and tell people to not
> be curious about changes, or make them useful again?
>
> Thanks,
>
> A Gentoo User.

Re: [gentoo-dev] ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 7:33 AM, Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
> And why don't just only generate them on rsync mirrors, but remove them from git repo (like was planned initially, AFAIRC)?
>

That is in fact how it works.  Or, at least how it is supposed to
work.  I don't use the rsync mirror, so I can't vouch for whether
they're producing ChangeLogs.

Personally I'd just as soon see them go away entirely, but if somebody
wants to make them work I won't stop them.

-- 
Rich

Re: [gentoo-dev] ChangeLog

[Patrick Lauer]

On 11/01/2015 01:53 PM, Rich Freeman wrote:
> On Sun, Nov 1, 2015 at 7:33 AM, Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
>> And why don't just only generate them on rsync mirrors, but remove them from git repo (like was planned initially, AFAIRC)?
>>
> That is in fact how it works.  Or, at least how it is supposed to
> work.  I don't use the rsync mirror, so I can't vouch for whether
> they're producing ChangeLogs.
Supposed to, but it doesn't
>
> Personally I'd just as soon see them go away entirely, but if somebody
> wants to make them work I won't stop them.
>
I'd really not prefer to fly blind, ChangeLogs are awesome for users.

I am a user too. I really would like to know why something changed, and
maybe who did it.

Re: [gentoo-dev] ChangeLog

[Pacho Ramos]

El dom, 01-11-2015 a las 14:25 +0100, Patrick Lauer escribió:
> 
> On 11/01/2015 01:53 PM, Rich Freeman wrote:
> > On Sun, Nov 1, 2015 at 7:33 AM, Мисбах-Соловьёв Вадим <mva@mva.name
> > > wrote:
> > > And why don't just only generate them on rsync mirrors, but
> > > remove them from git repo (like was planned initially, AFAIRC)?
> > > 
> > That is in fact how it works.  Or, at least how it is supposed to
> > work.  I don't use the rsync mirror, so I can't vouch for whether
> > they're producing ChangeLogs.
> Supposed to, but it doesn't
> > 
> > Personally I'd just as soon see them go away entirely, but if
> > somebody
> > wants to make them work I won't stop them.
> > 
> I'd really not prefer to fly blind, ChangeLogs are awesome for users.
> 
> I am a user too. I really would like to know why something changed,
> and
> maybe who did it.
> 

Yeah, I also miss ChangeLogs many times when I want to review last
changes without needing to go to open a browser instance and visit
gitweb and search for the relevant package for the same purpose :|

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/01/2015 01:16 PM, Patrick Lauer wrote:
> Ahoi,
> 
> I'm getting mildly very irritated with the lack of easily accessible
> ChangeLogs for our packages.
> 
> Apparently updating them stopped some time in August, so now there are
> some outdated ChangeLogs that don't really serve any purpose, and the
> easiest way for users to figure out why something changed is to yell at
> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
> patience.
> 
> This does not look reasonable to me.
> 
> Can we please either properly remove ChangeLogs and tell people to not
> be curious about changes, or make them useful again?
> 


ChangeLogs are a deprecated and unreliable method of the times we were
still on CVS. E.g. some people didn't find it useful to add ChangeLog
entries when they did large eclass changes. This problem is gone now.

git log -- app-misc/foo
or
git log -- eclass/autotools.eclass

will give you _any_ commit that has touched that file/directory, even if
it was part of a huge mass commit.

There's really not much ChangeLogs add for you here, except duplicating
git functionality. It's more useful to familiarize yourself with git
log. There's no reason to depend on the gitweb interface.

If you want the history from before the migration to work with that as
well, you can use this method:
https://wiki.gentoo.org/wiki/Gentoo_git_workflow#Grafting_Gentoo_History_Onto_the_Active_Repo

Also see
https://www.atlassian.com/git/tutorials/git-log/filtering-the-commit-history

Re: [gentoo-dev] ChangeLog

[Patrick Lauer]

On 11/01/2015 02:24 PM, hasufell wrote:
> On 11/01/2015 01:16 PM, Patrick Lauer wrote:
>> Ahoi,
>>
>> I'm getting mildly very irritated with the lack of easily accessible
>> ChangeLogs for our packages.
>>
>> Apparently updating them stopped some time in August, so now there are
>> some outdated ChangeLogs that don't really serve any purpose, and the
>> easiest way for users to figure out why something changed is to yell at
>> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
>> patience.
>>
>> This does not look reasonable to me.
>>
>> Can we please either properly remove ChangeLogs and tell people to not
>> be curious about changes, or make them useful again?
>>
>
> ChangeLogs are a deprecated and unreliable method of the times we were
> still on CVS. E.g. some people didn't find it useful to add ChangeLog
> entries when they did large eclass changes. This problem is gone now.
... ?!??!#??$>@%%*%**%@!!!

From the point of view of a user ChangeLogs are VERY VERY USEFUL.

Why would you claim they are deprecated?
>
> git log -- app-misc/foo
> or
> git log -- eclass/autotools.eclass
>
> will give you _any_ commit that has touched that file/directory, even if
> it was part of a huge mass commit.

 $ cd /usr/portage/app-admin/rex/; git log
fatal: Not a git repository (or any of the parent directories): .git

sooo ... ???

>
> There's really not much ChangeLogs add for you here, except duplicating
> git functionality. It's more useful to familiarize yourself with git
> log. There's no reason to depend on the gitweb interface.
>
> If you want the history from before the migration to work with that as
> well, you can use this method:
> https://wiki.gentoo.org/wiki/Gentoo_git_workflow#Grafting_Gentoo_History_Onto_the_Active_Repo
>
> Also see
> https://www.atlassian.com/git/tutorials/git-log/filtering-the-commit-history
>
I have no idea what you are trying to say, but for most users this
advice is not even useless but actively wrong.

With that out of the way,

can we please return to the original discussion?

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/01/2015 02:28 PM, Patrick Lauer wrote:
>>
>> ChangeLogs are a deprecated and unreliable method of the times we were
>> still on CVS. E.g. some people didn't find it useful to add ChangeLog
>> entries when they did large eclass changes. This problem is gone now.
> ... ?!??!#??$>@%%*%**%@!!!
> 
> From the point of view of a user ChangeLogs are VERY VERY USEFUL.
> 
> Why would you claim they are deprecated?

Because they are unreliable and we have a better method.

>>
>> git log -- app-misc/foo
>> or
>> git log -- eclass/autotools.eclass
>>
>> will give you _any_ commit that has touched that file/directory, even if
>> it was part of a huge mass commit.
> 
>  $ cd /usr/portage/app-admin/rex/; git log
> fatal: Not a git repository (or any of the parent directories): .git
> 
> sooo ... ???
> 

You shouldn't use rsync anymore, it is inherently insecure. The git tree
is _properly_ gpg signed so you can verify it's correctness.

With the following portage configuration/hooks, any user can run the
tree directly from git:
https://github.com/hasufell/portage-gentoo-git-config

At some point, rsync schould be deprecated completely.

Re: [gentoo-dev] ChangeLog

[Alexis Ballier]

On Sun, 1 Nov 2015 14:33:07 +0100
hasufell <hasufell@gentoo.org> wrote:
> >>
> >> git log -- app-misc/foo
> >> or
> >> git log -- eclass/autotools.eclass
> >>
> >> will give you _any_ commit that has touched that file/directory,
> >> even if it was part of a huge mass commit.  
> > 
> >  $ cd /usr/portage/app-admin/rex/; git log
> > fatal: Not a git repository (or any of the parent directories): .git
> > 
> > sooo ... ???
> >   
> 
> You shouldn't use rsync anymore, it is inherently insecure. The git
> tree is _properly_ gpg signed so you can verify it's correctness.
> 
> With the following portage configuration/hooks, any user can run the
> tree directly from git:
> https://github.com/hasufell/portage-gentoo-git-config

More secure by fetching metadata cache via rsync ?
Better by running egencache after each sync ?
I don't think so.

> At some point, rsync schould be deprecated completely.


Considering the original plan was to have changelogs auto-generated
from git and still serving the tree via rsync, where's the relevant
discussion and decision about this?

There's no technical reason for not doing this *today*, the only reason
not to is the lack of decision and concrete plan on how to properly
serve what is in the rsync'ed tree and not in gentoo.git.


Until then, we are serving outdated and useless changelogs via rsync
and Patrick's point still holds: Either remove them or serve proper
ones.


Alexis.

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/01/2015 02:47 PM, Alexis Ballier wrote:
> On Sun, 1 Nov 2015 14:33:07 +0100
> hasufell <hasufell@gentoo.org> wrote:
>>>>
>>>> git log -- app-misc/foo
>>>> or
>>>> git log -- eclass/autotools.eclass
>>>>
>>>> will give you _any_ commit that has touched that file/directory,
>>>> even if it was part of a huge mass commit.  
>>>
>>>  $ cd /usr/portage/app-admin/rex/; git log
>>> fatal: Not a git repository (or any of the parent directories): .git
>>>
>>> sooo ... ???
>>>   
>>
>> You shouldn't use rsync anymore, it is inherently insecure. The git
>> tree is _properly_ gpg signed so you can verify it's correctness.
>>
>> With the following portage configuration/hooks, any user can run the
>> tree directly from git:
>> https://github.com/hasufell/portage-gentoo-git-config
> 
> More secure by fetching metadata cache via rsync ?
> Better by running egencache after each sync ?
> I don't think so.
> 

Yes it is. You have the option of generating it yourself (only takes
long for the first time) or fetch it from the following gentoo mirror
instead https://github.com/gentoo-mirror/gentoo

We might adjust those hooks to do that.

>> At some point, rsync schould be deprecated completely.
> 
> 
> Considering the original plan was to have changelogs auto-generated
> from git and still serving the tree via rsync, where's the relevant
> discussion and decision about this?
> 
> There's no technical reason for not doing this *today*, the only reason
> not to is the lack of decision and concrete plan on how to properly
> serve what is in the rsync'ed tree and not in gentoo.git.
> 
> 
> Until then, we are serving outdated and useless changelogs via rsync
> and Patrick's point still holds: Either remove them or serve proper
> ones.
> 

+1 for removing.

Re: [gentoo-dev] ChangeLog

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On Sun, 1 Nov 2015 14:53:20 +0100 hasufell wrote:
> >> You shouldn't use rsync anymore, it is inherently insecure. The git
> >> tree is _properly_ gpg signed so you can verify it's correctness.
> >>
> >> With the following portage configuration/hooks, any user can run the
> >> tree directly from git:
> >> https://github.com/hasufell/portage-gentoo-git-config
> > 
> > More secure by fetching metadata cache via rsync ?
> > Better by running egencache after each sync ?
> > I don't think so.
> > 
> 
> Yes it is.

No, it is not. The whole git tree is insecure and no better than
rsync or CVS in terms of data security because SHA1 is vulnerable.

What we really need for security is GnuPG-signed tree. Right now we
have only signed commits and pushes. This is work in progress if
understand correctly current situation.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/04/2015 09:56 AM, Andrew Savchenko wrote:
> On Sun, 1 Nov 2015 14:53:20 +0100 hasufell wrote:
>>>> You shouldn't use rsync anymore, it is inherently insecure. The git
>>>> tree is _properly_ gpg signed so you can verify it's correctness.
>>>>
>>>> With the following portage configuration/hooks, any user can run the
>>>> tree directly from git:
>>>> https://github.com/hasufell/portage-gentoo-git-config
>>>
>>> More secure by fetching metadata cache via rsync ?
>>> Better by running egencache after each sync ?
>>> I don't think so.
>>>
>>
>> Yes it is.
> 
> No, it is not. The whole git tree is insecure and no better than
> rsync or CVS in terms of data security because SHA1 is vulnerable.
> 

Another one who is confusing _any_ collision with _preimage attack_ ;)

Re: [gentoo-dev] ChangeLog

[Kristian Fiskerstrand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 11/04/2015 05:18 PM, hasufell wrote:
> On 11/04/2015 09:56 AM, Andrew Savchenko wrote:
>> No, it is not. The whole git tree is insecure and no better than 
>> rsync or CVS in terms of data security because SHA1 is
>> vulnerable.
>> 
> 
> Another one who is confusing _any_ collision with _preimage attack_
> ;)
> 

Or even worse, 2nd preimage :) In all seriousness, though, it is
indeed an important distinction. As for OpenPGP signed distribution of
files in rsync as well, it is certainly something I look forwards to
and Gentoo Keys project is working hard on.

- -- 
Kristian Fiskerstrand
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWOjIcAAoJECULev7WN52FivEH/RssmJdQLug2E4B0ZUMUBDum
fp5E4PipD9WBFIfqwK36acp/QoJIAjsQrA6B8bfOoK+AVCryQGbMNlR2OAWZzZrG
ISn3TTsXjfBeyP0ajiFT1qfTe9OvLNpweyB1GUBvq0vnvtDdmET1DO2d2Yxagmyz
41+QtEWw0s3yypinpgyWqkz5ddJxCAnIXPrOVwwdJJx1yRvAP3rnoM7vvoSCjJps
SannPK1ks6ChXtXhEpIX0cHTgm9oXAnn+BhbEGWISuziOfOXmIrBLmPZG9ZYdwEM
vttt3uRXc42VBG4zLgKq0Qc5TtD4IsWtGn+Hm4sNYV3atHPS78LW05h82HrE7Fo=
=63hW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ChangeLog

[Chí-Thanh Christopher Nguyễn]

hasufell schrieb:
> On 11/04/2015 09:56 AM, Andrew Savchenko wrote:
>> No, it is not. The whole git tree is insecure and no better than
>> rsync or CVS in terms of data security because SHA1 is vulnerable.
>>
> Another one who is confusing _any_ collision with _preimage attack_ ;)

While Andrew's view is very pessimistic here, yours is decidedly optimistic.

There is no known computationally feasible preimage attack against MD5, 
still that hash function is broken in serious ways with attacks already 
having real-world consequences.

It would be quite naïve to assume that SHA1 will remain secure until a 
preimage attack is found.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/04/2015 05:33 PM, Chí-Thanh Christopher Nguyễn wrote:
> hasufell schrieb:
>> On 11/04/2015 09:56 AM, Andrew Savchenko wrote:
>>> No, it is not. The whole git tree is insecure and no better than
>>> rsync or CVS in terms of data security because SHA1 is vulnerable.
>>>
>> Another one who is confusing _any_ collision with _preimage attack_ ;)
> 
> While Andrew's view is very pessimistic here, yours is decidedly
> optimistic.
> 
> There is no known computationally feasible preimage attack against MD5,
> still that hash function is broken in serious ways with attacks already
> having real-world consequences.
> 
> It would be quite naïve to assume that SHA1 will remain secure until a
> preimage attack is found.
> 

I didn't. Numerous crypto-analysts have already expressed that SHA-1 is
not future-proof.

However, saying "it is vulnerable" is simply exaggeration and suggests
people should do the math before posting such things.

We already had that discussion before the git migration and it is quite
pointless. If you want to improve the situation, go talk to git upstream
and send patches.

Re: [gentoo-dev] ChangeLog

[Chí-Thanh Christopher Nguyễn]

hasufell schrieb:

> If you want to improve the situation, go talk to git upstream
> and send patches.

Or do what Andrew suggested should happen.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/04/2015 05:44 PM, Chí-Thanh Christopher Nguyễn wrote:
> hasufell schrieb:
> 
>> If you want to improve the situation, go talk to git upstream
>> and send patches.
> 
> Or do what Andrew suggested should happen.
> 
> 

If you want to break the whole git workflow yes. Good suggestion.

Re: [gentoo-dev] ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 8:47 AM, Alexis Ballier <aballier@gentoo.org> wrote:
> Considering the original plan was to have changelogs auto-generated
> from git and still serving the tree via rsync, where's the relevant
> discussion and decision about this?

What discussion or decision is necessary?  As far as I'm aware nobody
has forbidden making changelogs available via rsync.  It just sounds
like there is a bug in their generation.  What is needed is for those
who want changelogs to fix the bug, not endless discussion.  If the
issue is infra access, just make your own mirror with working code and
I'm sure infra will borrow it, and if not nobody really HAS to use
infra.  I'm syncing from the github mirror that contains pre-generated
metadata for convenience, which is just one of the many options
available.

Nobody is actively preventing anybody from having what they want.
This isn't some corporation where we are paying people and we can
demand that the responsible parties fix things or lose their jobs.

Lots of effort has gone into making the git migration as seamless as
possible, but it was bound to be imperfect.  Personally I would have
been fine with less effort being spent on it than actually was.
Gentoo has never been a hand-holding distro.  I have nothing against
people who choose to invest their time into making it more helpful to
those who wish to use alternate tools (like changelogs), but I don't
favor telling those who are working on new features to not actually
deploy them unless THEY spend their time on such things as long as we
have a reasonable path forward for everybody.

So, if you want to see what has changed there are half a dozen ways of
doing it without using changelogs.

-- 
Rich

Re: [gentoo-dev] ChangeLog

[Alexis Ballier]

On Sun, 1 Nov 2015 09:19:25 -0500
Rich Freeman <rich0@gentoo.org> wrote:

[...]
> What discussion or decision is necessary?

One that announces the initial and current plan has changed and
describes the new plan maybe?

[...]
> So, if you want to see what has changed there are half a dozen ways of
> doing it without using changelogs.


I imagine the 'you' in your long tirade is a generic 'you', otherwise,
let me make this clear: I'm in favor of removing changelogs. They made
sense with the per-file tracking of cvs, git made them much less useful.

Re: [gentoo-dev] ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 10:00 AM, Alexis Ballier <aballier@gentoo.org> wrote:
> On Sun, 1 Nov 2015 09:19:25 -0500
> Rich Freeman <rich0@gentoo.org> wrote:
>
> [...]
>> What discussion or decision is necessary?
>
> One that announces the initial and current plan has changed and
> describes the new plan maybe?
>

I haven't heard anybody propose a new plan.  I certainly am not proposing one.

Like I said, I don't have a problem with there being Changelogs in
rsync.  By all means go fix it.

-- 
Rich

Re: [gentoo-dev] ChangeLog

[Alexis Ballier]

On Sun, 1 Nov 2015 10:17:54 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Sun, Nov 1, 2015 at 10:00 AM, Alexis Ballier <aballier@gentoo.org>
> wrote:
> > On Sun, 1 Nov 2015 09:19:25 -0500
> > Rich Freeman <rich0@gentoo.org> wrote:
> >
> > [...]  
> >> What discussion or decision is necessary?  
> >
> > One that announces the initial and current plan has changed and
> > describes the new plan maybe?
> >  
> 
> I haven't heard anybody propose a new plan.  I certainly am not
> proposing one.

The part you cut:

> 
> You shouldn't use rsync anymore, it is inherently insecure. The git
> tree is _properly_ gpg signed so you can verify it's correctness.
> 
> With the following portage configuration/hooks, any user can run the
> tree directly from git:
> https://github.com/hasufell/portage-gentoo-git-config  

Re: [gentoo-dev] ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 10:24 AM, Alexis Ballier <aballier@gentoo.org> wrote:
> On Sun, 1 Nov 2015 10:17:54 -0500
> Rich Freeman <rich0@gentoo.org> wrote:
>>
>> I haven't heard anybody propose a new plan.  I certainly am not
>> proposing one.
>
> The part you cut:
>
>>
>> You shouldn't use rsync anymore, it is inherently insecure. The git
>> tree is _properly_ gpg signed so you can verify it's correctness.
>>

That was just a random developer offering random advice.  People are
welcome to do that on the lists.  Nobody is preventing anybody from
fixing the bug.

There is no approved grandiose plan to obsolete rsync.

-- 
Rich

Re: [gentoo-dev] ChangeLog

[Alexis Ballier]

On Sun, 1 Nov 2015 12:26:04 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Sun, Nov 1, 2015 at 10:24 AM, Alexis Ballier <aballier@gentoo.org>
> wrote:
> > On Sun, 1 Nov 2015 10:17:54 -0500
> > Rich Freeman <rich0@gentoo.org> wrote:  
> >>
> >> I haven't heard anybody propose a new plan.  I certainly am not
> >> proposing one.  
> >
> > The part you cut:
> >  
> >>
> >> You shouldn't use rsync anymore, it is inherently insecure. The git
> >> tree is _properly_ gpg signed so you can verify it's correctness.
> >>  
> 
> That was just a random developer offering random advice.  People are
> welcome to do that on the lists.  Nobody is preventing anybody from
> fixing the bug.
> 
> There is no approved grandiose plan to obsolete rsync.

Hence me asking where the discussion took place...
Y'know, the email you replied to :)

[gentoo-dev] Re: ChangeLog

[Martin Vaeth]

Rich Freeman <rich0@gentoo.org> wrote:
>
> What discussion or decision is necessary?
> What is needed is for those who want changelogs
> to fix the bug

The bug can only be fixed by somebody who knows
the details how the rsync mirrors are set up.

As mentioned in the discussion in bug 561454,
*essentially* all it needs to fix the issue is to call

egencache --repo=gentoo --update-changelog

on the main server providing the data for the mirrors,
but details (e.g. when to call it) depend on how
exactly the server is organized.

For instance, it might be necessary to put
ChangeLog
/.gitignore
into a local .gitignore file or something similar,
and somehow the Manifests due to modified ChangeLogs
might need to be updated, too.

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 10:29 AM, Martin Vaeth <martin@mvath.de> wrote:
> Rich Freeman <rich0@gentoo.org> wrote:
>>
>> What discussion or decision is necessary?
>> What is needed is for those who want changelogs
>> to fix the bug
>
> The bug can only be fixed by somebody who knows
> the details how the rsync mirrors are set up.

And that is really the bigger problem here.  I think we really need to
minimize dependency on stuff that only a few people have access to,
since they're overworked.  I'd certainly encourage somebody to offer
up a fixed rsync server.  While you're at it, a gitlab/whatever server
so that we don't have to keep arguing over github would also be nice.

I'd really like to see our infrastructure get to a point where it is
all published FOSS minus maybe a few authentication tokens so that
anybody can just fork it on demand.  Even though we'd probably still
want the officially-designated servers it would make it far easier for
others to contribute if they could actually test it all out.
Authentication could use openid or whatever so that a clone could be a
first-class alternative.

In any case though, this isn't some vast conspiracy to get rid of
Changelogs.  More likely there are only a few people who can fix them,
and they simply haven't gotten around to it.

-- 
Rich

Re: [gentoo-dev] ChangeLog

[Мисбах-Соловьёв Вадим]

> You shouldn't use rsync anymore, it is inherently insecure. The git tree
> is _properly_ gpg signed so you can verify it's correctness.
>
> With the following portage configuration/hooks, any user can run the
> tree directly from git:
> https://github.com/hasufell/portage-gentoo-git-config
>
> At some point, rsync schould be deprecated completely.

Nice try, but sometimes (say, in case of very unstable internet connection) it is IMPOSSIBLE to properly sync git repo (because it tries to fetch all the queue (between local HEAD and remote one) in a row, from the exactly same place every time it fails), while it still possible to sync tree by rsync, because it only fetches differences and do not drop properly downloaded files.

Do NOT (!) assume flawless shiny 40Gbit internet everywhere in the world. Stop being so selfish and thinking only from position of the developer. Always do imagine you as the gentoo user in the worst possible case.
Please.

Re: [gentoo-dev] ChangeLog

[hasufell]

On 11/01/2015 02:51 PM, Мисбах-Соловьёв Вадим wrote:
>> You shouldn't use rsync anymore, it is inherently insecure. The git tree
>> is _properly_ gpg signed so you can verify it's correctness.
>>
>> With the following portage configuration/hooks, any user can run the
>> tree directly from git:
>> https://github.com/hasufell/portage-gentoo-git-config
>>
>> At some point, rsync schould be deprecated completely.
> 
> Nice try, but sometimes (say, in case of very unstable internet connection) it is IMPOSSIBLE to properly sync git repo (because it tries to fetch all the queue (between local HEAD and remote one) in a row, from the exactly same place every time it fails), while it still possible to sync tree by rsync, because it only fetches differences and do not drop properly downloaded files.
> 
> Do NOT (!) assume flawless shiny 40Gbit internet everywhere in the world. Stop being so selfish and thinking only from position of the developer. Always do imagine you as the gentoo user in the worst possible case.
> Please.
> 

git clone --depth=1 (you can also put that into your repos.conf, the
option is called 'sync-depth'). --depth is also available for regular pulls.

And thanks for calling me selfish for trying to make it possible for
users to directly use a git checkout as the local tree. I appreciate it. :)

[gentoo-dev] Re: ChangeLog

[Martin Vaeth]

hasufell <hasufell@gentoo.org> wrote:
>
> git clone --depth=1

The only reasonable option for the gentoo user
(not for the gentoo developer) if he does not have
megabytes to waste on his harddisk (which probably
many users don't), if you want to *force* him
to use git.

Now how can this user display the ChangeLog for
a certain package?

Re: [gentoo-dev] Re: ChangeLog

[Мисбах-Соловьёв Вадим]

> Now how can this user display the ChangeLog for
> a certain package?
```
git log -- pkg-category/pkg-name/
```
// assuming user in portage directory or passed GIT_DIR with path to it.

Although, it is only if user has successfully cloned/synced the repo ;)
Which is hardcore quest when you're living in, say, Syria, Egypt, Somalia, or something like that. Or, if you're, say, in transsiberian train ride for a week.

Re: [gentoo-dev] Re: ChangeLog

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Sun, 01 Nov 2015 22:19:24 +0600
Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
> > Now how can this user display the ChangeLog for
> > a certain package?
> ```
> git log -- pkg-category/pkg-name/
> ```
> // assuming user in portage directory or passed GIT_DIR with path to
> it.
> 
> Although, it is only if user has successfully cloned/synced the
> repo ;) Which is hardcore quest when you're living in, say, Syria,
> Egypt, Somalia, or something like that. Or, if you're, say, in
> transsiberian train ride for a week.

Perhaps there is a better choice of distribution for you if you are.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: ChangeLog

[Мисбах-Соловьёв Вадим]

> Perhaps there is a better choice of distribution for you if you are.

Or you can just... use rsync.

[gentoo-dev] Re: ChangeLog

[Martin Vaeth]

Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
>> Perhaps there is a better choice of distribution for you if you are.
>
> Or you can just... use rsync.

Or emerge-webrsync, emerge-delta-webrsync or squashdelta
(I strongly hope that the latter will be available again
in some future - IMHO it is perfect for most users).

Please do not break all these possibilities for users
who do not have to waste the resources for a full git
clone and want to see regularly ChangeLogs nevertheless!

Re: [gentoo-dev] Re: ChangeLog

[Kristian Fiskerstrand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 11/01/2015 09:33 PM, Martin Vaeth wrote:
> Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
>>> Perhaps there is a better choice of distribution for you if you
>>> are.
>> 
>> Or you can just... use rsync.
> 
> Or emerge-webrsync, emerge-delta-webrsync or squashdelta (I
> strongly hope that the latter will be available again in some
> future - IMHO it is perfect for most users).
> 
> Please do not break all these possibilities for users who do not
> have to waste the resources for a full git clone and want to see
> regularly ChangeLogs nevertheless!
> 
> 


I wouldn't be too worried about that, although I do hope that we get
proper OpenPGP signing[0] into the main tree sooner rather than later.

References:
[0] https://wiki.gentoo.org/wiki/Project:Gentoo-keys

- -- 
Kristian Fiskerstrand
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJWNngoAAoJECULev7WN52F+g0H+wW9EuEbz6xwg+4s/lET8uau
86WOJQTFSAuvYuaqVT0gwbY0v+0CCDg+nsaIRnl+jRJWcFuZ9Wbu1H/ehWaJ5sVK
a1Q1aTakuc/szMZS3zQpykUG1PUNkyCXtKxA5ubsUcagPfBD7yjkFZ9/CrHiIUoF
u8oXi+hGCOpIpS6hjizMUuBiJTiXu3cbb7/BKqZ2Lz234kumfMAPDd95CYMCaayC
33NFQitP1d1dhk0z9O5LrrIyHBrXbQF/hbTY/oxsuZ95Cn4JdmU2GUQriHzvraPy
iZGgYRxUXPh/0PQNsdSV4YIWUb9CwmA8vfkLnND+hs3haE3U63uBUjiMDznIukA=
=lFUN
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 3:33 PM, Martin Vaeth <martin@mvath.de> wrote:
> Please do not break all these possibilities for users
> who do not have to waste the resources for a full git
> clone and want to see regularly ChangeLogs nevertheless!

I don't think anybody has proposed breaking anything.  It sounds like
it is already broken, and somebody needs to fix it.

Keep in mind that "resources" is a vague term and for some resources
either git or rsync can be cheaper.  Rsync will tend to require less
local disk space than git (unless you try to purge all the history out
of the repositories, which of course defeats the goal of having
Changelogs).  On the other hand, git should require far less disk io
and CPU to sync since it doesn't have to rely on stat-ing every file
(and if you want rsync to be truly reliable you need to tell it to
hash everything which adds a boatload of additional IO - git doesn't
rely on mtime).

-- 
Rich

[gentoo-dev] Re: ChangeLog

[Martin Vaeth]

Rich Freeman <rich0@gentoo.org> wrote:
>
> Keep in mind that "resources" is a vague term [...]
> disk io and CPU to sync [...]

For syncing, I think these latter resources are less important,
because they influence only the *time* of a syncing action,
which is normally not so important for the user.

Bandwidth and (effective) harddisk space are more crucial IMHO.
For the latter, we have a _clear_ winner: squashdelta
(which BTW could also be signed).

Concerning bandwidth, comparison is harder - it varies for
the different methods dramatically on how often you sync -
but squashdelta is also not bad here.

At least the mentioned problems with connection loss can
be solved easily for squashdelta, too, because only one
file has to be transferred whose transfer might be resumed.

[gentoo-dev] Re: ChangeLog

[Martin Vaeth]

Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
>
>> Now how can this user display the ChangeLog for
>> a certain package?
> ```
> git log -- pkg-category/pkg-name/

You removed the crucial part of my posting:

>> git clone --depth=1

Re: [gentoo-dev] Re: ChangeLog

[Tobias Klausmann]

Hi! 

On Sun, 01 Nov 2015, Martin Vaeth wrote:
> Мисбах-Соловьёв Вадим <mva@mva.name> wrote:
> >
> >> Now how can this user display the ChangeLog for
> >> a certain package?
> > ```
> > git log -- pkg-category/pkg-name/
> 
> You removed the crucial part of my posting:
> 
> >> git clone --depth=1

For the combined purpose of "quick checkout", "low local disk
usage" and "I can look at the logs", that doesn't quite work,
though:


$ git clone --depth=1 git+ssh://git@git.gentoo.org/repo/gentoo.git
Cloning into 'gentoo'...
[...]
Checking out files: 100% (100227/100227), done.
$ cd gentoo/
gentoo (master) $ git log |wc -l
7
$

Regards,
Tobias




-- 
Sent from aboard the Culture ship
	GCU Very Little Gravitas Indeed

Re: [gentoo-dev] Re: ChangeLog

[Chí-Thanh Christopher Nguyễn]

Мисбах-Соловьёв Вадим schrieb:
>>> git clone --depth=1 
>> Now how can this user display the ChangeLog for
>> a certain package?
> ```
> git log -- pkg-category/pkg-name/

I think his point was that shallow clones don't have the full log.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] ChangeLog

[Мисбах-Соловьёв Вадим]

> git clone --depth=1 (you can also put that into your repos.conf, the
> option is called 'sync-depth'). --depth is also available for regular pulls.
```
$ LC_ALL=C git clone --depth=1 git://git.gentoo.org/repo/gentoo.git
Cloning into 'gentoo'...
remote: Counting objects: 113359, done.
remote: Compressing objects: 100% (99921/99921), done.
Receiving objects:  17% (19272/113359), 7.32 MiB | 30 KiB/s
# <connection freezed here and dropped by user (after "ethernity" of waiting) or by timeout (although I tried to wait almost hour, and can't get it dropped by itself, but hear about that from another people with bad internet connection>

$ LC_ALL=C ls gentoo
ls: cannot access gentoo: No such file or directory
```

So? How should I sync then? Git can *not* sync file by file, while rsync can.

> And thanks for calling me selfish for trying to make it possible for
> users to directly use a git checkout as the local tree. I appreciate it. :)

No. Making it *possible* for all users to use git (instead of old cvs+rsync behaviour) is a great thing and you're awesome in making it *possible*. But do not be authoritarian and do not *force* (!) users to "use only git and forget about rsync".

I called you selfish not for doing good things like introducing possibility to use git, but for authoritarian thinking, that all users has as good internet connection (taking that thread) or tech. knowledge (talking prev. threads) as you. They are not. At least, far not always.

Re: [gentoo-dev] ChangeLog

[Michael Orlitzky]

On 11/01/2015 07:16 AM, Patrick Lauer wrote:
> Ahoi,
> 
> I'm getting mildly very irritated with the lack of easily accessible
> ChangeLogs for our packages.
> 
> Apparently updating them stopped some time in August, so now there are
> some outdated ChangeLogs that don't really serve any purpose, and the
> easiest way for users to figure out why something changed is to yell at
> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
> patience.
> 

How about e.g. https://packages.gentoo.org/packages/dev-lang/php ?

[gentoo-dev] Re: ChangeLog

[Duncan]

Patrick Lauer posted on Sun, 01 Nov 2015 13:16:31 +0100 as excerpted:

> I'm getting mildly very irritated with the lack of easily accessible
> ChangeLogs for our packages.
> 
> Apparently updating them stopped some time in August, so now there are
> some outdated ChangeLogs that don't really serve any purpose, and the
> easiest way for users to figure out why something changed is to yell at
> the clumsy gitweb.g.o interface. So instead of grep we now need lots of
> patience.
> 
> This does not look reasonable to me.
> 
> Can we please either properly remove ChangeLogs and tell people to not
> be curious about changes, or make them useful again?

++

I got irritated with that within a week of the git switch, and pretty 
quickly switched to the github mirror with pregenerated metadata.

That gives me the git log, which is perfectly fine by me. =:^)

But, that's not at all suitable for the general user.

For one thing, while one would hope that github should have no problem 
with the traffic, there's little question that recommending that as the 
general gentoo-wide solution runs headlong into the social-contract issue 
currently being debated on the NFP list.  While simply taking commits via 
github from users who want to use it for that is one thing (and fine by 
me, I'm not /that/ extreme, as long as the proprietary method doesn't 
come to be favored), having pretty much the entire userbase syncing from 
it would be pretty hard *not* to call a "dependency", and thus a 
violation of the contract.

Which means the generally recommended sync method really needs to stay on 
gentoo infrastructure, or at least with direct gentoo volunteer hosts, 
tho some may be hosting on proprietary platforms.


Personally, I'd love the primary sync method to be git, and the primary 
change logs conveyed via native git log methods, but in ordered to do 
that, all those rsync mirrors need to switch to git mirrors.

Meanwhile, as pointed out elsewhere in the thread, git syncing simply 
won't work under some circumstances, as it's all or nothing, and for some 
people the connection simply isn't stable enough to get all, so it ends 
up being nothing.  

But as long as webrsync continues to work, and assuming it can be 
incrementally fetched/resumed, that should be a sufficient solution for 
them, so again, rsync isn't really necessary.  And for those users, if 
they do need to check the changelog, the packages.gentoo.org method, 
following the link to the git log if necessary, should be fine.


But clearly, something needs to change.  Either the changelogs need to be 
fixed to be generated again, or the old ones need removed, with people 
not using git pointed at packages.gentoo.org as mentioned.  If the 
general rsync mirrors are eventually switched to git, with webrsync 
remaining for those who can't do git, great, but meanwhile, I know if I 
were still on rsync (or webrsync), I'd be raising hell about the lack of 
changelogs well before now, and the github mirror I'm actually using 
clearly isn't suited as general gentoo user solution because it /would/ 
then be a gentoo dependency and thus a violation of the social contract, 
so I'm definitely with Patrick on this one.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Sun, Nov 1, 2015 at 8:22 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>  I know if I were still on rsync (or webrsync), I'd be raising hell about the lack of
> changelogs well before now

Perhaps rather than raising hell you'd do better to raise money to
hire an infra team to fix the bug or something.

I get the frustration, but we only have a few people who have the
necessary access to fix the problem.  Infra is also a difficult
project to deal with in general because it is fairly closed due to the
implications of having random people messing with it.  I don't really
see anybody stepping up to try to change anything fundamental about it
either.  This isn't the sort of thing that will get better if the
council votes on something.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Rich Freeman wrote:
> On Sun, Nov 1, 2015 at 8:22 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>  I know if I were still on rsync (or webrsync), I'd be raising hell about the lack of
>> changelogs well before now
> Perhaps rather than raising hell you'd do better to raise money to
> hire an infra team to fix the bug or something.
>
> I get the frustration, but we only have a few people who have the
> necessary access to fix the problem.  Infra is also a difficult
> project to deal with in general because it is fairly closed due to the
> implications of having random people messing with it.  I don't really
> see anybody stepping up to try to change anything fundamental about it
> either.  This isn't the sort of thing that will get better if the
> council votes on something.
>


Then perhaps all this should have been worked out BEFORE switching to
github? 

As a user, I would look at the change logs pretty regular, more than the
ebuilds to be honest.  Now, there is none.  If a package changes, I have
no clue why it changed unless I go dig that information out somewhere
and that somewhere doesn't seem to be in one place.  When I tried to dig
some info out a while back, I found some on github thingy and then some
more on gentoo.org itself.  I'm still not sure what change lead to what
because there is no real order of events that I could see.  This was
shortly after the change.  After that, screw it. 

I don't mind change but it seem this one wasn't really ready to be done
yet although most made it sound like it was.  I been using Gentoo since
2003, the 1.4 days, and even I can't figure out where to find
information easily and I have a stable DSL connection.  I feel real
sorry for people who don't have one.  I might add, I had a really
limited dial-up connection when I first started using Gentoo so I know
how it is to be in that situation.  I haven't forgot those days. 

Going back to my hole for the simple reason, it's screwed up and no one
seems to think it worth fixing.  I noticed that as soon as I saw the you
need to figure out a way to fix it yourself comment.  One thing about
being around so long, when you see that comment, you may as well kiss it
good bye.  That's code for we aren't going to fix it, you figure out a
way for yourself.  It's rare that anything gets fixed after that. 

Dale

:-)  :-) 

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>
> Then perhaps all this should have been worked out BEFORE switching to
> github?

We didn't switch to github.

>
> I don't mind change but it seem this one wasn't really ready to be done
> yet although most made it sound like it was.

IMO we took too long as it is.  I'd have pushed it faster, but just as
I don't have access to fix this bug I didn't have access to implement
git, and those with access wanted to keep Changelogs, so we did.

If you haven't already gotten the impression, Gentoo is mostly a
conglomeration of various associations of devs who tend to do their
own thing, each with different sets of knowledge/access.  When there
is a direct conflict between two groups we generally (if often not
enthusiastically) accept the votes of the Council to settle disputes.
For the most part the Council has the ability to tell devs to NOT do
things.  When something needs to be done somebody has to step up and
do it.  Complaining on the lists mainly seems to de-motivate people
from working on anything.  By all means point out that something isn't
working, but attitudes tend to be contagious.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Rich Freeman wrote:
> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>> Then perhaps all this should have been worked out BEFORE switching to
>> github?
> We didn't switch to github.

Then why are people saying to use git to look at the logs?   I don't
want to use git.  I liked being able to go to the tree and look at the
change logs when I needed to which is sometimes often.  Now, that is
gone because it is somewhere else and people are saying to use git to
get them.  Sounds like a switch to me.  


>
>> I don't mind change but it seem this one wasn't really ready to be done
>> yet although most made it sound like it was.
> IMO we took too long as it is.  I'd have pushed it faster, but just as
> I don't have access to fix this bug I didn't have access to implement
> git, and those with access wanted to keep Changelogs, so we did.
>
> If you haven't already gotten the impression, Gentoo is mostly a
> conglomeration of various associations of devs who tend to do their
> own thing, each with different sets of knowledge/access.  When there
> is a direct conflict between two groups we generally (if often not
> enthusiastically) accept the votes of the Council to settle disputes.
> For the most part the Council has the ability to tell devs to NOT do
> things.  When something needs to be done somebody has to step up and
> do it.  Complaining on the lists mainly seems to de-motivate people
> from working on anything.  By all means point out that something isn't
> working, but attitudes tend to be contagious.
>


Yea, breaking things does tend to start attitudes when the people that
break them act like it isn't their problem to fix when they broke it.  I
didn't do anything to remove the logs.  That wasn't me and it wasn't a
few others that want/need them back either.  Argue and point the finger
somewhere else if you want but I didn't create the problem, the devs
that supported this move did and it seems it wasn't as well thought as
some of us users was lead to believe.  How about fixing it now.   I
might add, this makes me very leery of trusting changes like this in the
future.  If something as important as logs can't be thought about before
doing something like this, then what else can be overlooked in the
future?  As it is, it seems like that could be a lot. 

Dale

:-)  :-) 

Re: [gentoo-dev] Re: ChangeLog

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

On Mon, 2 Nov 2015 14:00:18 -0600
Dale <rdalek1967@gmail.com> wrote:
> Rich Freeman wrote:
> > On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
> >> Then perhaps all this should have been worked out BEFORE switching
> >> to github?
> > We didn't switch to github.
> 
> Then why are people saying to use git to look at the logs?   I don't
> want to use git.

git != github...

> I liked being able to go to the tree and look at the
> change logs when I needed to which is sometimes often.

I think you have a technology comprehension problem here, rather than a
technology problem. The problem is your workflow, not the tools.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Ciaran McCreesh wrote:
> On Mon, 2 Nov 2015 14:00:18 -0600
> Dale <rdalek1967@gmail.com> wrote:
>> Rich Freeman wrote:
>>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>>>> Then perhaps all this should have been worked out BEFORE switching
>>>> to github?
>>> We didn't switch to github.
>> Then why are people saying to use git to look at the logs?   I don't
>> want to use git.
> git != github...
>
>> I liked being able to go to the tree and look at the
>> change logs when I needed to which is sometimes often.
> I think you have a technology comprehension problem here, rather than a
> technology problem. The problem is your workflow, not the tools.
>

What I understand is this.  The logs since github started being used are
no longer updated like they used to be.  I realize that git is a command
but it seems that git gets its info from github since the logs are no
longer downloaded when I sync the tree.  It was really nice to be able
to see those logs and be able to find out what changed especially if you
want to hold off on a update that isn't important.  Generally, the only
way to know if it is important or not is to look at the logs and see why
something is being changed.   Also, I don't want to turn into Duncan and
write a book to be sure that every specific detail is covered.  I try to
keep it short and basic. 

If the problem is me not understanding the problem then why are others
complaining about the missing logs too?  If it was just me then I would
be the only one complaining about them being no longer updated. 
Obviously, it is not just me.  Well, obvious to some but maybe not some
others.  We just want the logs back so we can see what is going on and
for it to be easily available, preferably done with the sync like it
used to be.  The recent changes broke that and it needs to be fixed. 

Dale

:-)  :-) 

Re: [gentoo-dev] Re: ChangeLog

[hasufell]

On 11/02/2015 10:54 PM, Dale wrote:
> Ciaran McCreesh wrote:
>> On Mon, 2 Nov 2015 14:00:18 -0600
>> Dale <rdalek1967@gmail.com> wrote:
>>> Rich Freeman wrote:
>>>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>>>>> Then perhaps all this should have been worked out BEFORE switching
>>>>> to github?
>>>> We didn't switch to github.
>>> Then why are people saying to use git to look at the logs?   I don't
>>> want to use git.
>> git != github...
>>
>>> I liked being able to go to the tree and look at the
>>> change logs when I needed to which is sometimes often.
>> I think you have a technology comprehension problem here, rather than a
>> technology problem. The problem is your workflow, not the tools.
>>
> 
> What I understand is this.  The logs since github started being used are
> no longer updated like they used to be.  I realize that git is a command
> but it seems that git gets its info from github since the logs are no
> longer downloaded when I sync the tree.

No. You should really read some tutorials/introductions on git to
understand how it works and this ML isn't the place for that. It's not
like SVN, so you don't even need an internet connection for the tree to
work (e.g. if you've copied it via USB even). Github is totally
unrelated here.

Re: [gentoo-dev] Re: ChangeLog

[Dale]

hasufell wrote:
> On 11/02/2015 10:54 PM, Dale wrote:
>> Ciaran McCreesh wrote:
>>> On Mon, 2 Nov 2015 14:00:18 -0600
>>> Dale <rdalek1967@gmail.com> wrote:
>>>> Rich Freeman wrote:
>>>>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>>>>>> Then perhaps all this should have been worked out BEFORE switching
>>>>>> to github?
>>>>> We didn't switch to github.
>>>> Then why are people saying to use git to look at the logs?   I don't
>>>> want to use git.
>>> git != github...
>>>
>>>> I liked being able to go to the tree and look at the
>>>> change logs when I needed to which is sometimes often.
>>> I think you have a technology comprehension problem here, rather than a
>>> technology problem. The problem is your workflow, not the tools.
>>>
>> What I understand is this.  The logs since github started being used are
>> no longer updated like they used to be.  I realize that git is a command
>> but it seems that git gets its info from github since the logs are no
>> longer downloaded when I sync the tree.
> No. You should really read some tutorials/introductions on git to
> understand how it works and this ML isn't the place for that. It's not
> like SVN, so you don't even need an internet connection for the tree to
> work (e.g. if you've copied it via USB even). Github is totally
> unrelated here.
>
>


So you are telling me that people using github and the switch that took
place has absolutely nothing to do with the changelogs going dead? 
Before I sent my first message here, I looked at the changelogs for
several packages in the portage tree  that I know for certain have been
updated.  Just as the OP stated in his email, those updates to those
changelogs stopped in August.  That is about the same time this
git/github happened.  While I do believe the git/github change has
helped in a lot of ways, some of us still want our changelogs up to date
so that we can see them and know what is going on, especially if we run
into trouble and need to know what changed and when and need it to fix
things. 

I find it odd that I have a problem, others are having the same
complaint but it is me not wanting to use git/github that is the
problem.  I thought Gentoo was not depending on git/github either. 

Dale

:-)  :-)

Re: [gentoo-dev] Re: ChangeLog

[Matt Turner]

On Mon, Nov 2, 2015 at 5:20 PM, Dale <rdalek1967@gmail.com> wrote:
> hasufell wrote:
>> On 11/02/2015 10:54 PM, Dale wrote:
>>> Ciaran McCreesh wrote:
>>>> On Mon, 2 Nov 2015 14:00:18 -0600
>>>> Dale <rdalek1967@gmail.com> wrote:
>>>>> Rich Freeman wrote:
>>>>>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>>>>>>> Then perhaps all this should have been worked out BEFORE switching
>>>>>>> to github?
>>>>>> We didn't switch to github.
>>>>> Then why are people saying to use git to look at the logs?   I don't
>>>>> want to use git.
>>>> git != github...
>>>>
>>>>> I liked being able to go to the tree and look at the
>>>>> change logs when I needed to which is sometimes often.
>>>> I think you have a technology comprehension problem here, rather than a
>>>> technology problem. The problem is your workflow, not the tools.
>>>>
>>> What I understand is this.  The logs since github started being used are
>>> no longer updated like they used to be.  I realize that git is a command
>>> but it seems that git gets its info from github since the logs are no
>>> longer downloaded when I sync the tree.
>> No. You should really read some tutorials/introductions on git to
>> understand how it works and this ML isn't the place for that. It's not
>> like SVN, so you don't even need an internet connection for the tree to
>> work (e.g. if you've copied it via USB even). Github is totally
>> unrelated here.
>>
>>
>
>
> So you are telling me that people using github and the switch that took
> place has absolutely nothing to do with the changelogs going dead?

You keep saying GitHub. Github is not relevant to this discussion.

Yes, the ChangeLogs stopped being updated because of git.

The git transition had been 9 years in the making and has massively
improved Gentoo development. Look at the graph of contributions per
month: https://www.openhub.net/p/gentoo

It was decided that the missing infrastructure for ChangeLogs was not
a sufficient reason to block a hugely important change.

People care about finishing this, as evidence by this thread. Please
stop complaining.

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Matt Turner wrote:
> On Mon, Nov 2, 2015 at 5:20 PM, Dale <rdalek1967@gmail.com> wrote:
>> hasufell wrote:
>>> On 11/02/2015 10:54 PM, Dale wrote:
>>>> Ciaran McCreesh wrote:
>>>>> On Mon, 2 Nov 2015 14:00:18 -0600
>>>>> Dale <rdalek1967@gmail.com> wrote:
>>>>>> Rich Freeman wrote:
>>>>>>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com> wrote:
>>>>>>>> Then perhaps all this should have been worked out BEFORE switching
>>>>>>>> to github?
>>>>>>> We didn't switch to github.
>>>>>> Then why are people saying to use git to look at the logs?   I don't
>>>>>> want to use git.
>>>>> git != github...
>>>>>
>>>>>> I liked being able to go to the tree and look at the
>>>>>> change logs when I needed to which is sometimes often.
>>>>> I think you have a technology comprehension problem here, rather than a
>>>>> technology problem. The problem is your workflow, not the tools.
>>>>>
>>>> What I understand is this.  The logs since github started being used are
>>>> no longer updated like they used to be.  I realize that git is a command
>>>> but it seems that git gets its info from github since the logs are no
>>>> longer downloaded when I sync the tree.
>>> No. You should really read some tutorials/introductions on git to
>>> understand how it works and this ML isn't the place for that. It's not
>>> like SVN, so you don't even need an internet connection for the tree to
>>> work (e.g. if you've copied it via USB even). Github is totally
>>> unrelated here.
>>>
>>>
>>
>> So you are telling me that people using github and the switch that took
>> place has absolutely nothing to do with the changelogs going dead?
> You keep saying GitHub. Github is not relevant to this discussion.
>
> Yes, the ChangeLogs stopped being updated because of git.
>
> The git transition had been 9 years in the making and has massively
> improved Gentoo development. Look at the graph of contributions per
> month: https://www.openhub.net/p/gentoo
>
> It was decided that the missing infrastructure for ChangeLogs was not
> a sufficient reason to block a hugely important change.
>
> People care about finishing this, as evidence by this thread. Please
> stop complaining.
>
>


What bothers me is that people make it sound like it is ONLY me that is
having this issue when I have seen it posted here and on -user as well
by others.  I agree that the change has improved things because even tho
I don't update as often as some do, I see more updates.  I'm not arguing
that point for sure.  Even I can see that improvement.  I just ask that
people not point at me like it is only me that has a problem when even
Robin replied that the problem is known about and some changes seem to
be coming, hopefully soon since I miss the changelogs. 

I also find it odd that early on, I was told to go to the github website
to get info regarding changes to Gentoo packages.  I even bookmarked the
link.  Even package.gentoo.org was out of date for a while.  Thing is,
that doesn't match what I have locally because that includes changes
made since my last sync.  It's not easy to find out what applies to me
locally since they are not in sync.  Having it in the tree like it once
was means they are synced at the same time and the info there matches
what is in the tree locally because they get updated together. 

As it is, if it is being worked on, fine.  Just don't tell me that I'm
the only one with outdated changelogs and that I don't know the
difference between a software package and a website.  I've been to
gituhub, as I was told to, and I know about the git software package
too.  I just prefer not to use the website and would like up to date
logs locally like it should be and it appears from Robins reply is being
worked on at some point.  It just seems they want to finish something
else first. 

Dale

:-)  :-) 

Re: [gentoo-dev] Re: ChangeLog

[Patrick Lauer]

On 11/03/2015 02:52 AM, Matt Turner wrote:
>
> So you are telling me that people using github and the switch that took
> place has absolutely nothing to do with the changelogs going dead?
> You keep saying GitHub. Github is not relevant to this discussion.
>
> Yes, the ChangeLogs stopped being updated because of git.
>
> The git transition had been 9 years in the making and has massively
> improved Gentoo development. Look at the graph of contributions per
> month: https://www.openhub.net/p/gentoo
Yes, the rate of git commits has gone up since the git switcheroo.
But that's not what you wanted to say I think ;)

> It was decided that the missing infrastructure for ChangeLogs was not
> a sufficient reason to block a hugely important change.
>
> People care about finishing this, as evidence by this thread. Please
> stop complaining.
>
Apparently my complaining finally re-triggered some action, so sadly
this looks like the currently best strategy.

Sigh,

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Tue, Nov 3, 2015 at 2:22 AM, Patrick Lauer <patrick@gentoo.org> wrote:
> Apparently my complaining finally re-triggered some action, so sadly
> this looks like the currently best strategy.

You could have simply made a simple post pointing out that changelog
generation appears to be broken and likely had the same effect.

The thing with complaining to trigger action is that it can be like
taking one step forward and two steps back.  It might appear to have
the desired effect, and then nobody wants to work in that part of the
community, or in the community at all, so in the long run we're all
worse off.  It doesn't inspire people to contribute.  The areas where
contributions are lost may not even appear to be directly related to
the areas where the complaints are made, so it is hard to demonstrate
a clear relationship, and thus people still feel vindicated in their
complaining or reluctant to do anything about it.

Can I prove that the above is true?  No.  Does it concern me?
Absolutely.  I think the fact that so many are torn between getting
rid of key contributors who tend to create a lot of drama and
continuing to tolerate them just makes people not want to try to fix
it as well.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Chí-Thanh Christopher Nguyễn]

Matt Turner schrieb:
> The git transition had been 9 years in the making and has massively 
> improved Gentoo development. Look at the graph of contributions per 
> month: https://www.openhub.net/p/gentoo

I'd like to point out that some stuff that has previously been done in a 
single commit is now several commits (e.g. bump + removal of old 
version). How much of the rise in commit activity is attributable to 
actual development increase is not clear to me.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Re: ChangeLog

[hasufell]

On 11/03/2015 04:04 PM, Chí-Thanh Christopher Nguyễn wrote:
> Matt Turner schrieb:
>> The git transition had been 9 years in the making and has massively
>> improved Gentoo development. Look at the graph of contributions per
>> month: https://www.openhub.net/p/gentoo
> 
> I'd like to point out that some stuff that has previously been done in a
> single commit is now several commits (e.g. bump + removal of old
> version).

1. not all developers do that and I'd go so far to say that we shouldn't
dictate these detail too much. It has advantages, yes, but it doesn't
really break anything.

2. if you follow the github PRs you will clearly notice that people have
started to actively contribute in a way that wasn't possible on bugzilla
in the same way... there are no hard numbers, but as someone who had
been following the maintainer-wanted/needed aliases, I am pretty sure
the amount of contributions has increased considerably. You can also ask
the proxy-maintainers team, which has all hands full merging PRs.

So yes, git has improved the number of collaborations. It's hard to not
notice, unless you really don't want to notice.

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Tue, Nov 3, 2015 at 10:04 AM, Chí-Thanh Christopher Nguyễn
<chithanh@gentoo.org> wrote:
> Matt Turner schrieb:
>>
>> The git transition had been 9 years in the making and has massively
>> improved Gentoo development. Look at the graph of contributions per month:
>> https://www.openhub.net/p/gentoo
>
> I'd like to point out that some stuff that has previously been done in a
> single commit is now several commits (e.g. bump + removal of old version).
> How much of the rise in commit activity is attributable to actual
> development increase is not clear to me.
>

How were previous cvs commit stats generated?  CVS has not concept of
a commit across multiple files in its data model.  You can of course
look for commits to multiple files that share the same timestamp and
author and infer that these are a single commit, but if you make two
commits at the same time with the same name and description in cvs
there is no way to distinguish that from one commit that hits both
files.  In git these would be captured differently.

For the historical migration to git commits were consolidated using a
window.  Otherwise you'd get a bazillion Manifest commits on top of
everything else, to say nothing of simultaneous commits to
filesdir/etc.

But, yours is a fair point all the same.  In any case, git should be a
lot more useful overall.  Not to mention that while we might be
arguing about which 3rd-party tools are the best for improving our
workflow at least we have a choice now.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Alexis Ballier]

On Tue, 3 Nov 2015 16:04:38 +0100
Chí-Thanh Christopher Nguyễn <chithanh@gentoo.org> wrote:

> Matt Turner schrieb:
> > The git transition had been 9 years in the making and has massively 
> > improved Gentoo development. Look at the graph of contributions per 
> > month: https://www.openhub.net/p/gentoo
> 
> I'd like to point out that some stuff that has previously been done
> in a single commit is now several commits (e.g. bump + removal of old 
> version). How much of the rise in commit activity is attributable to 
> actual development increase is not clear to me.


Also, last I checked, openhub couldn't even process our CVS tree.
Meaning the only stats it had were some random overlays hosted on git.

Re: [gentoo-dev] Re: ChangeLog

[Raymond Jennings]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

Isn't the whole anongit.gentoo.org concept designed to allow anonymous,
read-only git to scale indefinitely in the future?

Are there any plans in the works on how to utilize this domain name?

On Thu, Nov 5, 2015 at 6:33 AM, Alexis Ballier <aballier@gentoo.org> wrote:

> On Tue, 3 Nov 2015 16:04:38 +0100
> Chí-Thanh Christopher Nguyễn <chithanh@gentoo.org> wrote:
>
> > Matt Turner schrieb:
> > > The git transition had been 9 years in the making and has massively
> > > improved Gentoo development. Look at the graph of contributions per
> > > month: https://www.openhub.net/p/gentoo
> >
> > I'd like to point out that some stuff that has previously been done
> > in a single commit is now several commits (e.g. bump + removal of old
> > version). How much of the rise in commit activity is attributable to
> > actual development increase is not clear to me.
>
>
> Also, last I checked, openhub couldn't even process our CVS tree.
> Meaning the only stats it had were some random overlays hosted on git.
>
>

[-- Attachment #2: Type: text/html, Size: 1670 bytes --]

Re: [gentoo-dev] Re: ChangeLog

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Fri, Nov 06, 2015 at 08:25:53PM -0800, Raymond Jennings wrote:
> Isn't the whole anongit.gentoo.org concept designed to allow anonymous,
> read-only git to scale indefinitely in the future?
> 
> Are there any plans in the works on how to utilize this domain name?
What do you mean plans?

It works, and has worked for a long time already (even prior to the main
repo conversion).

$ git clone git://anongit.gentoo.org/repo/gentoo.git
...

Go and look on the gitweb/cgit, and you'll see every public repo has a
anongit URL available. Private repos are not available by anongit, for
self-evident reasons.

Anongit is backed by multiple hosts already, with GeoDNS to hopefully
direct you to the nearest mirror (note that we're down to just one US
host presently, but Europe does still have two).

anongit.gentoo.org.	3600	IN	CNAME	anongit.geodns.gentoo.org.
; GeoDNS targeting...
anongit.geodns.gentoo.org. 3600	IN	CNAME anongit-v4v6.geodns-americas.gentoo.org.
anongit.geodns.gentoo.org. 3600	IN	CNAME anongit-v4v6.geodns-europe.gentoo.org.
...

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Mon, Nov 2, 2015 at 8:20 PM, Dale <rdalek1967@gmail.com> wrote:
> I thought Gentoo was not depending on git/github either.

Take 5min and read the wikipedia articles on both git and github, please.

Gentoo is not going to depend on github, because of the social contract issues.

Gentoo absolutely does depend on git, and it is 100% FOSS.

If these statements seem contradictory, you really need to look up a
video on git 101/etc.  To be fair, I don't think you can truly use git
without first groking it, and you won't accomplish that until you
understand its data model.  Git is a terrific data model wrapped in a
mediocre command line utility.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Rich Freeman wrote:
> On Mon, Nov 2, 2015 at 8:20 PM, Dale <rdalek1967@gmail.com> wrote:
>> I thought Gentoo was not depending on git/github either.
> Take 5min and read the wikipedia articles on both git and github, please.
>
> Gentoo is not going to depend on github, because of the social contract issues.
>
> Gentoo absolutely does depend on git, and it is 100% FOSS.
>
> If these statements seem contradictory, you really need to look up a
> video on git 101/etc.  To be fair, I don't think you can truly use git
> without first groking it, and you won't accomplish that until you
> understand its data model.  Git is a terrific data model wrapped in a
> mediocre command line utility.
>

Based on Robin's earlier post, it seems this is going to be fixed, how
and when may still be open tho.  The reason github entered into this is
not because I am confusing the two but because very shortly after this
change, I had to go to the github website to find out what changed with
some packages because I was having upgrade issues.  That is what made me
mention github.  If I have to go to github to get info about a Gentoo
package and it isn't available on a Gentoo site, that sounds fishy.  I
also recall that at that time, package.gentoo.org was not updating
either.  That is why a link was posted for me to use github instead.  I
do realize and understand that git and github are two different things
but it seems they can work together as well.  It ended up that the info
I needed was on github but not to be found on any Gentoo site at the time. 

My other issue is how these two match up time wise.  I synced the tree
yesterday.  If I go to github or even packages.gentoo.org to see a
changelog, that info may be out of date.  That is why some of us want
them to be done at the same time and stored locally.  Changes that were
made overnight and today don't apply to me because my tree is older than
that.  If I sync then run to the website, it may be more relevant but it
can't be guaranteed that it is. 

Dale

:-) 

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Mon, Nov 2, 2015 at 9:31 PM, Dale <rdalek1967@gmail.com> wrote:
> That is why a link was posted for me to use github instead.  I
> do realize and understand that git and github are two different things
> but it seems they can work together as well.  It ended up that the info
> I needed was on github but not to be found on any Gentoo site at the time.

Anything in /usr/portage that you can find on github is also on
https://gitweb.gentoo.org/repo/gentoo.git/, which is a Gentoo site.

That's kind of the point of git - there are a bazillion tools
available for it and it makes it very easy to clone a full repository
with full history.  It also lowers the bar to contribution.

I get that you're frustrated with the change, and there are a few
others that are as well, but thousands of people use Gentoo, and
generally people only bother to post on lists when they're frustrated.
We can't go into panic mode every time somebody raises a complaint,
and ultimately everybody here is a volunteer.

The complaints don't really bother me much personally, but I do get
concerned that they'll discourage others from contributing.  I can
just ignore threads like these easily enough, but people get
frustrated when they contribute and others just criticize.

-- 
Rich

[gentoo-dev] Re: ChangeLog

[Duncan]

Rich Freeman posted on Mon, 02 Nov 2015 22:17:22 -0500 as excerpted:

> Anything in /usr/portage that you can find on github is also on
> https://gitweb.gentoo.org/repo/gentoo.git/, which is a Gentoo site.

Well... not the metadata in the github repo with pregenerated metadata.  
While it's generated for the rsync tree as well, it's not in the gentoo 
infra git repo.

Similarly, I guess there are various user clones/forks of the gentoo tree 
on github, some of which will certainly have stuff that's not yet on 
gentoo infra, as generically, that's rather the point of having a forked 
git repo on github in the first place, no matter what the origin and 
contents of the repo itself.

But I get your point.  In terms of the git history of the primary gentoo 
repo mirror (as opposed to other forks of it) on github, it's identical 
to that of the gentoo infra master, as that's what it syncs from, with 
the purpose being that it /is/ a mirror with no changes from that of 
gentoo's git master repo.

Tho I get Dale's somewhat confused and now belabored point as well.  For 
people who don't know how to do git on their own, as clearly he doesn't, 
the official signaling of what the stopgap changelog alternatives were 
until the scripts were running to regenerate them from git, simply wasn't 
there.  Without that, confusion reigned, and blaming the user for 
confusion in the absence of official signaling really doesn't help the 
situation, either.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: ChangeLog

[Duncan]

Duncan posted on Tue, 03 Nov 2015 06:43:45 +0000 as excerpted:

> Without that, confusion reigned, and blaming the user for confusion in
> the absence of official signaling really doesn't help the situation,
> either.

I originally added, but unfortunately it got lost in the editing and I 
forgot to re-add...

Happily, things are moving once again, and the original automatically 
generated changelogs should hopefully be appearing relatively soon, 
within a couple weeks hopefully.  The portage patches are now approved on 
the portage-dev list and the code should be in git, tho I'd guess infra 
won't want to run live-git portage, so testing and a release should be 
next, followed by coding and testing the generation scripts, and final 
deployment. =:^)

With a bit of luck and no further unexpected bumps in the road...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Tue, Nov 3, 2015 at 1:43 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>
> Tho I get Dale's somewhat confused and now belabored point as well.  For
> people who don't know how to do git on their own, as clearly he doesn't,
> the official signaling of what the stopgap changelog alternatives were
> until the scripts were running to regenerate them from git, simply wasn't
> there.  Without that, confusion reigned, and blaming the user for
> confusion in the absence of official signaling really doesn't help the
> situation, either.
>

Fair enough.  If we had outright decided to get rid of changelogs we'd
probably have published a news item with clear instructions on how to
just obtain this info from git.  Instead we opted to keep them, but
they broke, so there was no news.

-- 
Rich

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Rich Freeman wrote:
> On Mon, Nov 2, 2015 at 9:31 PM, Dale <rdalek1967@gmail.com> wrote:
>> That is why a link was posted for me to use github instead.  I
>> do realize and understand that git and github are two different things
>> but it seems they can work together as well.  It ended up that the info
>> I needed was on github but not to be found on any Gentoo site at the time.
> Anything in /usr/portage that you can find on github is also on
> https://gitweb.gentoo.org/repo/gentoo.git/, which is a Gentoo site.
>
> That's kind of the point of git - there are a bazillion tools
> available for it and it makes it very easy to clone a full repository
> with full history.  It also lowers the bar to contribution.
>
> I get that you're frustrated with the change, and there are a few
> others that are as well, but thousands of people use Gentoo, and
> generally people only bother to post on lists when they're frustrated.
> We can't go into panic mode every time somebody raises a complaint,
> and ultimately everybody here is a volunteer.
>
> The complaints don't really bother me much personally, but I do get
> concerned that they'll discourage others from contributing.  I can
> just ignore threads like these easily enough, but people get
> frustrated when they contribute and others just criticize.
>


What makes it so bad is the confusion as Duncan pointed out.  I been
following this list for a long time, several years actually.  I don't
read some threads because they are just devs talking about ebuilds and
such and generally that doesn't interest me.  That said, I do watch for
future changes and was even glad that the change was finally happening. 
It had been talked about for ages.  My understanding was this.  If a
person didn't want to use the new tools, nothing would change.  Us
regular users could continue on like we always have.  The change was
mostly for devs and other people who wanted to submit fixes that make it
to the tree and use github etc to do it.  When the change first
happened, there was several issues that popped up and I watched as the
threads explained the situation as best as I could.  Anytime a change
happens, things are going to pop up.  It just seemed to me that after
all the years of talking about this change, it would seem that some of
the basic things should be worked out before the change.  It's not like
this new tool set and method of doing things just popped up one day and
the switch happened a week later.  It had been a work in progress for
something close to forever in Gentoo time. 

Frustrated, yea.  Not at first but this has been going on for a while. 
Right now, I have a package that fails to build and I don't know where
the changelogs are for it and which ones would match what I have
locally.  In the meantime, I'm skipping it.  It's not that I can't fix
the problem, it's that I can't find information to find out what
changed.  After all, it compiled fine several times before.  Something
changed but no clue what, yet. 

I might also add, the only thing getting updated when I sync in
/usr/portage is the ebuilds.  The changelogs haven't updated in months.

Just keep in mind, I'm not a dev.  I'm a user.  I post from a user
perspective.  Something is broke and it affects me and others. 
Hopefully someone will find a fix, soon I hope and I suspect others hope
the same.

Dale

:-)  :-) 

Re: [gentoo-dev] Re: ChangeLog

[Dale]

Rich Freeman wrote:
> On Mon, Nov 2, 2015 at 8:20 PM, Dale <rdalek1967@gmail.com> wrote:
>> I thought Gentoo was not depending on git/github either.
> Take 5min and read the wikipedia articles on both git and github, please.
>
> Gentoo is not going to depend on github, because of the social contract issues.
>
> Gentoo absolutely does depend on git, and it is 100% FOSS.
>
> If these statements seem contradictory, you really need to look up a
> video on git 101/etc.  To be fair, I don't think you can truly use git
> without first groking it, and you won't accomplish that until you
> understand its data model.  Git is a terrific data model wrapped in a
> mediocre command line utility.
>

Based on Robin's earlier post, it seems this is going to be fixed, how
and when may still be open tho.  The reason github entered into this is
not because I am confusing the two but because very shortly after this
change, I had to go to the github website to find out what changed with
some packages because I was having upgrade issues.  That is what made me
mention github.  If I have to go to github to get info about a Gentoo
package and it isn't available on a Gentoo site, that sounds fishy.  I
also recall that at that time, package.gentoo.org was not updating
either.  That is why a link was posted for me to use github instead.  I
do realize and understand that git and github are two different things
but it seems they can work together as well.  It ended up that the info
I needed was on github but not to be found on any Gentoo site at the time. 

My other issue is how these two match up time wise.  I synced the tree
yesterday.  If I go to github or even packages.gentoo.org to see a
changelog, that info may be out of date.  That is why some of us want
them to be done at the same time and stored locally.  Changes that were
made overnight and today don't apply to me because my tree is older than
that.  If I sync then run to the website, it may be more relevant but it
can't be guaranteed that it is. 

Dale

:-) 

Re: [gentoo-dev] Re: ChangeLog

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/02/2015 12:00 PM, Dale wrote:
> Rich Freeman wrote:
>> On Mon, Nov 2, 2015 at 1:08 AM, Dale <rdalek1967@gmail.com>
>> wrote:
>>> Then perhaps all this should have been worked out BEFORE
>>> switching to github?
>> We didn't switch to github.
> 
> Then why are people saying to use git to look at the logs?   I
> don't want to use git.  I liked being able to go to the tree and
> look at the change logs when I needed to which is sometimes often.
> Now, that is gone because it is somewhere else and people are
> saying to use git to get them.  Sounds like a switch to me.
> 
> [snip]
> 
> 

As Ciaran has already said, git and GitHub are not the same thing.
GitHub is a proprietary website for hosting repositories; git is the
software written by Linus Torvalds that we use for revision control.

You can still go to the tree and look at logs. If you're syncing using
hasufell's git repo method (which has been great, btw; thanks
hasufell), then you already have a copy of the git repo on your
system. `cd` to the directory and you can do all sorts of stuff with
`git log` to check out what's happened.

Let's say I want to see what's happened with Firefox lately:

git log -- www-client/firefox/

Bam, you'll get a complete log of the commits that have happened to
*all* files in that directory. You can add `--oneline` to make it
easier to scan through, too. If a specific commit stands out to you,
you can use `git show <commit-id>` to get details on what that commit di
d.

For example, let's see what happened with a somewhat larger commit
that happened about a month ago. Note that you don't need the complete
commit ID; just enough so git finds a unique match.

git show 3a8d9727

That'll show where Manifests were updated, variables changed in
ebuilds, and so on.

- ---

So the workflow is basically `git log -- category/package/`, find a
commit that looks interesting, then `git show commit-id` to look at
what it did.

It's actually much better than changelogs, imo.
- -- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWN9g0AAoJEAEkDpRQOeFw/z8P/2tBVybg+ShZoOJtQkXL0ZZK
eZg1REfydxifZlaaOl3FvfTSe4SkaAfvOoxKN97SivsKPI17uO0qPMfccevotbeX
cVkOptLF6oVcSP9tGK59cV8NvPk6sI6PJkfY3VhAZnM7NAW5VDXEdnQMQDPg3C9C
BNiUKYs+iOAZXOQta5FMNn/V833tJhUEdF5dNyw5JwWNMEAfuXfOfmqJ3Fdty7tz
tIsdGVV22t6XiIeAdxI3QqNEzla+jeUauHGz69LsJMS68FCJfO6jUwioEuwwD+7c
CkVBMB2aWogfHzcRIaJQ0ywMQaYRnhZTzT3BfijaQKi5vGVp69elywaMa9VFCkeo
AKq+OQbzWzmTfUg9O6iKGPNi7dGyRS2U8Xn+8S6HD5hcIrgh052NZeRQ2YBe0DHg
X0+dkjIBzMNiVC2NF7WCvfZ7YxjvrkKkR+IipLU9NxGMRcMHSMkFjvXotsfkRuO0
OPG34U4hEEsUjoWNw4/kfkC6+ISz4+KlITm9h40rJytuLkdcVyZNPZP1yBekWK31
wgMuLsaSg0w2D2zdr4aNN7eFba4tGRYPcRqq9cU6/dAyeKzvrnI4FTkjfGVrf2OX
pSupy0bQY1bYT6MYiKCXG+jaE0IOzNakSlgl9c3JSUQgo3a7D0iwS7oB95X2B0gf
S9or/fPUkcJC8o0dNdSz
=X623
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: ChangeLog

[Patrick Lauer]

On 11/02/2015 02:56 AM, Rich Freeman wrote:
> On Sun, Nov 1, 2015 at 8:22 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>  I know if I were still on rsync (or webrsync), I'd be raising hell about the lack of
>> changelogs well before now
> Perhaps rather than raising hell you'd do better to raise money to
> hire an infra team to fix the bug or something.
Hire?
I'm still willing to fix things, if I were given access. And I would
presume that I'm not the only one.

But since I don't have access, and can only affect things by motivating
or upsetting people, we have a large mail thread that is mostly about
smug people being smug. Which confuses me since it's not Tuesday, but I
digress ...
>
> I get the frustration, but we only have a few people who have the
> necessary access to fix the problem.
So fix that.
>   Infra is also a difficult
> project to deal with in general because it is fairly closed due to the
> implications of having random people messing with it.  I don't really
> see anybody stepping up to try to change anything fundamental about it
> either.  This isn't the sort of thing that will get better if the
> council votes on something.
>
Yes, voting is not going to fix anything directly. So I didn't even
suggest it.

But one of the conditions for tolerating the git migration was that we
have no regressions.
Now it's about 3 months later and *basic* functionality is still
"Oopsiedaisy, I must have missed that"

Which is confusing me because ... uhm... didn't anyone ... test things?
Document? How can something be deployed that is obviously missing
features like this?

(And as a consequence, why doesn't it then get fixed in a reasonable time?)


But at least now we get some good information what is broken how, and
maybe someone can fix it. And then I won't have to be the stone in
people's shoe anymore ;)

Re: [gentoo-dev] Re: ChangeLog

[Rich Freeman]

On Mon, Nov 2, 2015 at 1:24 AM, Patrick Lauer <patrick@gentoo.org> wrote:
>
> On 11/02/2015 02:56 AM, Rich Freeman wrote:
>> On Sun, Nov 1, 2015 at 8:22 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>>  I know if I were still on rsync (or webrsync), I'd be raising hell about the lack of
>>> changelogs well before now
>> Perhaps rather than raising hell you'd do better to raise money to
>> hire an infra team to fix the bug or something.
> Hire?
> I'm still willing to fix things, if I were given access. And I would
> presume that I'm not the only one.

Well, if somebody who has a good history of working with teams and is
generally well-liked wants to step up and volunteer for the job, they
could probably ping the council for endorsement.

>
> But since I don't have access, and can only affect things by motivating
> or upsetting people,

I think you'll find that the former works better than the latter with
volunteers.  Upsetting people generally tends to result in everybody
walking out in a huff and leaving users high and dry, so you're not
really doing anybody any favors with the attitude.

>>
>> I get the frustration, but we only have a few people who have the
>> necessary access to fix the problem.
> So fix that.

Proposals are welcome.

>
> But one of the conditions for tolerating the git migration was that we
> have no regressions.

I certainly never put that condition on it.  I think we were already
trying too hard to make things perfect vs accept the change and work
out some of the details later.  Changelogs being down for a few months
isn't a big deal IMO, especially since the data is all available in
git anyway.

If I didn't think git was ready to go I'd have voted to stop it, and
the Council probably could have done this.  Sometimes you just have to
embrace a change and deal with the fallout around minor issues like
this.  It isn't like Gentoo ground to a halt for a week.

> (And as a consequence, why doesn't it then get fixed in a reasonable time?)

I don't know, why don't you ask the responsible person's manager to
document this on their annual performance review?

> But at least now we get some good information what is broken how, and
> maybe someone can fix it. And then I won't have to be the stone in
> people's shoe anymore ;)

You don't have to be that now.

I get it, you don't like the state of affairs.  Join the club - I
think we're all in it already.  IMO part of the reason we're
struggling is that Gentoo was structured to work the way it does back
when we had at least 3x as much activity in some of the key projects.
The model doesn't scale down well, especially around key roles like
Infra where the design isn't open enough for others to effectively
contribute patches.

-- 
Rich

[gentoo-dev] Re: ChangeLog

[Duncan]

Rich Freeman posted on Sun, 01 Nov 2015 20:56:24 -0500 as excerpted:

> On Sun, Nov 1, 2015 at 8:22 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>  I know if I were still on rsync (or webrsync), I'd be raising hell
>>  about the lack of
>> changelogs well before now
> 
> Perhaps rather than raising hell you'd do better to raise money to hire
> an infra team to fix the bug or something.
> 
> I get the frustration, but we only have a few people who have the
> necessary access to fix the problem.  Infra is also a difficult project
> to deal with in general because it is fairly closed due to the
> implications of having random people messing with it.  I don't really
> see anybody stepping up to try to change anything fundamental about it
> either.  This isn't the sort of thing that will get better if the
> council votes on something.

robbat2's infra response changes some but not all of where this was 
going...

Not wanting to burden limited infra further is why I, and I believe 
Patrick, both offered the alternative of simply officially deleting the 
existing changelogs and officially pointing people at the packages.g.o 
short log and its link to the package-filtered git log for more.

That would at least match official claims with reality once again, plus 
give people some guidance as to actually where/how to look up that info 
when they do need it.

Given robbat2's response, effectively waiting on some egencache code 
changes, if that's going to take more than a week, perhaps at least an 
announcement on www.gentoo.org that we're working on it, and meanwhile, 
packages.gentoo.org and its links to per-package git logs, can be used, 
would be useful.

If it's likely to be fixed within a week or 10 days anyway, probably not 
worth the bother for now, but after the fix, a general news item and 
gentoo.org article pointing out they've been fixed, and pointing out the 
package.g.o alternative for times when a quick web check is easier than a 
local tree check, is probably in order.

Meanwhile, looks like the thread is having the intended effect.  We got a 
status, and people aware of the holdup and actively pursuing it, now. 
=:^)  Without Patrick's post to bump the problem, who knows how long it 
would have stayed in exactly that situation, apparently with only about 
three people knowing what the status and holdup was?

So thanks, Patrick... and the people now actively working on fixing the 
holdup now, as well! =:^)

And not to forget infra, either.  We can all be cranky at times, but I 
think we all appreciate the benefit we get from your hard work.  =:^)

(And despite the thread a couple months or whatever ago, I'm still 
worried about infra's bus factor; that robbat2 has had to reply while in 
theory on vacation recharging certainly supports both the bus factor 
worry and even more the burnout worry.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: ChangeLog

[Michael Orlitzky]

On 11/01/2015 08:22 PM, Duncan wrote:
> 
> Personally, I'd love the primary sync method to be git, and the primary 
> change logs conveyed via native git log methods, but in ordered to do 
> that, all those rsync mirrors need to switch to git mirrors.
> 

I wonder what would happen if we just... dropped a full clone of
gentoo.git onto the rsync mirrors?

Re: [gentoo-dev] Re: ChangeLog

[Patrick Lauer]

On 11/02/2015 03:04 AM, Michael Orlitzky wrote:
> On 11/01/2015 08:22 PM, Duncan wrote:
>> Personally, I'd love the primary sync method to be git, and the primary 
>> change logs conveyed via native git log methods, but in ordered to do 
>> that, all those rsync mirrors need to switch to git mirrors.
>>
> I wonder what would happen if we just... dropped a full clone of
> gentoo.git onto the rsync mirrors?
>
>
You find out how many users have limited bandwidth (again)

Why on earth would you do the most wasteful hybrid strategy when people
are in general trying to optimize for efficiency?

And why am I still sober?

Re: [gentoo-dev] Re: ChangeLog

[Michael Orlitzky]

On 11/02/2015 01:27 AM, Patrick Lauer wrote:
>>>
>> I wonder what would happen if we just... dropped a full clone of
>> gentoo.git onto the rsync mirrors?
>>
>>
> You find out how many users have limited bandwidth (again)
> 
> Why on earth would you do the most wasteful hybrid strategy when people
> are in general trying to optimize for efficiency?
> 

I was only jokingly pointing out that we don't have to switch away from
rsync to update a git repo on the users' machines.

And it's OBVIOUSLY stupid... but is it really?

Limited bandwidth wasn't the problem (stated in this thread) with using
git, an unreliable connection was. Using rsync to transfer the git tree
takes care of that, even if its wholly inefficient.

Once users have the full git repo on their machines, they have two
options. They can update it efficiently with `git pull`, or they can
update it with rsync by using `emerge --sync`. You can even mix the two,
and the repo that we're pointing at could have a copy of the metadata
pregenerated like the github mirror does. The full git log would be
there, and it doesn't require us to overhaul any of the mirror
infrastructure.

I'm not actually suggesting this so nobody has to talk me out of it.

Re: [gentoo-dev] Re: ChangeLog

[Peter Stuge]

Michael Orlitzky wrote:
> Once users have the full git repo on their machines, they have two
> options. They can update it efficiently with `git pull`, or they can
> update it with rsync by using `emerge --sync`. You can even mix the two,

I don't think you can mix the two, because how my local clone looks
depends on when I run git pull. My local Git doesn't repack to match
packs on the remote.

The logical contents will be equivalent but the underlying data
structure will be different, so rsync only works the first time.

FWIW I also liked the look of squashdelta.


//Peter

Re: [gentoo-dev] ChangeLog - Infra Response

[Robin H. Johnson]

I'm replying to the top level of the thread, because I've been on
offline vacation recharging myself for a week, and this thread seems to
have degenerated into ways to avoid the issue, rather than focusing with
what's actually wrong.

rsync-as-a-way-to-get-the-tree is NOT being deprecated, it has very
valid use cases, and places where git is not suited.

I asked zmedico & dol-sen for TWO critical changes to egencache's
--update-changelogs code.

1. Control of the OUTPUT filename for the generated changelog
- the from-git generated changelog will go to 'ChangeLog.git'
2. Control for the order ENTRY for the generated changelog 
- changing to OLDEST-first, with appending the new data at the end
- this massively improves rsync performance.

dol-sen said he was busy with the repoman rewrite, and didn't want to
introduce the change at the time, so this has been deferred for the
moment.

Without #1, we have to rename ALL of the old changelogs, otherwise they
will be overwritten by the new ones from Git history.

I probably should have created a bug for both of these, because I don't
know if they got tracked accurately since I asked for them in August,
and I certainly don't see the code being updated in the repoman or
master branches of the portage repo (it also still generates a $Header$
entry, which does have an open bug as well).

Since dol-sen and zmedico are so busy as well, somebody from this thread
with time to complain, please implement & test these changes!
It's NOT as trivial as dropping a variable into the place where it opens
the file, because there is other code later that also hardcodes the
filename.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] ChangeLog - Infra Response

[Michał Górny]

Dnia 2 listopada 2015 06:50:39 CET, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
>I'm replying to the top level of the thread, because I've been on
>offline vacation recharging myself for a week, and this thread seems to
>have degenerated into ways to avoid the issue, rather than focusing
>with
>what's actually wrong.
>
>rsync-as-a-way-to-get-the-tree is NOT being deprecated, it has very
>valid use cases, and places where git is not suited.
>
>I asked zmedico & dol-sen for TWO critical changes to egencache's
>--update-changelogs code.
>
>1. Control of the OUTPUT filename for the generated changelog
>- the from-git generated changelog will go to 'ChangeLog.git'
>2. Control for the order ENTRY for the generated changelog 
>- changing to OLDEST-first, with appending the new data at the end
>- this massively improves rsync performance.
>
>dol-sen said he was busy with the repoman rewrite, and didn't want to
>introduce the change at the time, so this has been deferred for the
>moment.
>
>Without #1, we have to rename ALL of the old changelogs, otherwise they
>will be overwritten by the new ones from Git history.
>
>I probably should have created a bug for both of these, because I don't
>know if they got tracked accurately since I asked for them in August,
>and I certainly don't see the code being updated in the repoman or
>master branches of the portage repo (it also still generates a $Header$
>entry, which does have an open bug as we.

You should have indeed. Both seem trivial and I'd have done them a long time ago if anybody bothered telling me about it.

>
>Since dol-sen and zmedico are so busy as well, somebody from this
>thread
>with time to complain, please implement & test these changes!
>It's NOT as trivial as dropping a variable into the place where it
>opens
>the file, because there is other code later that also hardcodes the
>filename.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] ChangeLog - Infra Response

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

>>>>> On Mon, 2 Nov 2015, Robin H Johnson wrote:

> 1. Control of the OUTPUT filename for the generated changelog
> - the from-git generated changelog will go to 'ChangeLog.git'

> [...]

> Without #1, we have to rename ALL of the old changelogs, otherwise
> they will be overwritten by the new ones from Git history.

What would be the problem with renaming? IMHO it would be nicer to
keep the ChangeLog name for the autogenerated files and rename the
ones from CVS. We already have files renamed to ChangeLog-<year> when
they became to large, so we could just use ChangeLog-2015 to stay
within that scheme.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] ChangeLog - Infra Response

[Robin H. Johnson]

On Mon, Nov 02, 2015 at 08:05:56AM +0100, Ulrich Mueller wrote:
> >>>>> On Mon, 2 Nov 2015, Robin H Johnson wrote:
> 
> > 1. Control of the OUTPUT filename for the generated changelog
> > - the from-git generated changelog will go to 'ChangeLog.git'
> 
> > [...]
> 
> > Without #1, we have to rename ALL of the old changelogs, otherwise
> > they will be overwritten by the new ones from Git history.
> What would be the problem with renaming? IMHO it would be nicer to
> keep the ChangeLog name for the autogenerated files and rename the
> ones from CVS. We already have files renamed to ChangeLog-<year> when
> they became to large, so we could just use ChangeLog-2015 to stay
> within that scheme.
In the rsync tree, but NOT the Git tree, we have
ChangeLog
ChangeLog-YYYY
Where YYYY so far goes as far as 2014.

And 'ChangeLog' files stopped getting updates on August 8th.

The old ChangeLog files from CVS are explicitly NOT in Git, but manually
injected during the rsync tree creation.

If we rename the old ChangeLog files from CVS to ChangeLog-2015, then
we'll have both 'ChangeLog-2015' and 'ChangeLog' (generated from Git)
containing 2015 entries. Worse, what happens when we hit 2016? Do we
merge the old files?

The main tree Git history starts at August 8th, so it can't be used
to generate older entries either (ignoring using the full history to
generate full ChangeLog.git files since the start of time, because CVS
commit messages used to be a LOT worse than the manual ChangeLog
messages).

Thus, I was proposing to explicitly name the from-git ChangeLog as
ChangeLog.git or ChangeLog-git.



-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] ChangeLog - Infra Response

[Alexis Ballier]

On Mon, 2 Nov 2015 20:18:07 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Mon, Nov 02, 2015 at 08:05:56AM +0100, Ulrich Mueller wrote:
> > >>>>> On Mon, 2 Nov 2015, Robin H Johnson wrote:
> > 
> > > 1. Control of the OUTPUT filename for the generated changelog
> > > - the from-git generated changelog will go to 'ChangeLog.git'
> > 
> > > [...]
> > 
> > > Without #1, we have to rename ALL of the old changelogs, otherwise
> > > they will be overwritten by the new ones from Git history.
> > What would be the problem with renaming? IMHO it would be nicer to
> > keep the ChangeLog name for the autogenerated files and rename the
> > ones from CVS. We already have files renamed to ChangeLog-<year>
> > when they became to large, so we could just use ChangeLog-2015 to
> > stay within that scheme.
> In the rsync tree, but NOT the Git tree, we have
> ChangeLog
> ChangeLog-YYYY
> Where YYYY so far goes as far as 2014.
> 
> And 'ChangeLog' files stopped getting updates on August 8th.
> 
> The old ChangeLog files from CVS are explicitly NOT in Git, but
> manually injected during the rsync tree creation.
> 
> If we rename the old ChangeLog files from CVS to ChangeLog-2015, then
> we'll have both 'ChangeLog-2015' and 'ChangeLog' (generated from Git)
> containing 2015 entries. Worse, what happens when we hit 2016? Do we
> merge the old files?

It's not perfectly clean but I don't see any problem here:
ChangeLog-2015 : all ChangeLog from CVS
ChangeLog: autogenerated from git


if/when there is a need to split git changelogs, autogenerated
changelogs will start from say, Jan. 1st 2016, and previous changes
will now be static. Merging CVS2015 and git2015 changelogs is just a
matter of running a script. Or just skip splitting them for 2016, and
start splitting in 2017, so that ChangeLog-2015 is CVS ones,
ChangeLog-2016 is git logs from Aug. 8. 2015 to Dec. 31 2016.

IMHO this is still better than having ChangeLog stopping in 2015 and
ChangeLog.git starting from this date: Having ChangeLog-2015 from CVS
still carries partial information on the timeline.


Alexis.

Re: [gentoo-dev] ChangeLog - Infra Response

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

>>>>> On Thu, 5 Nov 2015, Alexis Ballier wrote:

> On Mon, 2 Nov 2015 20:18:07 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:

>> On Mon, Nov 02, 2015 at 08:05:56AM +0100, Ulrich Mueller wrote:
>> > What would be the problem with renaming? IMHO it would be nicer to
>> > keep the ChangeLog name for the autogenerated files and rename the
>> > ones from CVS. We already have files renamed to ChangeLog-<year>
>> > when they became to large, so we could just use ChangeLog-2015 to
>> > stay within that scheme.

>> If we rename the old ChangeLog files from CVS to ChangeLog-2015, then
>> we'll have both 'ChangeLog-2015' and 'ChangeLog' (generated from Git)
>> containing 2015 entries. Worse, what happens when we hit 2016? Do we
>> merge the old files?

> It's not perfectly clean but I don't see any problem here:
> ChangeLog-2015 : all ChangeLog from CVS
> ChangeLog: autogenerated from git

> if/when there is a need to split git changelogs, autogenerated
> changelogs will start from say, Jan. 1st 2016, and previous changes
> will now be static. Merging CVS2015 and git2015 changelogs is just a
> matter of running a script. Or just skip splitting them for 2016, and
> start splitting in 2017, so that ChangeLog-2015 is CVS ones,
> ChangeLog-2016 is git logs from Aug. 8. 2015 to Dec. 31 2016.

> IMHO this is still better than having ChangeLog stopping in 2015 and
> ChangeLog.git starting from this date: Having ChangeLog-2015 from CVS
> still carries partial information on the timeline.

+1

You said it better than I could have.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] ChangeLog - Infra Response

[Markos Chandras]

On 11/05/2015 12:39 PM, Ulrich Mueller wrote:
>>>>>> On Thu, 5 Nov 2015, Alexis Ballier wrote:
> 
>> On Mon, 2 Nov 2015 20:18:07 +0000
>> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> 
>>> On Mon, Nov 02, 2015 at 08:05:56AM +0100, Ulrich Mueller wrote:
>>>> What would be the problem with renaming? IMHO it would be nicer to
>>>> keep the ChangeLog name for the autogenerated files and rename the
>>>> ones from CVS. We already have files renamed to ChangeLog-<year>
>>>> when they became to large, so we could just use ChangeLog-2015 to
>>>> stay within that scheme.
> 
>>> If we rename the old ChangeLog files from CVS to ChangeLog-2015, then
>>> we'll have both 'ChangeLog-2015' and 'ChangeLog' (generated from Git)
>>> containing 2015 entries. Worse, what happens when we hit 2016? Do we
>>> merge the old files?
> 
>> It's not perfectly clean but I don't see any problem here:
>> ChangeLog-2015 : all ChangeLog from CVS
>> ChangeLog: autogenerated from git
> 
>> if/when there is a need to split git changelogs, autogenerated
>> changelogs will start from say, Jan. 1st 2016, and previous changes
>> will now be static. Merging CVS2015 and git2015 changelogs is just a
>> matter of running a script. Or just skip splitting them for 2016, and
>> start splitting in 2017, so that ChangeLog-2015 is CVS ones,
>> ChangeLog-2016 is git logs from Aug. 8. 2015 to Dec. 31 2016.
> 
>> IMHO this is still better than having ChangeLog stopping in 2015 and
>> ChangeLog.git starting from this date: Having ChangeLog-2015 from CVS
>> still carries partial information on the timeline.
> 
> +1
> 
> You said it better than I could have.
> 
> Ulrich
> 
yeah, +1 on that too. I am not too bothered with the name to be honest.
However, using 'ChangeLog' for git logs sounds like something most of us
and users are familiar with already so that should work.

-- 
Regards,
Markos Chandras

Re: [gentoo-dev] ChangeLog - Infra Response

[Andreas K. Huettel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Am Donnerstag, 5. November 2015, 12:54:06 schrieb Alexis Ballier:
> 
> if/when there is a need to split git changelogs, autogenerated
> changelogs will start from say, Jan. 1st 2016, and previous changes
> will now be static. Merging CVS2015 and git2015 changelogs is just a
> matter of running a script. Or just skip splitting them for 2016, and
> start splitting in 2017, so that ChangeLog-2015 is CVS ones,
> ChangeLog-2016 is git logs from Aug. 8. 2015 to Dec. 31 2016.
> 
> IMHO this is still better than having ChangeLog stopping in 2015 and
> ChangeLog.git starting from this date: Having ChangeLog-2015 from CVS
> still carries partial information on the timeline.
> 
> 
> Alexis.

+1

- -- 
Andreas K. Huettel
Gentoo Linux developer (council, perl, libreoffice)
dilfridge@gentoo.org
http://www.akhuettel.de/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWPzNBAAoJEB9VdM6hupKVwPYP/joJImpsmJ+jZQ89Mm6HzaOT
FSTmL3zpRxw1XNYQGPO7WUyDwtbVaD+RufiSOINME6+KqhGrII2LxUYeEjd+MvA3
jjA2pNKK/wmYV+inGDvjdXjZ5QpJLVSjpnwf+Igv1NvbA6g87xFkTz55vSZaELIq
ytmeIS8gzSN89QeOzq+4YljvfQcEzWR73URCwegjmD/Bh7i/rmDQ2gGswMuVZg+z
VsX1OMTeiAo7mRFeT5hzfcquRP4UAULSy4jU9L1Aw5tVMhp6A81easV1TH3oL2S4
WgWu9UutdhZpUqF4W1ZNLMbu5ePseP0DcVuLwo8PoWtxteh0I+oMqaNj/M8xKnPT
tolJR0UwHgEQoilJHFH+E6SEICe+hmGgDFlpuRlmdC0UC/YOdQbC/aeaJWZMnNYB
5sMpQ4OgXyXWzIQ4rEBnJeqnh1IThg0CZuubG1X10x9+T0yCUiEDKerlfMgPGz1d
GaFF/q91rjxsIrxe487so9BPjdO7oaZ2L2IYSq8GMyXHtMCftLaYIAPlvZZtdgjh
QYEmk1k1epWg+LeOKoOvMjvuLcsUmDmAdOr2GtQDb0egdcWFKXwWDfgcV71K8tO4
5QmAcJFz8oQx+n1mtjvjoQU9ummPucJGZVag1mr2kwYjiZrt/3UbIHKvyXyii5ui
9O5dgzM462g50kU30UY7
=3hon
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Robin H. Johnson]

On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:
> It's not perfectly clean but I don't see any problem here:
> ChangeLog-2015 : all ChangeLog from CVS
> ChangeLog: autogenerated from git
FYI, this was implemented.

For reference, the old CVS changelogs are now taken from HEAD of this
repo:
https://gitweb.gentoo.org/data/gentoo-changelogs.git/

mgorny and I have been poking at the generation issue, with the features
I requested now implemented, plus one patch I pushed up to portage-dev.

There are still some issues remaining.

I filed bugs for some of them:
565536 - need to exclude some commits/paths
565538 - need to exclude some lines
565540 - need parallel threads

However, the largest sticking point, even with parallel threads, is that
it seems the base ChangeLog generation is incredibly slow. It averages
above 350ms per package right now (at 19k packages in a full cycle, it's
a long time), but some packages can take up to 5 seconds so far.

Incremental processing does help this hugely, but isn't always
available.

Right now, I'm considering promising 30 minute syncs as a best case
interval; if changelog generation causes it to take longer, then a push
window WILL be missed. 

How often might this happen? Since we converted to Git, excluding the
initial large commits, there were three instances where it would have
added more than 10 minutes without the improvements I created bugs for.
Plus, any other changes that cause loss of timestamps/reference for
comparison will trigger a full run, at ~6 hours of delay.

(Yes, that's why there hasn't been an rsync update in the last 3 hours,
and won't be for another ~3 hours: because it's crunching to generate
ChangeLogs).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-dev] Re: ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Duncan]

Robin H. Johnson posted on Wed, 11 Nov 2015 23:11:48 +0000 as excerpted:

> On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:
>> It's not perfectly clean but I don't see any problem here:
>> ChangeLog-2015 : all ChangeLog from CVS ChangeLog: autogenerated from
>> git

> FYI, this was implemented [but generation takes some time...].

As one of those originally posting agreement with the OP... 

Thanks!! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Alexis Ballier]

On Wed, 11 Nov 2015 23:11:48 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:
> > It's not perfectly clean but I don't see any problem here:
> > ChangeLog-2015 : all ChangeLog from CVS
> > ChangeLog: autogenerated from git  
> FYI, this was implemented.


Thanks!

How should one report bugs ? to infra or portage ?
From my just rsynced tree, I see changelogs in reverse order: oldest
come first, latest come last

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Jason Zaman]

On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:
> On Wed, 11 Nov 2015 23:11:48 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> 
> > On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:
> > > It's not perfectly clean but I don't see any problem here:
> > > ChangeLog-2015 : all ChangeLog from CVS
> > > ChangeLog: autogenerated from git  
> > FYI, this was implemented.
> 
> 
> Thanks!
> 
> How should one report bugs ? to infra or portage ?
> From my just rsynced tree, I see changelogs in reverse order: oldest
> come first, latest come last

NOTABUG, it was changed because rsync can deal really well with
appending to the end of files. rsyncing a file where things were things
were added to the beginning of the file means rsync will copy the whole
file from scratch which is pretty sub-optimal.

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Alexis Ballier]

On Thu, 12 Nov 2015 18:49:38 +0800
Jason Zaman <perfinion@gentoo.org> wrote:

> On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:
> > On Wed, 11 Nov 2015 23:11:48 +0000
> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> >   
> > > On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:  
> > > > It's not perfectly clean but I don't see any problem here:
> > > > ChangeLog-2015 : all ChangeLog from CVS
> > > > ChangeLog: autogenerated from git    
> > > FYI, this was implemented.  
> > 
> > 
> > Thanks!
> > 
> > How should one report bugs ? to infra or portage ?
> > From my just rsynced tree, I see changelogs in reverse order: oldest
> > come first, latest come last  
> 
> NOTABUG, it was changed because rsync can deal really well with
> appending to the end of files. rsyncing a file where things were
> things were added to the beginning of the file means rsync will copy
> the whole file from scratch which is pretty sub-optimal.


k, rsync limitation then :)

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Alexander Tsoy]

On Thu, 12 Nov 2015 18:49:38 +0800
Jason Zaman <perfinion@gentoo.org> wrote:

> On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:
> > On Wed, 11 Nov 2015 23:11:48 +0000
> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> >   
> > > On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:  
> > > > It's not perfectly clean but I don't see any problem here:
> > > > ChangeLog-2015 : all ChangeLog from CVS
> > > > ChangeLog: autogenerated from git    
> > > FYI, this was implemented.  
> > 
> > 
> > Thanks!
> > 
> > How should one report bugs ? to infra or portage ?
> > From my just rsynced tree, I see changelogs in reverse order: oldest
> > come first, latest come last  
> 
> NOTABUG, it was changed because rsync can deal really well with
> appending to the end of files. rsyncing a file where things were things
> were added to the beginning of the file means rsync will copy the whole
> file from scratch which is pretty sub-optimal.
> 

PORTAGE_RSYNC_OPTS by default contains "--whole-file". So I guess
appending to the ChangeLog files doesn't really help. :)

-- 
Alexander Tsoy

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Alexander Tsoy]

On Thu, 12 Nov 2015 13:57:58 +0300
Alexander Tsoy <alexander@tsoy.me> wrote:

> On Thu, 12 Nov 2015 18:49:38 +0800
> Jason Zaman <perfinion@gentoo.org> wrote:
> 
> > On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:  
> > > On Wed, 11 Nov 2015 23:11:48 +0000
> > > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> > >     
> > > > On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:    
> > > > > It's not perfectly clean but I don't see any problem here:
> > > > > ChangeLog-2015 : all ChangeLog from CVS
> > > > > ChangeLog: autogenerated from git      
> > > > FYI, this was implemented.    
> > > 
> > > 
> > > Thanks!
> > > 
> > > How should one report bugs ? to infra or portage ?
> > > From my just rsynced tree, I see changelogs in reverse order: oldest
> > > come first, latest come last    
> > 
> > NOTABUG, it was changed because rsync can deal really well with
> > appending to the end of files. rsyncing a file where things were things
> > were added to the beginning of the file means rsync will copy the whole
> > file from scratch which is pretty sub-optimal.
> >   
> 
> PORTAGE_RSYNC_OPTS by default contains "--whole-file". So I guess
> appending to the ChangeLog files doesn't really help. :)

Additionally rsync module appends --whole-file to rsync_opts:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/sync/modules/rsync/rsync.py#n361

-- 
Alexander Tsoy

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

>>>>> On Thu, 12 Nov 2015, Jason Zaman wrote:

> On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:
>> How should one report bugs ? to infra or portage ?
>> From my just rsynced tree, I see changelogs in reverse order: oldest
>> come first, latest come last

> NOTABUG, it was changed because rsync can deal really well with
> appending to the end of files. rsyncing a file where things were
> things were added to the beginning of the file means rsync will copy
> the whole file from scratch which is pretty sub-optimal.

Our ChangeLogs were always in newest-first order, so why would this
suddenly be an issue?

Also readability for users should take priority over technical
matters. Newest first is the usual order (e.g. it agrees with the
default of git log), and ChangeLog having different order from
ChangeLog-20* seems rather confusing to me.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On Thu, 12 Nov 2015 12:12:28 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Thu, 12 Nov 2015, Jason Zaman wrote:  
> 
> > On Thu, Nov 12, 2015 at 11:46:19AM +0100, Alexis Ballier wrote:  
> >> How should one report bugs ? to infra or portage ?
> >> From my just rsynced tree, I see changelogs in reverse order: oldest
> >> come first, latest come last  
> 
> > NOTABUG, it was changed because rsync can deal really well with
> > appending to the end of files. rsyncing a file where things were
> > things were added to the beginning of the file means rsync will copy
> > the whole file from scratch which is pretty sub-optimal.  
> 
> Our ChangeLogs were always in newest-first order, so why would this
> suddenly be an issue?
> 
> Also readability for users should take priority over technical
> matters. Newest first is the usual order (e.g. it agrees with the
> default of git log), and ChangeLog having different order from
> ChangeLog-20* seems rather confusing to me.

I imagine it breaks emerge --changelog output as well?


-- 
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 475 bytes --]

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Peter Stuge]

Robin H. Johnson wrote:
> However, the largest sticking point, even with parallel threads, is that
> it seems the base ChangeLog generation is incredibly slow. It averages
> above 350ms per package right now (at 19k packages in a full cycle, it's
> a long time), but some packages can take up to 5 seconds so far.

Which code is doing this generation? Sorry - ENOOVERVIEW. :\


Thanks

//Peter

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Peter Stuge]

Peter Stuge wrote:
> Robin H. Johnson wrote:
> > However, the largest sticking point, even with parallel threads, is that
> > it seems the base ChangeLog generation is incredibly slow. It averages
> > above 350ms per package right now (at 19k packages in a full cycle, it's
> > a long time), but some packages can take up to 5 seconds so far.
> 
> Which code is doing this generation? Sorry - ENOOVERVIEW. :\

Bump. Does anyone know where I can take a look at this code?


//Peter

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Michael Orlitzky]

On 11/18/2015 09:48 AM, Peter Stuge wrote:
> Peter Stuge wrote:
>> Robin H. Johnson wrote:
>>> However, the largest sticking point, even with parallel threads, is that
>>> it seems the base ChangeLog generation is incredibly slow. It averages
>>> above 350ms per package right now (at 19k packages in a full cycle, it's
>>> a long time), but some packages can take up to 5 seconds so far.
>>
>> Which code is doing this generation? Sorry - ENOOVERVIEW. :\
> 
> Bump. Does anyone know where I can take a look at this code?
> 

I don't know, but since no one else is answering, I'll try to find out.
There are a few bugs on b.g.o. (search "changelog") that suggest
`egencache --update-changelog` is being used. The egencache command is
part of portage, so....

  $ git clone http://anongit.gentoo.org/git/proj/portage.git

Looking at bin/egencache, you'll find a bunch of indirection, but
ultimately, the generate_changelog() method of the GenChangeLogs class
is doing the work. The implementation is straightforward. I suspect the
slow part is,

  # now grab all the commits
  revlist_cmd = ['git', self._work_tree, 'rev-list']
  if self._changelog_reversed:
      revlist_cmd.append('--reverse')
      revlist_cmd.extend(['HEAD', '--', '.'])
      commits = self.grab(revlist_cmd).split()

where

  @staticmethod
  def grab(cmd):
      p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
      return _unicode_decode(p.communicate()[0],
                             encoding=_encodings['stdio'],
                             errors='strict')

That's taking about half a second if I run it from the command-line.

Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Michael Orlitzky]

On 11/18/2015 12:55 PM, Michael Orlitzky wrote:
> 
> That's taking about half a second if I run it from the command-line.
> 

...and this takes even longer:

  cinfo = self.grab(['git', self._work_tree, 'diff-tree',
                     '--name-status',
                     '--no-renames',
                     '--format=%ct %cN <%cE>%n%B',
                     '--root',
                     '--relative=%s' % (cp, ),
                     '-r',
                     c,
                     '--', '.']).rstrip('\n').split('\n')

That happens for every commit.

Re: [gentoo-dev] ChangeLog - Infra Response

[Brian Dolbec]

On Mon, 2 Nov 2015 05:50:39 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> I'm replying to the top level of the thread, because I've been on
> offline vacation recharging myself for a week, and this thread seems
> to have degenerated into ways to avoid the issue, rather than
> focusing with what's actually wrong.
> 
> rsync-as-a-way-to-get-the-tree is NOT being deprecated, it has very
> valid use cases, and places where git is not suited.
> 
> I asked zmedico & dol-sen for TWO critical changes to egencache's
> --update-changelogs code.
> 
> 1. Control of the OUTPUT filename for the generated changelog
> - the from-git generated changelog will go to 'ChangeLog.git'
> 2. Control for the order ENTRY for the generated changelog 
> - changing to OLDEST-first, with appending the new data at the end
> - this massively improves rsync performance.
> 
> dol-sen said he was busy with the repoman rewrite, and didn't want to
> introduce the change at the time, so this has been deferred for the
> moment.
> 
> Without #1, we have to rename ALL of the old changelogs, otherwise
> they will be overwritten by the new ones from Git history.
> 
> I probably should have created a bug for both of these, because I
> don't know if they got tracked accurately since I asked for them in
> August, and I certainly don't see the code being updated in the
> repoman or master branches of the portage repo (it also still
> generates a $Header$ entry, which does have an open bug as well).
> 
> Since dol-sen and zmedico are so busy as well, somebody from this
> thread with time to complain, please implement & test these changes!
> It's NOT as trivial as dropping a variable into the place where it
> opens the file, because there is other code later that also hardcodes
> the filename.
> 

Well, to be honest, I kinda forgot about this. (too many things on my
plate)  The repoman code is released and mostly stable, and
starting work on stage2 of that process.  So That is one thing less
to worry about for the moment. Any other required repoman changes are
certainly possible now. 

The $HEADER$ did get an initial change, but as I recall there has been
no official/final decision on whether it is to be dropped completely.
Plus I'm not finding that bug this morning.  If it is to be dropped
completely, that can be done.  As I recall it was you that wanted to
keep it, at least for an interim period for some other infra use in the
rsync tree generation.  I don't recall the exact reason.

Most certainly patches are welcome.

-- 
Brian Dolbec <dolsen>