[gentoo-dev] CIA replacement

[gentoo-dev] CIA replacement

[Ben de Groot]

Since CIA.vc is dead [1], I think we should be looking into a
replacement service, or host our own [2].
Is infra already looking into this?

1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
2: http://www.donarmstrong.com/posts/switching_to_kgb/
-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

[gentoo-dev] Re: CIA replacement

[Duncan]

Ben de Groot posted on Mon, 01 Oct 2012 16:14:25 +0800 as excerpted:

> Since CIA.vc is dead [1], I think we should be looking into a
> replacement service, or host our own [2].
> Is infra already looking into this?
> 
> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2:
> http://www.donarmstrong.com/posts/switching_to_kgb/

This has been discussed previously.  

Thread: CIA.VC down for the count?
Original post by blueness on August 23, last post on the 26th.


Here's a link to the gmane thread:

http://comments.gmane.org/gmane.linux.gentoo.devel/79499

Old interface:

http://thread.gmane.org/gmane.linux.gentoo.devel/79499


It may be that there are further developments, but a read of the old 
thread (which you don't appear to have participated in and didn't mention 
so I'm assuming you missed) should help in any case.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: CIA replacement

[Ben de Groot]

On 1 October 2012 17:48, Duncan <1i5t5.duncan@cox.net> wrote:
> Ben de Groot posted on Mon, 01 Oct 2012 16:14:25 +0800 as excerpted:
>
>> Since CIA.vc is dead [1], I think we should be looking into a
>> replacement service, or host our own [2].
>> Is infra already looking into this?
>>
>> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2:
>> http://www.donarmstrong.com/posts/switching_to_kgb/
>
> This has been discussed previously.
>
> Thread: CIA.VC down for the count?
> Original post by blueness on August 23, last post on the 26th.
>
>
> Here's a link to the gmane thread:
>
> http://comments.gmane.org/gmane.linux.gentoo.devel/79499
>
> Old interface:
>
> http://thread.gmane.org/gmane.linux.gentoo.devel/79499
>
>
> It may be that there are further developments, but a read of the old
> thread (which you don't appear to have participated in and didn't mention
> so I'm assuming you missed) should help in any case.

I didn't actually miss that thread, but in that case it was a short
outage, and cia.vc came back the next day. People were pointed at the
-commits ML, and that was it.

Now that it's down for good (unless someone recreates it, but no news
of that so far), we need a more permanent solution. The -commits ML is
okay (tho I don't want to subscribe to such a high-volume ML), but we
miss an IRC interface. The website and statistics of cia.vc were nice
too.

The irker proxy was mentioned in this thread. I think we should look
into this. Unless someone has a better idea.

(which is why I brought this up in the first place)
-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] Re: CIA replacement

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 305 bytes --]

On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
> The irker proxy was mentioned in this thread. I think we should look
> into this. Unless someone has a better idea.

I think it might be more beneficial to try and relay -commits email to
irc.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Re: CIA replacement

[Jeroen Roovers]

Responding to no one in particular, but to the sub-thread about IRC
bots:

On Tue, 2 Oct 2012 08:32:26 +0200
Fabian Groffen <grobian@gentoo.org> wrote:

> On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
> > The irker proxy was mentioned in this thread. I think we should look
> > into this. Unless someone has a better idea.
> 
> I think it might be more beneficial to try and relay -commits email to
> irc.

An IRC bot is nice, but CIA also provided current (!) statistics, which
isn't merely useful in order to boost your own morale/ego/self-esteem,
but also provided up to date information about other developers'
activity, which can help other developers decide if and when to step in
and start (temporarily) maintaining packages that need immediate
attention. ohloh cannot do that for us, presently, and a simple IRC
bot cannot either.


     jer

Re: [gentoo-dev] Re: CIA replacement

[Anthony G. Basile]

On 10/02/2012 08:21 PM, Jeroen Roovers wrote:
> Responding to no one in particular, but to the sub-thread about IRC
> bots:
>
> On Tue, 2 Oct 2012 08:32:26 +0200
> Fabian Groffen<grobian@gentoo.org>  wrote:
>
>> On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
>>> The irker proxy was mentioned in this thread. I think we should look
>>> into this. Unless someone has a better idea.
>> I think it might be more beneficial to try and relay -commits email to
>> irc.
> An IRC bot is nice, but CIA also provided current (!) statistics, which
> isn't merely useful in order to boost your own morale/ego/self-esteem,
> but also provided up to date information about other developers'
> activity, which can help other developers decide if and when to step in
> and start (temporarily) maintaining packages that need immediate
> attention. ohloh cannot do that for us, presently, and a simple IRC
> bot cannot either.
>
>
>       jer
>

jer thanks for saying that.  That's exactly how I was using CIA.  Now 
I'm just using the gentoo-commits@ list which gives the same info but 
requires more sorting effort on my brain.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 8040 5A4D 8709 21B1 1A88  33CE 979C AF40 D045 5535
GnuPG ID  : D0455535

Re: [gentoo-dev] Re: CIA replacement

[Michael Mol]

On Tue, Oct 2, 2012 at 8:21 PM, Jeroen Roovers <jer@gentoo.org> wrote:
> Responding to no one in particular, but to the sub-thread about IRC
> bots:
>
> On Tue, 2 Oct 2012 08:32:26 +0200
> Fabian Groffen <grobian@gentoo.org> wrote:
>
>> On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
>> > The irker proxy was mentioned in this thread. I think we should look
>> > into this. Unless someone has a better idea.
>>
>> I think it might be more beneficial to try and relay -commits email to
>> irc.
>
> An IRC bot is nice, but CIA also provided current (!) statistics, which
> isn't merely useful in order to boost your own morale/ego/self-esteem,
> but also provided up to date information about other developers'
> activity, which can help other developers decide if and when to step in
> and start (temporarily) maintaining packages that need immediate
> attention. ohloh cannot do that for us, presently, and a simple IRC
> bot cannot either.

Bouncing over to -scm list. Maybe this can be implemented in a post-commit hook.


-- 
:wq

Re: [gentoo-dev] Re: CIA replacement

[Ben de Groot]

On 3 October 2012 08:21, Jeroen Roovers <jer@gentoo.org> wrote:
> Responding to no one in particular, but to the sub-thread about IRC
> bots:
>
> On Tue, 2 Oct 2012 08:32:26 +0200
> Fabian Groffen <grobian@gentoo.org> wrote:
>
>> On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
>> > The irker proxy was mentioned in this thread. I think we should look
>> > into this. Unless someone has a better idea.
>>
>> I think it might be more beneficial to try and relay -commits email to
>> irc.
>
> An IRC bot is nice, but CIA also provided current (!) statistics, which
> isn't merely useful in order to boost your own morale/ego/self-esteem,
> but also provided up to date information about other developers'
> activity, which can help other developers decide if and when to step in
> and start (temporarily) maintaining packages that need immediate
> attention. ohloh cannot do that for us, presently, and a simple IRC
> bot cannot either.

Indeed. I mentioned that in passing as well (that their website was
useful), and you explain why.

I don't think any of the replacement options I've seen have this
feature. Does anyone know of something that might offer us this? Or
should we develop something in-house?

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] Re: CIA replacement

[Jeroen Roovers]

On Wed, 3 Oct 2012 11:43:09 +0800
Ben de Groot <yngwin@gentoo.org> wrote:

> I don't think any of the replacement options I've seen have this
> feature. Does anyone know of something that might offer us this? Or
> should we develop something in-house?

AFAIK some devrel related group already has some tool running that
checks how long developers have been inactive (on our source code
repositories). I don't think that information is disclosed publicly at
this point, whereas CIA clearly did that publicly.


     jer

[gentoo-dev] Re: CIA replacement

[Duncan]

Jeroen Roovers posted on Wed, 03 Oct 2012 06:45:48 +0200 as excerpted:

> On Wed, 3 Oct 2012 11:43:09 +0800 Ben de Groot <yngwin@gentoo.org>
> wrote:
> 
>> I don't think any of the replacement options I've seen have this
>> feature. Does anyone know of something that might offer us this? Or
>> should we develop something in-house?
> 
> AFAIK some devrel related group already has some tool running that
> checks how long developers have been inactive (on our source code
> repositories). I don't think that information is disclosed publicly at
> this point, whereas CIA clearly did that publicly.

Undertakers.

(Watching that and closing unused accounts does keep them from hanging 
around to be discovered by the bad guys.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: CIA replacement

[Rafael Goncalves Martins]

On Tue, Oct 2, 2012 at 9:21 PM, Jeroen Roovers <jer@gentoo.org> wrote:
> Responding to no one in particular, but to the sub-thread about IRC
> bots:
>
> On Tue, 2 Oct 2012 08:32:26 +0200
> Fabian Groffen <grobian@gentoo.org> wrote:
>
>> On 02-10-2012 12:40:20 +0800, Ben de Groot wrote:
>> > The irker proxy was mentioned in this thread. I think we should look
>> > into this. Unless someone has a better idea.
>>
>> I think it might be more beneficial to try and relay -commits email to
>> irc.
>
> An IRC bot is nice, but CIA also provided current (!) statistics, which
> isn't merely useful in order to boost your own morale/ego/self-esteem,
> but also provided up to date information about other developers'
> activity, which can help other developers decide if and when to step in
> and start (temporarily) maintaining packages that need immediate
> attention. ohloh cannot do that for us, presently, and a simple IRC
> bot cannot either.
>

Yeah, I miss the stats page too, but unfortunately I`m not aware of
anybody working on this feature. some people may list ohloh as a
replacement for CIA stats, but it is way behind the simplicity of the
CIA stats, and currently uses an almost always outdated git clone of
our cvs tree to gather the stats.

-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

Re: [gentoo-dev] Re: CIA replacement

[Peter Stuge]

Ben de Groot wrote:
> The -commits ML is okay (tho I don't want to subscribe to such a
> high-volume ML), but we miss an IRC interface. The website and
> statistics of cia.vc were nice too.

What is the source data, and what does the desired output look like?

(I mean what should the messages in channel look like.)


//Peter

Re: [gentoo-dev] Re: CIA replacement

[Ben de Groot]

On 2 October 2012 23:56, Peter Stuge <peter@stuge.se> wrote:
> Ben de Groot wrote:
>> The -commits ML is okay (tho I don't want to subscribe to such a
>> high-volume ML), but we miss an IRC interface. The website and
>> statistics of cia.vc were nice too.
>
> What is the source data, and what does the desired output look like?
>
> (I mean what should the messages in channel look like.)

We used to have it like this:

<CIA-5> tetromino * gentoo-x86/x11-themes/gnome-themes-standard/ (7 files):
<CIA-5> Version bump for gtk+-3.6; high contrast and low contrast
themes are no longer provided. Make license more precise..
<CIA-5> (Portage version: 2.2.0_alpha132/cvs/Linux x86_64)
<CIA-5> ssuominen * gentoo-x86/xfce-extra/xfce4-power-manager/ (3
files in 2 dirs):
<CIA-5> Fix crash with en_GB locale wrt #419973 by Ciprian Ciubotariu
<CIA-5> (Portage version: 2.2.0_alpha128/cvs/Linux x86_64)
<CIA-5> mr_bones_ * gentoo-x86/games-roguelike/falconseye/ (5 files in
2 dirs): games-roguelike/falconseye is gone
<CIA-5> mr_bones_ * gentoo-x86/profiles/package.mask:
games-roguelike/falconseye is gone
<CIA-5> tetromino proj/gnome:master * r78959387
/dev-util/gdbus-codegen/ (gdbus-codegen-9999.ebuild
gdbus-codegen-2.33.14.ebuild): dev-util/gdbus-codegen: 2.34.0 now in
gx86

So basically:

$committer $repo $path:
$changelog

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] Re: CIA replacement

[Peter Stuge]

Ben de Groot wrote:
> > What is the source data,

Still unanswered. I'll ask something which would be equally helpful:

Where is the software that currently sends out emails to the -commits list?


> > and what does the desired output look like?
> 
> <CIA-5> tetromino * gentoo-x86/x11-themes/gnome-themes-standard/ (7 files):
> <CIA-5> Version bump for gtk+-3.6; high contrast and low contrast
> themes are no longer provided. Make license more precise..
> <CIA-5> (Portage version: 2.2.0_alpha132/cvs/Linux x86_64)
> <CIA-5> ssuominen * gentoo-x86/xfce-extra/xfce4-power-manager/ (3
> files in 2 dirs):
> <CIA-5> Fix crash with en_GB locale wrt #419973 by Ciprian Ciubotariu
> <CIA-5> (Portage version: 2.2.0_alpha128/cvs/Linux x86_64)
..
> <CIA-5> tetromino proj/gnome:master * r78959387
> /dev-util/gdbus-codegen/ (gdbus-codegen-9999.ebuild
> gdbus-codegen-2.33.14.ebuild): dev-util/gdbus-codegen: 2.34.0 now in
> gx86
> 
> So basically:
> 
> $committer $repo $path:
> $changelog

Well it's more complicated, but thanks for the sample output!


//Peter

Re: [gentoo-dev] Re: CIA replacement

[Ben de Groot]

On 3 October 2012 00:51, Peter Stuge <peter@stuge.se> wrote:
> Ben de Groot wrote:
>> > What is the source data,
>
> Still unanswered. I'll ask something which would be equally helpful:
>
> Where is the software that currently sends out emails to the -commits list?

I don't know, but I assume it's a commit hook in our CVS repository.
Someone from infra should be able to shed more light on that.

>> > and what does the desired output look like?
>>
>> <CIA-5> tetromino * gentoo-x86/x11-themes/gnome-themes-standard/ (7 files):
>> <CIA-5> Version bump for gtk+-3.6; high contrast and low contrast
>> themes are no longer provided. Make license more precise..
>> <CIA-5> (Portage version: 2.2.0_alpha132/cvs/Linux x86_64)
>> <CIA-5> ssuominen * gentoo-x86/xfce-extra/xfce4-power-manager/ (3
>> files in 2 dirs):
>> <CIA-5> Fix crash with en_GB locale wrt #419973 by Ciprian Ciubotariu
>> <CIA-5> (Portage version: 2.2.0_alpha128/cvs/Linux x86_64)
> ..
>> <CIA-5> tetromino proj/gnome:master * r78959387
>> /dev-util/gdbus-codegen/ (gdbus-codegen-9999.ebuild
>> gdbus-codegen-2.33.14.ebuild): dev-util/gdbus-codegen: 2.34.0 now in
>> gx86
>>
>> So basically:
>>
>> $committer $repo $path:
>> $changelog
>
> Well it's more complicated, but thanks for the sample output!

Yeah, but we don't need an exact replica. Just a quick way to see who
committed what where, and the changelog message.

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] CIA replacement

[Chí-Thanh Christopher Nguyễn]

Ben de Groot schrieb:
> Since CIA.vc is dead [1], I think we should be looking into a
> replacement service, or host our own [2].

I understand that ohloh is already tracking us (sometimes at least).
http://www.ohloh.net/p/gentoo/


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] CIA replacement

[Rafael Goncalves Martins]

Hi Ben,

On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot <yngwin@gentoo.org> wrote:
> Since CIA.vc is dead [1], I think we should be looking into a
> replacement service, or host our own [2].
> Is infra already looking into this?
>
> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
> 2: http://www.donarmstrong.com/posts/switching_to_kgb/

Maybe someone with good cvs knowledge can contribute a hook for irker
[1], so we can have #gentoo-commits flooding our irc clients again! :)

[1] http://www.catb.org/esr/irker/

Best regards.

-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

Re: [gentoo-dev] CIA replacement

[Rich Freeman]

On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Maybe someone with good cvs knowledge can contribute a hook for irker
> [1], so we can have #gentoo-commits flooding our irc clients again! :)

Why exactly are we still using cvs?  Rather than building enhancements
for cvs, why not just migrate everything to git, and spend our time
building the git hooks/etc necessary to make this work?

Looking at the tracker [1], we need a pre-upload hook (I'm not quite
sure why), an rsync conversion script, the ability to validate the
converted tree, and documentation.  There is still an open bug for
commit signing, and I'm not quite sure why as this was implemented.

It seems like a lot has already been done with validation.  Checking
the active tree is pretty trivial - just compare the trees and they
should be the same.  I guess we need to check history, but it seems to
me like the risk of problems is low, and if we just keep a backup of
the cvs repository if there is ever a concern about who made some
commit 5 years ago we can always dig it up.

It really seems to me like little remains to be done here.  Mostly we
just need somebody to push a decision on things like workflow.  A few
of the bugs have comments like "no sense working on this with other
stuff still needed" - which seems to be outdated thinking with so
little left to do.

Am I missing some big concern that just isn't obvious in these bugs?

I also fear that we're refusing to take action on a great solution
because it isn't a perfect solution.  Nobody in the world is using
tree-signing with git, and we aren't really using it in cvs either.
We now have the ability to do it with git, but depending on workflow
3rd-party signatures might not end up in the history of head, or we
might not be able to verify them in an automated fashion.  Honestly, I
think the appropriate response here is whoop-de-doo.  We can't do any
of that stuff with cvs, but moving to git would have a lot of other
benefits.  We can always change our processes later once somebody has
a solution for the signing problem.  Right now we're making do without
it on cvs, and so is every other project using git.  We can also
continue to sign manifests as a workaround, which is what we'll be
doing anyway if we never migrate to git.

The git migration just strikes me as one of those cases where anybody
is free to come up with a reason not to use something, but nobody has
to defend keeping the status quo.  I think the question isn't whether
there is anything wrong with using git, but whether the problems with
git are worse than the problems we already have.

But, hey, if somebody wants to write an irc bot that posts cvs
commits, knock yourself out.

Rich

[1] - https://bugs.gentoo.org/show_bug.cgi?id=333531

Re: [gentoo-dev] CIA replacement

[Michael Mol]

On Mon, Oct 1, 2012 at 1:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
> <rafaelmartins@gentoo.org> wrote:
>> Maybe someone with good cvs knowledge can contribute a hook for irker
>> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> Why exactly are we still using cvs?  Rather than building enhancements
> for cvs, why not just migrate everything to git, and spend our time
> building the git hooks/etc necessary to make this work?
>
> Looking at the tracker [1], we need a pre-upload hook (I'm not quite
> sure why), an rsync conversion script, the ability to validate the
> converted tree, and documentation.  There is still an open bug for
> commit signing, and I'm not quite sure why as this was implemented.
>
> It seems like a lot has already been done with validation.  Checking
> the active tree is pretty trivial - just compare the trees and they
> should be the same.  I guess we need to check history, but it seems to
> me like the risk of problems is low, and if we just keep a backup of
> the cvs repository if there is ever a concern about who made some
> commit 5 years ago we can always dig it up.
>
> It really seems to me like little remains to be done here.  Mostly we
> just need somebody to push a decision on things like workflow.  A few
> of the bugs have comments like "no sense working on this with other
> stuff still needed" - which seems to be outdated thinking with so
> little left to do.
>
> Am I missing some big concern that just isn't obvious in these bugs?
>
> I also fear that we're refusing to take action on a great solution
> because it isn't a perfect solution.  Nobody in the world is using
> tree-signing with git, and we aren't really using it in cvs either.
> We now have the ability to do it with git, but depending on workflow
> 3rd-party signatures might not end up in the history of head, or we
> might not be able to verify them in an automated fashion.  Honestly, I
> think the appropriate response here is whoop-de-doo.  We can't do any
> of that stuff with cvs, but moving to git would have a lot of other
> benefits.  We can always change our processes later once somebody has
> a solution for the signing problem.  Right now we're making do without
> it on cvs, and so is every other project using git.  We can also
> continue to sign manifests as a workaround, which is what we'll be
> doing anyway if we never migrate to git.
>
> The git migration just strikes me as one of those cases where anybody
> is free to come up with a reason not to use something, but nobody has
> to defend keeping the status quo.  I think the question isn't whether
> there is anything wrong with using git, but whether the problems with
> git are worse than the problems we already have.
>
> But, hey, if somebody wants to write an irc bot that posts cvs
> commits, knock yourself out.
>
> Rich
>
> [1] - https://bugs.gentoo.org/show_bug.cgi?id=333531
>

I don't know to what depth this has been discussed in the past, but if
you use git, you also get an HTTP transport, which has a useful
feature: You could simplify updating the tree on end-users's machines
by using caching proxy servers (operating in accelerator mode) on the
various mirrors.

Those of us who have our own, local caching proxy servers (I have
squid running on my network gateway) can reduce the network load even
further by getting cache hits on our local network before even making
queries outside our network. (Personally, I find this a far easier
thing to maintain and do debugging reasoning on than, e.g. sharing a
network mount or running a local rsync server managed by a cron job.)

-- 
:wq

Re: [gentoo-dev] CIA replacement

[Rich Freeman]

On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> I don't know to what depth this has been discussed in the past, but if
> you use git, you also get an HTTP transport, which has a useful
> feature: You could simplify updating the tree on end-users's machines
> by using caching proxy servers (operating in accelerator mode) on the
> various mirrors.

The issue I see here is a tradeoff of bandwidth vs CPU.  I just ran an
emerge --sync and the total amount of transmitted data was 5M.  The
whole tree is 250M, though no doubt with compression that could be
reduced.

Now, one advantage of HTTP is that caching http servers are likely
more ubiquitous in general than rsync servers.  But, we have a whole
bunch of rsync servers already, and we don't have a bunch of caching
http servers.

I suspect bandwidth is going to cost more than CPU here.

In any case, not a reason to hold up git, just one more possibility if
we ever move.

Rich

Re: [gentoo-dev] CIA replacement

[Michael Mol]

On Mon, Oct 1, 2012 at 1:54 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>> I don't know to what depth this has been discussed in the past, but if
>> you use git, you also get an HTTP transport, which has a useful
>> feature: You could simplify updating the tree on end-users's machines
>> by using caching proxy servers (operating in accelerator mode) on the
>> various mirrors.
>
> The issue I see here is a tradeoff of bandwidth vs CPU.  I just ran an
> emerge --sync and the total amount of transmitted data was 5M.  The
> whole tree is 250M, though no doubt with compression that could be
> reduced.
>
> Now, one advantage of HTTP is that caching http servers are likely
> more ubiquitous in general than rsync servers.  But, we have a whole
> bunch of rsync servers already, and we don't have a bunch of caching
> http servers.
>
> I suspect bandwidth is going to cost more than CPU here.
>
> In any case, not a reason to hold up git, just one more possibility if
> we ever move.

It really depends on how efficient 'git pull' is over HTTP, IMO. I
mean, when I do pulls and pushes in my workflows, it doesn't do a full
data send or a full data pull; that's reserved for 'git clone'.

It may also depend on how often the pull is done; rsync doesn't pull a
tree history, just a copy at the time of sync. git pulls new objects,
which may include intermediate versions which are no longer necessary.
(git may be capable of only syncing to 'HEAD' without worrying about
intermediate states, but I don't know. I'm not advanced enough to
definitively say one way or another.)

As for setting up caching proxies at existing mirror sites...It should
be a case of publishing, a single squid (or pick whatever proxy you
prefer to standardize on) configuration file with appropriate ACLs and
copy it all over the place. I'm not on the infra team, so I don't know
what things look like under the hood, but I do know setting up squid
in forward and reverse roles isn't very painful.

-- 
:wq

Re: [gentoo-dev] CIA replacement

[Rafael Goncalves Martins]

On Mon, Oct 1, 2012 at 2:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
> <rafaelmartins@gentoo.org> wrote:
>> Maybe someone with good cvs knowledge can contribute a hook for irker
>> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> Why exactly are we still using cvs?  Rather than building enhancements
> for cvs, why not just migrate everything to git, and spend our time
> building the git hooks/etc necessary to make this work?

It is amazing how a simple thread about a quite simple topic, with an
easy solution, like this, turns to useless discussions about endless
topics, like this git conversion. This mailing-list is getting really
boring. :(

I'm not asking nobody to write any "enhancement for cvs". irker is
already "done", and working fine, but the authors just implemented
support for git and svn, because this is what they use. We *use* CVS
for now, then we would need to fix it to work with CVS, no matter if
we will have git repositories at some time. I stopped believing in
Santa Claus when I was 5 years old.

This should be doable with 10 lines of python, or using the cia->irker
proxy that jdhore mentioned in another email.

> Looking at the tracker [1], we need a pre-upload hook (I'm not quite
> sure why), an rsync conversion script, the ability to validate the
> converted tree, and documentation.  There is still an open bug for
> commit signing, and I'm not quite sure why as this was implemented.

Have you ever thought that people may be not really interested on this
move? or don't have the time to work on it? or don't care enough to
spend time on it? or just wants someone else to do the work?

If you want it, go ahead and push it, but in the right places, please.
I'm tired of bikeshedding here in this list. :(

[snip]

Best regards,

-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

Re: [gentoo-dev] CIA replacement

[Rich Freeman]

On Mon, Oct 1, 2012 at 2:08 PM, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Have you ever thought that people may be not really interested on this
> move? or don't have the time to work on it? or don't care enough to
> spend time on it? or just wants someone else to do the work?
>

I'd thought of every single one of those.  Hence my post on the list
to drum up interest.

> If you want it, go ahead and push it, but in the right places, please.
> I'm tired of bikeshedding here in this list. :(

Where else would one discuss it?  Certainly happy to have a separate
thread - it just seemed like an obvious question when somebody brought
up making an enhancement to our cvs infrastructure.  I have no
interest in posting this on a list that nobody actually reads, but if
there are a bunch of devs following some other list that is more
appropriate by all means let me know.

This just strikes me as something that is about at the point where we
could "just do it."  Obviously the discussion needs to involve at
least some core of developers, since the goal is to get everybody to
stop doing their commits in cvs, and start doing them in git, and I
doubt everybody is going to be happy if somebody convinces infra to
shut down cvs without any discussion first.

Rich

Re: [gentoo-dev] CIA replacement

[Diego Elio Pettenò]

On 01/10/2012 11:29, Rich Freeman wrote:
> Where else would one discuss it?

gentoo-scm

Yes, there is a mailing list for that.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] CIA replacement

[Peter Stuge]

Rich Freeman wrote:
> I doubt everybody is going to be happy if somebody convinces infra
> to shut down cvs without any discussion first.

I would do exactly that, actually.

There's been years of discussion. There's even a mailing list for
discussion.


//Peter

Re: [gentoo-dev] CIA replacement

[Gregory M. Turner]

On 10/1/2012 10:29 AM, Rich Freeman wrote:

>
> Looking at the tracker [1], we need a pre-upload hook (I'm not quite
> sure why), an rsync conversion script, the ability to validate the
> converted tree, and documentation.  There is still an open bug for
> commit signing, and I'm not quite sure why as this was implemented.

a job for gitolite? what does ogo use?

> Am I missing some big concern that just isn't obvious in these bugs?

Things will break.  There will be much wringing of virtual hands. But in 
1 weeks' time, everyone will get over it and find ways to fix whatever 
broke.  There will be one less thing on the big to-do list, and most 
people will be happier overall.

It's one of those "Social Security" type of things it's hard to build a 
clean consensus around /when/ to make a break like this.  I think we 
basically need a headstrong asshole in a position of sufficient 
authority to say, "fuck everyone, we are doing this, here is the 
changeover date."

Too much democracy, tactical planning and consensus-building will almost 
certainly perpetuate the status quo.

Look, Gentoo will probably fall to pieces for a few days.  The 
alternative, IMO, is a huge, elaborate multi-phase plan of some kind, 
like how Microsoft or NASA would do it: armies of 40-somethings toil for 
months or years in cubicles...  Elaborate Gantt charts and process 
diagrams are drawn up and approved by layer after layer of middle 
management.

Gentoo can tolerate a hiccup; it's survived way scarier challenges.  We 
have a huge community of very talented people highly motivated to keep 
this machinery working... there's literally zero chance that Gentoo will 
somehow have it's spirit broken by cvs->git.

In other words, devs -- imo -- just go for it.  Throw caution to the 
wind.  Break some bylaws if need be.  Ignore anyone saying "we're not 
ready."  It'll be easier to get forgiveness than permission, and exactly 
what process issues needed to be smoothed over will frankly only truly 
be clear in the aftermath.

Of course, that's easy for me to say since I'm not the one whose door 
everyone will be beating down when TSHTF.  There are worse things than 
CVS (Visual SourceSafe comes to mind).  But if we don't bite the bullet 
now, we all know we'll be having this discussion again later.

-gmt

Re: [gentoo-dev] CIA replacement

[Peter Stuge]

Gregory M. Turner wrote:
> "fuck everyone, we are doing this, here is the changeover date."

Well put. When is the date? I suggest October 5th, 18:00 UTC.


//Peter

Re: [gentoo-dev] CIA replacement

[Diego Elio Pettenò]

On 01/10/2012 13:17, Peter Stuge wrote:
> Gregory M. Turner wrote:
>> > "fuck everyone, we are doing this, here is the changeover date."
> Well put. When is the date? I suggest October 5th, 18:00 UTC.

And I suggest we stop here. We have a different mailing list for this
and it's getting tiring.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] CIA replacement

[Rich Freeman]

On Mon, Oct 1, 2012 at 4:35 PM, Diego Elio Pettenò
<flameeyes@flameeyes.eu> wrote:
>
> And I suggest we stop here. We have a different mailing list for this
> and it's getting tiring.

Ok, looking at the archives as far as I can tell nobody is really
monitoring that list (a post requesting a status update went
unanswered sep 9th).

So, rather than continuing to post here which seems to be annoying
people, I'll just issue a call to anybody who cares to move discussion
over to gentoo-scm.

That said, I plan to mark any blockers to the migration as resolved
where I don't see any remaining concerns unless somebody comments on
-scm or on the relevant bugs that this isn't a good idea.

I don't think we can keep the discussion off -dev forever though.  It
seems like we're close to being able to implement, which means lots of
changes that impact all devs.  I can't imagine that we'd want to
implement that without some kind of council vote.  Perhaps the
appropriate approach is to propose a GLEP?

Rich

[gentoo-dev] Discussing stuff that is not appropriate to discuss

[Diego Elio Pettenò]

On 01/10/2012 13:54, Rich Freeman wrote:
> I don't think we can keep the discussion off -dev forever though.  It
> seems like we're close to being able to implement, which means lots of
> changes that impact all devs.  I can't imagine that we'd want to
> implement that without some kind of council vote.  Perhaps the
> appropriate approach is to propose a GLEP?

No, the appropriate approach is first to _talk with Infra_. Guys I said
that before, but unless you actually factor in Infra when you want
infrastructure stuff done, you have to do the work yourself.
And it might not be nice and fun.

Honestly, this whole thread, with the exception of Rafael, makes me
facepalm incredibly, because everybody is saying "it's easy!" without
asking the people who have done the work up to now and will have to
manage it. And it pisses me off.

(Among other things, because it feels like most of the complains about
the way tinderbox's logs are handled, "it's easy!" but nobody but me is
ever going to pick up the task, ...)

So to close this in a few words: You walk the walk, you talk the talk.
And I have no intention to read another mail with "what other awesome
thing we can do if we migrate to git and we don't even have to worry
about what it might happen on the serverside because git is just magical
and will sort itself out", okay?

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Peter Stuge]

Diego Elio Pettenò wrote:
> Honestly, this whole thread, with the exception of Rafael, makes me
> facepalm incredibly, because everybody is saying "it's easy!"
> without asking the people who have done the work up to now and will
> have to manage it.

Noone said it's easy.

Please don't put words in my mouth.

Several said it needs to just-be-done, without further consensus.
I support that. Everyone also agrees that there will be issues, but
I think the idea is that switching sooner rather than later and
fixing up the bits that break is fine. Even if it takes a while.


> And it pisses me off.

It shouldn't. There may be others on the list besides Gentoo infra
who know a thing or two about infrastructure, operations, CVS, Git,
and so on.


> most of the complains about the way tinderbox's logs are handled

I have no idea about that issue, but it seems quite distinct.


> So to close this in a few words: You walk the walk, you talk the talk.

That's bullshit (and the saying is backwards). I could help out, I
know a couple of infra guys, but I can't even get recruited because
quizzes need too much contiguous time out of my schedule..


//Peter

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Diego Elio Pettenò]

On 01/10/2012 14:41, Peter Stuge wrote:
> Noone said it's easy.

«This just strikes me as something that is about at the point where we
could "just do it."»

This was Rich at 11.29 Pacific Time.

> Several said it needs to just-be-done, without further consensus.
> I support that. Everyone also agrees that there will be issues, but
> I think the idea is that switching sooner rather than later and
> fixing up the bits that break is fine. Even if it takes a while.

Okay so you have your idea. Keep it. But don't try to force it on people
who are actual developers.

> It shouldn't. There may be others on the list besides Gentoo infra
> who know a thing or two about infrastructure, operations, CVS, Git,
> and so on.

Sure, but they are not Gentoo Infra. And who has to maintain this?
Gentoo Infra. You want to do the conversion and maintain it yourself?
Feel free, nobody's stopping you. You're free to fork.

> I have no idea about that issue, but it seems quite distinct.

It's the same bullshit of insisting that someone else should do
something just because you think it's better, without actually asking
for feasibility.

> I could help out, I
> know a couple of infra guys, but I can't even get recruited because
> quizzes need too much contiguous time out of my schedule..

That's true for everybody. And knowing the infra guys doesn't mean that
you're infra still. I'm not infra either and I don't speak _for_ them,
but for this kind of stuff, instead of starting a SEVENTEEN POSTS thread
on a mailing list that should be dedicated to other stuff, the solution
is first _ask what the status is_.

With all due respect, but having Michael going off a tangent on caching
proxy servers, Rich starting to ponder between bandwidth and CPU, and
you calling for shutdown dates, all without changing a stupid subject
line to at least show you're no longer speaking about the original topic
(it's not like people can be psychic that you're talking about GIT
migrations when the topic says "CIA replacement"), is obnoxious.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Peter Stuge]

Diego Elio Pettenò wrote:
> With all due respect,
..
> you calling for shutdown dates
..
> is obnoxious.

I don't know about respectful, but oh well..


Another idea I have, besides the go-ahead+fix what breaks, is that
after everything has broken, Gentoo developers will not be spamming
this mailing list like three-year-olds screaming rude complaints
about how things do not work and calling infra bad names, but that
they will actually *help out* with whatever needs fixing.

Gentoo has a whole bunch of very competent developers in many
different areas, including yourself of course!, and I'm pretty sure
that there is no better way to get everything fixed *fast* than to
simply go-ahead. Some (of course not all, and that's fine too) devs
would surely get involved to help out with whatever issues need to
be solved.


//Peter

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Diego Elio Pettenò]

On 01/10/2012 15:21, Peter Stuge wrote:
> Diego Elio Pettenò wrote:
>> With all due respect,
> ..
>> you calling for shutdown dates
> ..

> all without changing a stupid subject
> line to at least show you're no longer speaking about the original topic
> (it's not like people can be psychic that you're talking about GIT
> migrations when the topic says "CIA replacement"),

Yes just cut out the part that makes it obnoxious, will ya?

>> is obnoxious.

> Another idea I have, besides the go-ahead+fix what breaks, is that
> after everything has broken, Gentoo developers will not be spamming
> this mailing list like three-year-olds screaming rude complaints
> about how things do not work and calling infra bad names, but that
> they will actually *help out* with whatever needs fixing.

Then you probably don't know half the Gentoo developers....

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Peter Stuge]

Diego Elio Pettenò wrote:
> > Another idea I have, besides the go-ahead+fix what breaks, is that
> > after everything has broken, Gentoo developers will not be spamming
> > this mailing list like three-year-olds screaming rude complaints
> > about how things do not work and calling infra bad names, but that
> > they will actually *help out* with whatever needs fixing.
> 
> Then you probably don't know half the Gentoo developers....

I think they are the ones who should fork. :)


//Peter

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Diego Elio Pettenò]

On 01/10/2012 15:53, Peter Stuge wrote:
>> > Then you probably don't know half the Gentoo developers....
> I think they are the ones who should fork. :)

Unfortunately the problem is that they tend to linger around even after
forking...

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Gregory M. Turner]

If you're going to paint me and the other folks expressing opinions as 
entitled mouth-breathers, certainly you can't expect not to hear any 
reply because it's "off-topic"!

On 10/1/2012 2:00 PM, Diego Elio Pettenò wrote:
> On 01/10/2012 13:54, Rich Freeman wrote:
>> I don't think we can keep the discussion off -dev forever though.  It
>> seems like we're close to being able to implement, which means lots of
>> changes that impact all devs.  I can't imagine that we'd want to
>> implement that without some kind of council vote.  Perhaps the
>> appropriate approach is to propose a GLEP?
>
> No, the appropriate approach is first to _talk with Infra_. Guys I said
> that before, but unless you actually factor in Infra when you want
> infrastructure stuff done, you have to do the work yourself.
> And it might not be nice and fun.

Well I definitely can't argue with the above, and I didn't know about 
-scm but, hey, you learn something new every day; anyhow I'm not sure 
it's makes it inappropriate to discuss.

> So to close this in a few words: You walk the walk, you talk the talk.
> And I have no intention to read another mail with "what other awesome
> thing we can do if we migrate to git and we don't even have to worry
> about what it might happen on the serverside because git is just magical
> and will sort itself out", okay?

OK, also agreed that git lacks entirely in magical properties -- I 
wasn't aware of having said anything suggesting otherwise.  I have some 
limited experience as a git admin and am aware of git's considerable 
limitations with respect to provisioning, security configuration, 
scalability and other non-optional features for a public-facing 
deployment.  There is more to be said on the matter but I'll take that 
to -scm.

Anyhow, I get it: administering the vcs for a huge project such as 
Gentoo is very hard work.  If I somehow gave some other impression, I'm 
sorry.   Perhaps Rich and I insensitively voiced our shared assumption 
that Gentoo's continued reliance on cvs stems from a lack of motivation 
and consensus, rather than a shortage of labor and resources.  Then 
again, if the folks in the trenches doing the work think I've slighted 
them, surely they are perfectly capable of chewing me out on their own 
behalfs and don't need you to do it for them?

I'm not looking for a fight, but after reading the above and some other 
remarks in this thread, by you and others, I did want to at least 
clarify my position: No, of course I couldn't possibly know all the 
repercussions of a change like this, but I also find it difficult to 
believe that whatever hurdles exist are so intractable that we'd might 
as well just throw in the towel.  Although I regret any bad feelings I 
may have caused, I stand by my statements.  To be clear, the "magic" of 
the Gentoo community -- not git -- is what I believe will make this 
doable, and yes, I appreciate that that "magic" is actually just a lot 
of people doing a lot of nasty, thankless chores.

As for the whole put-up/shut-up business, I'd be happy to help out any 
way I can (although tbh I'd still say whatever was on my mind even if I 
was too busy).  I'll contact Infrastructure to make sure they're aware 
of me.

-gmt

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Diego Elio Pettenò]

On 01/10/2012 17:51, Gregory M. Turner wrote:
> 
> Anyhow, I get it: administering the vcs for a huge project such as
> Gentoo is very hard work.  If I somehow gave some other impression, I'm
> sorry.   Perhaps Rich and I insensitively voiced our shared assumption
> that Gentoo's continued reliance on cvs stems from a lack of motivation
> and consensus, rather than a shortage of labor and resources. 

That's definitely not the case. While we do have had some complains
(mostly from Prefix last I knew) about git's working, the consensus for
going to git is there. The problems are vastly technical.

Problems such as "how many developers would be fine with having to
checkout 2GB of history to be able to commit"? git support shallow
clones but not if you want to commit to them.

> Then
> again, if the folks in the trenches doing the work think I've slighted
> them, surely they are perfectly capable of chewing me out on their own
> behalfs and don't need you to do it for them?

It is at the very least disturbing that you think that people whose
work, and commitment, has been overlooked for a whole discussion can't
be bothered by it unless they are actually reactive aggressively.

I know the kind of thankless tasks Infra has to deal with on a daily
basis, and I think they deserve more respect that most of the time they
are given here, especially when technical challenges are billed under
the "we just need to push harder for it to move" banner like this time.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Michael Mol]

On Mon, Oct 1, 2012 at 8:58 PM, Diego Elio Pettenò
<flameeyes@flameeyes.eu> wrote:
> On 01/10/2012 17:51, Gregory M. Turner wrote:
>>
>> Anyhow, I get it: administering the vcs for a huge project such as
>> Gentoo is very hard work.  If I somehow gave some other impression, I'm
>> sorry.   Perhaps Rich and I insensitively voiced our shared assumption
>> that Gentoo's continued reliance on cvs stems from a lack of motivation
>> and consensus, rather than a shortage of labor and resources.
>
> That's definitely not the case. While we do have had some complains
> (mostly from Prefix last I knew) about git's working, the consensus for
> going to git is there. The problems are vastly technical.
>
> Problems such as "how many developers would be fine with having to
> checkout 2GB of history to be able to commit"? git support shallow
> clones but not if you want to commit to them.
>
>> Then
>> again, if the folks in the trenches doing the work think I've slighted
>> them, surely they are perfectly capable of chewing me out on their own
>> behalfs and don't need you to do it for them?
>
> It is at the very least disturbing that you think that people whose
> work, and commitment, has been overlooked for a whole discussion can't
> be bothered by it unless they are actually reactive aggressively.
>
> I know the kind of thankless tasks Infra has to deal with on a daily
> basis, and I think they deserve more respect that most of the time they
> are given here, especially when technical challenges are billed under
> the "we just need to push harder for it to move" banner like this time.

Some fascinating problems (social and technical) to address. I joined
the -scm list because it was implied that these things would be
discussed over there.

Could we "take this outside"? I'd be interested in looking at
solutions...but I don't want to drop them in here, since it's been
pounded on a few times by a few people that this isn't the place.

-- 
:wq

[gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Brian Harring]

Cross-posting to scm; responses should go to scm please (and the 
people who whinge about cross posting should go promptly to hell if 
I have any say in the matter).

On Mon, Oct 01, 2012 at 05:58:43PM -0700, Diego Elio Petten?? wrote:
> On 01/10/2012 17:51, Gregory M. Turner wrote:
> > 
> > Anyhow, I get it: administering the vcs for a huge project such as
> > Gentoo is very hard work.  If I somehow gave some other impression, I'm
> > sorry.   Perhaps Rich and I insensitively voiced our shared assumption
> > that Gentoo's continued reliance on cvs stems from a lack of motivation
> > and consensus, rather than a shortage of labor and resources. 
> 
> That's definitely not the case. While we do have had some complains
> (mostly from Prefix last I knew) about git's working, the consensus for
> going to git is there. The problems are vastly technical.
> 
> Problems such as "how many developers would be fine with having to
> checkout 2GB of history to be able to commit"? git support shallow
> clones but not if you want to commit to them.

Few corrections;
1) You can commit to shallow clones.  You can actually push from them 
too- you just have to know what you're doing (your parent *has* to be 
known to the other side, else you're trying to push a disconnected 
history/graph to the other side, which doesn't know how to connect the 
two).  We won't be doing that fortunately, just noting that it is 
possible if you're careful (and I know what the man page says; what 
I'm saying is the full version, rather than the short version they 
list there).

2) graft's are what we'll be doing there; kind of shallow, but now.  
Basically the same thing the kernel folk did.


As for the "quit your bitching and contribute already" rant angle; 
Diego's accurate; minimally, it's more productive to contribute and 
you're less likely to crap on folks motivation, let alone risk the 
wraith of a pissy person like me yelling at you.

Here in is the kicker; certain chunks of this can't be handled by 
random joe blow off the street- they require core infra access.  

Bluntly (no disrespect to people, just being brutally direct) I don't 
care if you have infra friends, I don't care if you maintain a couple 
of boxes; if you're doing heavy OPs in a production environment, 
you'll understand the issue of trust/access- thus you'll understand 
that some of this work, cannot be done by anyone but infra.

Like it or not, very few people have access to the core cvs -> rsync 
hosts/machinery- since each/every/one/of/us means it's a security 
angle that has to be tracked.  That's not arguable, so don't even try 
please.

That said, there are non-infra contributions people can make.

I suggest people do that; here's the list off the top of my head 
(these are things worst case, I'll sort- which means it'll be months 
out till I finish them considering my own time constraints and focus 
on getting eapi5 support into pkgcore first).

0) First the rules of the road for this discussion; assume that I'll 
be bitchy if you violate this.

0.a) We're not dropping the existing history.  Suggesting this is 
asking for a killfile entry, it's viable for small or throw-away 
projects; gentoo-x86 cvs repository is not a throw-away project.

0.b) Lesser offence since it's not obvious; the various suggestions 
that we just snapshot this, then try to fix history after the fact 
won't work- look into git's transitive trust via sha1's of the 
parent's sha1.  To do that sort of proposal means forcing a full 
history rewrite down the line; this doesn't fly.

0.c) For whatever I've missed, assume that if it craps on developers 
workflow... it's a no go, and needs to be addressed.  Does CVS suck?  
Yes, I hate having to use it.  But it *works*; switching to git has to 
be, minimally, a lateral move for developers in terms of their 
workflow- we cannot make it worse else what's the point of this whole 
exercise?  There may be an exception or two here- things that aren't 
sorted immediately upon conversion, but those exceptions will only fly 
if they're minor, don't require history rewrites, and someone is 
locked in/guranteed to be working on it now (else we have no gurantee 
it'll actually be sorted).


1) We need a thin manifest -> thick manifest converter.  Thin 
manifests are used for git- they store just DIST entries.  Thick (also 
known as 'full'), are what cvs/rsync users are familiar with- it holds 
checksums for all content.

1.a) This converter must use portage api's; ultimately, this 
thin->thick conversion will be signed by an infra key (rather than the 
current hodgepodge of devs).  I suggest nesting it under the emaint 
command.

1.b) This converter needs to be fast.  $VCS -> rsync updates occur 
every 30 minutes.  thin/thick sorting should be sub minute, frankly; 
go parallel (multiprocessing) being my suggestion, threadpool worst 
case (since most of the work won't be gil bound).

1.c) This absolutely has to be fucking stable.  This will be a core 
part of our infrastructure after all.

1.d) I will kneecap the first person who whines about portage on this, 
or suggests NIH "lets just hack it"- they won't have to support it, 
this goes into portage so it's proper, and so infra isn't stuck w/ 
more custom code.

1.e) This actually isn't that hard.  Ask in #gentoo-portage for 
details, look at portage source, look at repoman's existing manifest 
command- that manifest command already is the basics of it.

1.5) Incremental signing of a tree is basically required; meaning 
whatever scanner there is, shouldn't require resigning every single 
package, only those that have changed thick manifest wise.

1.6) Anyone looking to do this should pop into #gentoo-portage, talk 
w/ a user named 'carebear', zmedico, etc; zmedico is portage's 
maintainer, carebear is the current person volunteering to sort this 
(help may be appreciated, talk to him/her/it).


2) Building off of #1, although *NOT REQUIRED FOR CVS->GIT MIGRATION*, 
just very strongly desired, is sorting tree signing gleps while we're 
at it.  Start from http://www.gentoo.org/proj/en/glep/glep-0057.html ; 
whatever solution #1 takes (likely an emaint command), tree signing 
will be built right smack dab into it.


3) Robin afaik is putting together an email with the details; roughly, 
the conversion process is conversion of cvs to svn, then svn2git 
conversion; this is done since frankly it's the best/sanest conversion 
pathway, and the fastest.  The validation of that conversion, and 
getting it down to basically a set of known invocations is required.

3.a) Roughly, the plan will be snag the tree, start conversion.  
Validate the results, repeat as necessary till we're happy with it.  
This is the initial git core history,  This step should be <8h; mostly 
cpu time, frankly, although re-validation of that pathway is required 
(I did a fair amount of optimization to this, but I've not rechecked 
the runtime in a while- nor if there is a better option in existence).  
Basically, it's strongly preferable we're not sorting this at the time 
we're trying to do the live conversion- the core issues need to be 
sorted before.

3.b) Take all cvs activity that has occurred since the tree was 
snapshotted and conversion started, and replay it into git via tailor; 
this is minor- and avoidable if we just shut the tree down for however 
long 3.a takes; that said, the tailor route is the intention, and 
shouldn't be a problem.


4) People who strongly know git hooks would be useful; server side, 
all incoming pushes from devs will have their commits validated before 
touching the tree- bad validation, commit gets kicked back to them.  
The hooks for this need doing (development of this can be done locally 
w/out having to access infra either).  Hell, someone may already have 
done something similar- I've not seen it, but we need something akin 
to this; whoever does this, needs to write it such that the auth 
backend is configurable (upon deployment, this will be bound into 
ldap, or an ldap scraped set of data that it'll consult); assume that 
the auth backend will be user->gpg key level of validation (meaning I 
cannot take a random commit antarus had against current ToT, and push 
that on his behalf- robin may disagree on this point however).



Were that to be done, that would leave for infra basically the 
following- which is most definitely not a complete list-

1) gitolite configuration/setup, which afaik is basically sorted.
2) cvs -> rsync pathways being rebuilt to be git -> rsync (reliant on 
#1 from above, but there is more that occurs there).
3) Thanking people for stepping up and helping to take care of the 
stuff we're seriously low on time to sort.

People don't step up, I'll be working my way through that list; that 
said, my timetable were I to do this isn't "next week or the week 
after"- it's "over the next few months as time allows".

Also, it's entirely possible I missed something for the non-infra 
tasks people can contribute to; that's just a quick brain dump, pardon 
any incorrect statements.  If one has questions and answers aren't 
coming through via the scm ml, then worst case track me down on 
freenode via the ferringb nick; just assume I'll be wickedly laggy 
in responding.

Finally, pardon the strong tones; the tone in use isn't meant to 
dissuade people from contributing, it's meant to ensure people stay 
focused on what's required here to get the job done- discussions about 
building a git mirroring tier (for example) are for *after* the 
initial work is done (understand that 99% of users will be using rsync 
even when we switch dev's underlying vcs got git; longer term that may 
change, but it's a v2 type thing, not a v1 type thing).

Cheers-
~harring

Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Ben de Groot]

Thank you so much for taking the time to give us this clear list of
things that need to be done to take this forward!

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Theo Chatzimichos]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Tuesday 02 of October 2012 12:58:04 Ben de Groot wrote:
> Thank you so much for taking the time to give us this clear list of
> things that need to be done to take this forward!

Disclaimer: I haven't read Brian's long mail (and most of the mails in this 
mailing list for the past month)

One of the things that would be nice to have before the Git migration is 
Documentation. Feel free to submit docs in the wiki, and I'll help a lot after 
the conference as well.

Theo

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Ben de Groot]

On 3 October 2012 04:51, Theo Chatzimichos <tampakrap@gentoo.org> wrote:
> One of the things that would be nice to have before the Git migration is
> Documentation. Feel free to submit docs in the wiki, and I'll help a lot after
> the conference as well.

Can you be more specific as to what kind of docs are needed?

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Jeroen Roovers]

On Wed, 3 Oct 2012 11:46:15 +0800
Ben de Groot <yngwin@gentoo.org> wrote:

> On 3 October 2012 04:51, Theo Chatzimichos <tampakrap@gentoo.org>
> wrote:
> > One of the things that would be nice to have before the Git
> > migration is Documentation. Feel free to submit docs in the wiki,
> > and I'll help a lot after the conference as well.
> 
> Can you be more specific as to what kind of docs are needed?
> 

Just a quick browse through our docs, leaving out examples where there
is merely mention of "CVS repositories" (where CVS is equated with any
version control system) instead of CVS specific material:

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=1&chap=4

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=3#doc_chap1

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=4

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=5

http://devmanual.gentoo.org/general-concepts/cvs-to-rsync/index.html

http://www.gentoo.org/proj/en/infrastructure/cvs-sshkeys.xml

http://www.gentoo.org/doc/en/cvs-tutorial.xml

http://www.gentoo.org/proj/en/devrel/quiz/ebuild-quiz.txt


     jer

Re: [gentoo-dev] CVS -> git, list of where non-infra folk can contribute

[Gregory M. Turner]

Brian Harring wrote:
> 1) We need a thin manifest -> thick manifest converter.  Thin
> manifests are used for git- they store just DIST entries.  Thick (also
> known as 'full'), are what cvs/rsync users are familiar with- it holds
> checksums for all content.
>
> carebear is the current person volunteering to sort this
> (help may be appreciated, talk to him/her/it).

heh :)

I'll read up, spend some time on IRC, and see what I can do to help here.

>
> replay it into git via tailor;
>

Never knew about that tool... not sure about the wisdom of adding an 
extra moving part just to keep the lights on for those few hours... 
Given the "2G of history" issue Diego mentioned, which if I understand 
correctly, effectively means that the future gentoo git can never rebase 
its commit history, why chance it?

In my last experience with cvs->git (at the time I was building a rsync 
(binutils cvs)->git mirror for a client), the most difficult thing about 
cvs->git was trying to scrub the identity data.

I don't remember the exact issue, but somehow, git had identity 
uniqueness constraints that cvs happily ignored, or something like that. 
  I never thought to try using svn as an intermediate -- but I like that 
idea a lot and wish I had thought of it when I needed to.

Anyhow, wrong ml for this, I'll subscribe to -scm.

-gmt

[gentoo-dev] Re: Discussing stuff that is not appropriate to discuss

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On Mon, 01 Oct 2012 14:00:58 -0700
Diego Elio Pettenò <flameeyes@flameeyes.eu> wrote:

> (Among other things, because it feels like most of the complains about
> the way tinderbox's logs are handled, "it's easy!" but nobody but me is
> ever going to pick up the task, ...)

Well, duh.  You designed, developed, and are the sole architect of the
system.  You made an error in the design.  You might have had good reasons at
the time, and you can argue them til you turn blue to anyone who will listen,
but if your end consumers see it as a flaw it isn't going to change
a thing.  You're a service provider now.  You need to provide
everything your customers ask for, before they ask for it, or you'll
get nailed to the nearest tree.  Welcome to the wonderful world of
customer service. :)

Sorry to start yet another tangent.

-- 
gcc-porting
toolchain, wxwidgets          we were never more here, expanse getting broader
@ gentoo.org                          but bigger boats been done by less water

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Regarding the tinderbox logs

[Diego Elio Pettenò]

On 02/10/2012 21:14, Ryan Hill wrote:
> Well, duh.  You designed, developed, and are the sole architect of the
> system.

Not by choice...

> You made an error in the design.  You might have had good reasons at
> the time, and you can argue them til you turn blue to anyone who will listen,
> but if your end consumers see it as a flaw it isn't going to change
> a thing. 

Here's the problem, it's the 80-20 rule. Just in this case reversed, in
the sense that 80% of the noise comes from 20% of the people (and I'd
argue even less than 20%).

We have a system that works nicely, and most people don't even complain
about it. A few asked why, and when they were told they said "ok". A
couple complained that it's not as easy but kept going and _one_ is
destructively removing references to the logs and ignoring valid bugs on
his own packages (caused by his own packages most of the time), because
they smell.

It does seem logical that I'm not going to rollback months of work just
because one guy can't be bothered to play well with others, doesn't it?

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Regarding the tinderbox logs

[Markos Chandras]

On Wed, Oct 3, 2012 at 3:35 PM, Diego Elio Pettenò
<flameeyes@flameeyes.eu> wrote:
> On 02/10/2012 21:14, Ryan Hill wrote:
>> Well, duh.  You designed, developed, and are the sole architect of the
>> system.
>
> Not by choice...
>
>> You made an error in the design.  You might have had good reasons at
>> the time, and you can argue them til you turn blue to anyone who will listen,
>> but if your end consumers see it as a flaw it isn't going to change
>> a thing.
>
> Here's the problem, it's the 80-20 rule. Just in this case reversed, in
> the sense that 80% of the noise comes from 20% of the people (and I'd
> argue even less than 20%).
>
> We have a system that works nicely, and most people don't even complain
> about it. A few asked why, and when they were told they said "ok". A
> couple complained that it's not as easy but kept going and _one_ is
> destructively removing references to the logs and ignoring valid bugs on
> his own packages (caused by his own packages most of the time), because
> they smell.
>
> It does seem logical that I'm not going to rollback months of work just
> because one guy can't be bothered to play well with others, doesn't it?
>
> --
> Diego Elio Pettenò — Flameeyes
> flameeyes@flameeyes.eu — http://blog.flameeyes.eu/
>

Nobody asked you to rollback months of work just because a few people
can't deal with the way you submit your logs. But you should also
respect their preferences. Each one of us has its personal way of
dealing with his/her bugs (based on some loose standards and
policies). If someone doesn't like your way of submitting logs then
just accept it. It is not like they go and remove your tinderbox links
from bugs/packages they don't maintain.

-- 
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

Re: [gentoo-dev] Regarding the tinderbox logs

[Diego Elio Pettenò]

> Nobody asked you to rollback months of work just because a few people
> can't deal with the way you submit your logs.

Actually somebody did, suggesting I shouldn't file bugs until the log's
attached. And then proceeded to suggest that converting everything to
python and using pybugz is a cakewalk (obviosuly without offering to
even start the work).

> But you should also
> respect their preferences. Each one of us has its personal way of
> dealing with his/her bugs (based on some loose standards and
> policies).

Sure. Preferences are great. Until said preferences mean that bugs that
_are_ 100% valid get closed, repeatedly, without being looked at.

Let's make an example. In the recent past, an update to binutils broke
some of the libbfd (the library coming with it) users as headers were
changed around. How many copies of the same bug do you think one has to
file before policies should overrule preferences?
(and this is not a matter of just refusing the log, it's refusing the
_bug_ which is extremely easy to reproduce).

> If someone doesn't like your way of submitting logs then
> just accept it. It is not like they go and remove your tinderbox links
> from bugs/packages they don't maintain.

Actually, that happened as well. Maybe you should actually review facts
before posting sure that you know that's going on. Just saying.

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] Regarding the tinderbox logs

[Rich Freeman]

On Wed, Oct 10, 2012 at 10:41 AM, Diego Elio Pettenò
<flameeyes@flameeyes.eu> wrote:
> Sure. Preferences are great. Until said preferences mean that bugs that
> _are_ 100% valid get closed, repeatedly, without being looked at.

I can't speak to the specifics of whatever the elephant in the room
is, but keep in mind that when you have 100 developers on a project
there is only so much you can tailor things to individual preferences.

I've gotten tinderbox bugs that are upstream issues that I personally
accept as valid but which I'll probably never be able to influence
upstream to change.  I just read them, appreciate them, and then leave
them open in the hope that I'll be bored one weekend and find some way
to fix it.  Usually I find Diego's bugs to be helpfully worded both
with helpful logs and background info which saves me having to explore
some arcane linking issue.

If the result of a tinderbox run is we get 500 bugs that are legit, 3
false positives that waste somebody's time, and 5 legit but
debate-ably unimportant bugs that particular maintainers don't want to
look at, I'd say we came out ahead.

This could be a culture thing.  At work I tend to work with large
regulated applications that often have hundreds of open bugs at any
time - some for years with no intention whatsoever to actually close
them.  Bug lists aren't really used like worklists in this context,
except for the few times a year everybody sits down and prioritizes
the list and figures out what is worth fixing, if anything.  So,
having a few open bugs assigned to me doesn't really bother me.

Maybe the solution is some kind of maintainer priority field for
personal productivity which is to be set by maintainers only and can
be filtered on if a maintainer just doesn't like seeing things in
their daily view.

Rich

Re: [gentoo-dev] Regarding the tinderbox logs

[Markos Chandras]

On Wed, Oct 10, 2012 at 3:41 PM, Diego Elio Pettenò
<flameeyes@flameeyes.eu> wrote:
> Actually, that happened as well. Maybe you should actually review facts
> before posting sure that you know that's going on. Just saying.
>
Ok

-- 
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

Re: [gentoo-dev] CIA replacement

[Dirkjan Ochtman]

On Mon, Oct 1, 2012 at 7:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
> Why exactly are we still using cvs?  Rather than building enhancements
> for cvs, why not just migrate everything to git, and spend our time
> building the git hooks/etc necessary to make this work?

The CIA/irc bot issue is pretty much entirely orthogonal to the git
migration issues, so I don't think you should hijack this thread over
that. :)

Cheers,

Dirkjan

Re: [gentoo-dev] CIA replacement

[Rich Freeman]

On Mon, Oct 1, 2012 at 4:19 PM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 7:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> Why exactly are we still using cvs?  Rather than building enhancements
>> for cvs, why not just migrate everything to git, and spend our time
>> building the git hooks/etc necessary to make this work?
>
> The CIA/irc bot issue is pretty much entirely orthogonal to the git
> migration issues, so I don't think you should hijack this thread over
> that. :)

Yup, started a separate thread on -scm.

My main point was that any issues with cvs are short-term issues.  Do
they really need to be fixed?

Rich

Re: [gentoo-dev] CIA replacement

[Jeff Horelick]

Well nenolod has written a CIA -> Irker proxy that (I believe) takes
commit messages designed to go to CIA and makes irker read them and
such, but i haven't looked into it:

https://github.com/nenolod/irker-cia-proxy

On 1 October 2012 11:21, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Hi Ben,
>
> On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot <yngwin@gentoo.org> wrote:
>> Since CIA.vc is dead [1], I think we should be looking into a
>> replacement service, or host our own [2].
>> Is infra already looking into this?
>>
>> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
>> 2: http://www.donarmstrong.com/posts/switching_to_kgb/
>
> Maybe someone with good cvs knowledge can contribute a hook for irker
> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> [1] http://www.catb.org/esr/irker/
>
> Best regards.
>
> --
> Rafael Goncalves Martins
> Gentoo Linux developer
> http://rafaelmartins.eng.br/
>