Re: [gentoo-project] You have been researched (again)

[Jeroen Roovers <jer@gentoo.org>]

On Wed, 19 Jun 2013 12:45:05 +0200
Thomas Kahle <tomka@gentoo.org> wrote:

> On 06/19/2013 03:02 AM, Aaron W. Swenson wrote:
> > On 2013-06-19 00:01, Jeroen Roovers wrote:
> >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >> - - - David Garcia, Marcelo Serrano Zanetti and Frank Schweitzer
> >> Chair of Systems Design – www.sg.ethz.ch – ETH Zurich
> >>
> >> [...]
> >>
> >> http://arxiv.org/abs/1306.3612
> >>
> >>
> >> Merry reading,
> >>      jer
> >>
> > 
> > So, there ya go. Start taking happy pills and be more positive and
> > motivating!
> > 
> 
> I've not read their new paper but the first one was just ridiculous.
> They drew an edge from A to B if A reassigned a bug to B and
> intepreted this as "A knows that B is an expert on the subject".
> That is just ridiculous since A is usually a bug wrangler and B is to
> be found in metadata.  For instance the emacs team which gets very
> few bugs almost never appears as B (and certainly not A).  This
> introduces all sorts of bias.  I think it would be fair to say that
> their first study had very low predictive power and the effects they
> saw were created by their method.

Good to (finally) see some critical review on the "laymen's" side of
the subject matter. I didn't vent any opinion on the last paper since,
you know, they're doing science in their niche and I'm not going to
tell them how to do it - social research has its fundamental problems
and I have my own to deal with.

In reading the second paper on the alleged decline of Gentoo's bug
tracker performance, I went back to the first one, and I don't really
see how performance degraded and never returned to form after that
<sarcasm>cataclysmic event</sarcasm> in ~2008.

I think it's fair to say things are done very differently now and that
lots of bug reports now don't get CLOSED/INVALID, REOPENED, wrongly
assigned, reassigned, re-reported, redefined, entangled with unrelated
but similar bugs, and DUPLICATEd so much any more, so I've asked the
researchers some questions[1] on this issue of measuring performance[2].

I should also point out that doing social science properly is rather
tricky, and that not getting entangled in the historiography of the
subject matter is important. Lots of things went on in 2008 - good and
bad things, and the community was under a lot of stress. And while I am
at it: do note that nobody was interviewed this time around. The second
paper regards Gentoo's poor bug tracker performance after Alice's
retirement as a well-established fact found in the first paper.


Regards,
     jer


[1] And within hours I have three replies from the same person in my
    inbox! I'll probably follow up on this.
[2] Frankly it stings a little to read about my vain efforts to improve
    bug wrangling, but conversely I might simply blame the research
    metrics. :)