[gentoo-project] You have been researched (again)

[gentoo-project] You have been researched (again)

[Jeroen Roovers]

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 David Garcia, Marcelo Serrano Zanetti and Frank Schweitzer
Chair of Systems Design – www.sg.ethz.ch – ETH Zurich


 = Abstract

We analyze the relation between the emotions and the activity of
contributors in the Open Source Software project Gentoo. Our case study
builds on extensive data sets from the project’s bug tracking platform
bugzilla, to quantify the activity of contributors, and its mail
archives, to quantify the emotions of contributors by means of
sentiment analysis. The Gentoo project is known for a considerable drop
in development performance after the sudden retirement of a central
contributor. We analyze how this event correlates with the negative
emotions, both in bilateral email discussions with the central
contributor, and at the level of the whole community of contributors.
We then extend our study to consider the activity patters on Gentoo
contributors in general. We find that contributors are more likely to
become inactive when they express strong positive or negative emotions
in the bug tracker, or when they deviate from the expected value of
emotions in the mailing list. We use these insights to develop a
Bayesian classifier that detects the risk of contributors leaving the
project. Our analysis opens new perspectives for measuring online
contributor motivation by means of sentiment analysis and for real-time
predictions of contributor turnover in Open Source Software projects.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

http://arxiv.org/abs/1306.3612


Merry reading,
     jer

Re: [gentoo-project] You have been researched (again)

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]

On 2013-06-19 00:01, Jeroen Roovers wrote:
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>  David Garcia, Marcelo Serrano Zanetti and Frank Schweitzer
> Chair of Systems Design – www.sg.ethz.ch – ETH Zurich
> 
> 
>  = Abstract
> 
> We analyze the relation between the emotions and the activity of
> contributors in the Open Source Software project Gentoo. Our case study
> builds on extensive data sets from the project’s bug tracking platform
> bugzilla, to quantify the activity of contributors, and its mail
> archives, to quantify the emotions of contributors by means of
> sentiment analysis. The Gentoo project is known for a considerable drop
> in development performance after the sudden retirement of a central
> contributor. We analyze how this event correlates with the negative
> emotions, both in bilateral email discussions with the central
> contributor, and at the level of the whole community of contributors.
> We then extend our study to consider the activity patters on Gentoo
> contributors in general. We find that contributors are more likely to
> become inactive when they express strong positive or negative emotions
> in the bug tracker, or when they deviate from the expected value of
> emotions in the mailing list. We use these insights to develop a
> Bayesian classifier that detects the risk of contributors leaving the
> project. Our analysis opens new perspectives for measuring online
> contributor motivation by means of sentiment analysis and for real-time
> predictions of contributor turnover in Open Source Software projects.
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> 
> http://arxiv.org/abs/1306.3612
> 
> 
> Merry reading,
>      jer
> 

So, there ya go. Start taking happy pills and be more positive and motivating!

-- 
Mr. Aaron W. Swenson
Gentoo Linux Developer
Email : titanofold@gentoo.org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-project] You have been researched (again)

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

On 06/19/2013 03:02 AM, Aaron W. Swenson wrote:
> On 2013-06-19 00:01, Jeroen Roovers wrote:
>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>>  David Garcia, Marcelo Serrano Zanetti and Frank Schweitzer
>> Chair of Systems Design – www.sg.ethz.ch – ETH Zurich
>>
>> [...]
>>
>> http://arxiv.org/abs/1306.3612
>>
>>
>> Merry reading,
>>      jer
>>
> 
> So, there ya go. Start taking happy pills and be more positive and motivating!
> 

I've not read their new paper but the first one was just ridiculous.
They drew an edge from A to B if A reassigned a bug to B and intepreted
this as "A knows that B is an expert on the subject".  That is just
ridiculous since A is usually a bug wrangler and B is to be found in
metadata.  For instance the emacs team which gets very few bugs almost
never appears as B (and certainly not A).  This introduces all sorts of
bias.  I think it would be fair to say that their first study had very
low predictive power and the effects they saw were created by their method.

Cheers,
Thomas

-- 
Thomas Kahle


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: [gentoo-project] You have been researched (again)

[Jeroen Roovers]

On Wed, 19 Jun 2013 12:45:05 +0200
Thomas Kahle <tomka@gentoo.org> wrote:

> On 06/19/2013 03:02 AM, Aaron W. Swenson wrote:
> > On 2013-06-19 00:01, Jeroen Roovers wrote:
> >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >> - - - David Garcia, Marcelo Serrano Zanetti and Frank Schweitzer
> >> Chair of Systems Design – www.sg.ethz.ch – ETH Zurich
> >>
> >> [...]
> >>
> >> http://arxiv.org/abs/1306.3612
> >>
> >>
> >> Merry reading,
> >>      jer
> >>
> > 
> > So, there ya go. Start taking happy pills and be more positive and
> > motivating!
> > 
> 
> I've not read their new paper but the first one was just ridiculous.
> They drew an edge from A to B if A reassigned a bug to B and
> intepreted this as "A knows that B is an expert on the subject".
> That is just ridiculous since A is usually a bug wrangler and B is to
> be found in metadata.  For instance the emacs team which gets very
> few bugs almost never appears as B (and certainly not A).  This
> introduces all sorts of bias.  I think it would be fair to say that
> their first study had very low predictive power and the effects they
> saw were created by their method.

Good to (finally) see some critical review on the "laymen's" side of
the subject matter. I didn't vent any opinion on the last paper since,
you know, they're doing science in their niche and I'm not going to
tell them how to do it - social research has its fundamental problems
and I have my own to deal with.

In reading the second paper on the alleged decline of Gentoo's bug
tracker performance, I went back to the first one, and I don't really
see how performance degraded and never returned to form after that
<sarcasm>cataclysmic event</sarcasm> in ~2008.

I think it's fair to say things are done very differently now and that
lots of bug reports now don't get CLOSED/INVALID, REOPENED, wrongly
assigned, reassigned, re-reported, redefined, entangled with unrelated
but similar bugs, and DUPLICATEd so much any more, so I've asked the
researchers some questions[1] on this issue of measuring performance[2].

I should also point out that doing social science properly is rather
tricky, and that not getting entangled in the historiography of the
subject matter is important. Lots of things went on in 2008 - good and
bad things, and the community was under a lot of stress. And while I am
at it: do note that nobody was interviewed this time around. The second
paper regards Gentoo's poor bug tracker performance after Alice's
retirement as a well-established fact found in the first paper.


Regards,
     jer


[1] And within hours I have three replies from the same person in my
    inbox! I'll probably follow up on this.
[2] Frankly it stings a little to read about my vain efforts to improve
    bug wrangling, but conversely I might simply blame the research
    metrics. :)

Re: [gentoo-project] You have been researched (again)

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

Hi Jer,

On 06/19/2013 01:31 PM, Jeroen Roovers wrote:
[...] so I've asked the
> researchers some questions[1] on this issue of measuring performance[2].

I'd be interested in the follow-up.

Cheers,
Thomas


-- 
Thomas Kahle


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 381 bytes --]