[gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

I'm calling for participation in a new team working on things around
reports, bug analysis, heartbeat tracking in Gentoo:


Goals
=====
- help track the heartbeat/breath of Gentoo
    (bugs fixed per day, bug distribution per herd, ..)

- point to problems we may be overlooking in Gentoo
    (letting numbers speak to us and listening to them)

- help to prioritize when distributing available resources
    (specific calls for help, people switching teams, ..)


Data we will be working with
============================
- The Bug database  (XML from Bugzilla, ..)
- Package tree history  (VCS logs, ..)
- User setup details  (gentoo-smolt, ..)
- ..


Concrete tasks
==============
- bugday.g.o re-write  (work in progress)
- work on association between bugs and packages
  (for all bugs, not just bugday ones)
- get real numbers on how much active manpower we have


People
======
These poeple are in de facto:
- deathwing00
- sping
- <you here?>


If this is stuff which
- you have time for
- you are good at

please join us!



Sebastian

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Jesus Rivero (Neurogeek)]

On Sat, Apr 3, 2010 at 5:54 PM, Sebastian Pipping <sping@gentoo.org> wrote:
> I'm calling for participation in a new team working on things around
> reports, bug analysis, heartbeat tracking in Gentoo:
>
>
> Goals
> =====
> - help track the heartbeat/breath of Gentoo
>    (bugs fixed per day, bug distribution per herd, ..)
>
> - point to problems we may be overlooking in Gentoo
>    (letting numbers speak to us and listening to them)
>
> - help to prioritize when distributing available resources
>    (specific calls for help, people switching teams, ..)
>
>
> Data we will be working with
> ============================
> - The Bug database  (XML from Bugzilla, ..)
> - Package tree history  (VCS logs, ..)
> - User setup details  (gentoo-smolt, ..)
> - ..
>
>
> Concrete tasks
> ==============
> - bugday.g.o re-write  (work in progress)
> - work on association between bugs and packages
>  (for all bugs, not just bugday ones)
> - get real numbers on how much active manpower we have
>
>
> People
> ======
> These poeple are in de facto:
> - deathwing00
> - sping
> - <you here?>

   Sebastian, count me in. I believe this sort of stuff, or small
parts of what you are mentioning, was done in the newsletter. I
believe this is something real nice we should put somewhere public.

>
>
> If this is stuff which
> - you have time for
> - you are good at
>
> please join us!
>
>
>
> Sebastian
>

   Best regards,

--
Jesus Rivero (Neurogeek)
Gentoo Python Team

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Alec Warner]

On Sat, Apr 3, 2010 at 3:24 PM, Sebastian Pipping <sping@gentoo.org> wrote:
> I'm calling for participation in a new team working on things around
> reports, bug analysis, heartbeat tracking in Gentoo:

I will help with scripting; but probably not much else.

-A

>
>
> Goals
> =====
> - help track the heartbeat/breath of Gentoo
>    (bugs fixed per day, bug distribution per herd, ..)
>
> - point to problems we may be overlooking in Gentoo
>    (letting numbers speak to us and listening to them)
>
> - help to prioritize when distributing available resources
>    (specific calls for help, people switching teams, ..)
>
>
> Data we will be working with
> ============================
> - The Bug database  (XML from Bugzilla, ..)
> - Package tree history  (VCS logs, ..)
> - User setup details  (gentoo-smolt, ..)
> - ..
>
>
> Concrete tasks
> ==============
> - bugday.g.o re-write  (work in progress)
> - work on association between bugs and packages
>  (for all bugs, not just bugday ones)
> - get real numbers on how much active manpower we have
>
>
> People
> ======
> These poeple are in de facto:
> - deathwing00
> - sping
> - <you here?>
>
>
> If this is stuff which
> - you have time for
> - you are good at
>
> please join us!
>
>
>
> Sebastian
>
>
>

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/04/10 00:30, Jesus Rivero (Neurogeek) wrote:
>    Sebastian, count me in.

Cool!


> I believe this sort of stuff, or small
> parts of what you are mentioning, was done in the newsletter. I
> believe this is something real nice we should put somewhere public.

Can you elaborate on that?  I'm not sure what what you refer to.



Sebastian

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/04/10 00:42, Alec Warner wrote:
> I will help with scripting; but probably not much else.

Just right, there will be lots of scripting :-)

The current script dumping groud is hosted here:
http://git.goodpoint.de/?p=gentoo-bug-heartbeat.git;a=summary



Sebastian

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Jesus Rivero (Neurogeek)]

On Sat, Apr 3, 2010 at 6:27 PM, Sebastian Pipping <sping@gentoo.org> wrote:
> On 04/04/10 00:30, Jesus Rivero (Neurogeek) wrote:
>>    Sebastian, count me in.
>
> Cool!
>
>
>> I believe this sort of stuff, or small
>> parts of what you are mentioning, was done in the newsletter. I
>> believe this is something real nice we should put somewhere public.
>
> Can you elaborate on that?  I'm not sure what what you refer to.
>

   Yep. Take for instance this issue of GMN [1],  here you can find
some bugzie statistics like bugs assigned to teams/devs, bugs
resolutions and other cool things.

>
>
> Sebastian
>
>
[1] http://www.gentoo.org/news/en/gmn/20080121-newsletter.xml#bugs-stats

--
Jesus Rivero (Neurogeek)
Gentoo Python Team

[gentoo-dev] Re: [Gentoo Pheonix] Heartbeat team force

[Torsten Veller]

* Sebastian Pipping <sping@gentoo.org>:
> - Package tree history  (VCS logs, ..)

> - get real numbers on how much active manpower we have

I am generating monthly stats for gentoo-x86 for a year or so:
http://dev.gentoo.org/~tove/stats/gentoo-x86/

It lists the number of commits per month (cvs-log-20...) for all
"active" devs. Monthly commits per dev over time (dev/commits-$dev...).

cvs-log-sum.txt total number of commits per dev.

slacker.txt combines the last two month and adds further information for
devs without commits (date of last commit, date of last bugzilla
interaction, dev bug information, away status).

Since 2010 I also add cvs-log-all-.. which adds all not-retired devs
with gentoo-x86 access.

grep for Sum, Mean,...:
http://dev.gentoo.org/~tove/files/activedevs.txt
http://dev.gentoo.org/~tove/files/sum.txt
http://dev.gentoo.org/~tove/files/mean.txt
http://dev.gentoo.org/~tove/files/median.txt

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/04/10 00:24, Sebastian Pipping wrote:
> Concrete tasks
> ==============
> - [..]
> - work on association between bugs and packages
>   (for all bugs, not just bugday ones)

I've been playing with code answering the questions:
- How do bugs and packages relate?
- How is bugload distributed across developers?

I came to a point where
- I have something that starts to be useful
- I am facing troubles that need further
  consideration first, see below.


Current results
===============
Bug load per developer
----------------------
http://dev.gentoo.org/~sping/bug-heartbeat/report--bug-count-by-person.html

Bug load per herd
-----------------
http://dev.gentoo.org/~sping/bug-heartbeat/report--bug-count-by-herd.html

Bugs in a tree of category and package
--------------------------------------
http://dev.gentoo.org/~sping/bug-heartbeat/report--bugs-by-package.html

Bugs that may lack mention of package name
------------------------------------------
http://dev.gentoo.org/~sping/bug-heartbeat/report--bugs-without-package.html


Question to be answered
=======================
 - herds.xml does not hold membership lists for
   all projects, not even herds.  The load
   evaluator needs access to complete mappings
   to generate output close to reality.
   How can such a mapping be made without
   duplicating data?

 - Pulling XML out of Bugzilla does no longer
   feel right with this amount of data:
   it's 15,000 open bugs that need to be
   refreshed periodically in chunks of 100 bugs
   (Bugzilla's limit) each.  That's 150 request
   for a full sync.  How can that be improved?


Generating reports yourself
===========================
In case you want to play with the code yourself:
http://git.goodpoint.de/?p=gentoo-bug-heartbeat.git;a=summary

Run the report generator as following:

  # bzip2 -d xmlbugs.xml.bz2
  # python modules/heartbeat/reporter.py

The .bz2 file needed is up at
<http://dev.gentoo.org/~sping/bug-heartbeat/>, too.



Sebastian

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Ben de Groot]

On 7 April 2010 13:20, Sebastian Pipping <sping@gentoo.org> wrote:
>  - herds.xml does not hold membership lists for
>   all projects, not even herds.  The load
>   evaluator needs access to complete mappings
>   to generate output close to reality.
>   How can such a mapping be made without
>   duplicating data?

herds.xml should be the definitive resource for herds, so any herds
that are not on there should be added, and outdated information should
be brought up to date.

Other projects should have membership listed on their project pages.
Again, where these are outdated, they should be brought up to date.

Cheers,
-- 
Ben de Groot
Gentoo Qt project lead developer
Gentoo Wiki project lead

Re: [gentoo-dev] Re: [Gentoo Pheonix] Heartbeat team force

[Markos Chandras]

On Tuesday 06 April 2010 10:03:26 Torsten Veller wrote:
> * Sebastian Pipping <sping@gentoo.org>:
> > - Package tree history  (VCS logs, ..)
> > 
> > - get real numbers on how much active manpower we have
> 
> I am generating monthly stats for gentoo-x86 for a year or so:
> http://dev.gentoo.org/~tove/stats/gentoo-x86/
> 
> It lists the number of commits per month (cvs-log-20...) for all
> "active" devs. Monthly commits per dev over time (dev/commits-$dev...).
> 
> cvs-log-sum.txt total number of commits per dev.
> 
> slacker.txt combines the last two month and adds further information for
> devs without commits (date of last commit, date of last bugzilla
> interaction, dev bug information, away status).
> 
> Since 2010 I also add cvs-log-all-.. which adds all not-retired devs
> with gentoo-x86 access.
> 
> grep for Sum, Mean,...:
> http://dev.gentoo.org/~tove/files/activedevs.txt
> http://dev.gentoo.org/~tove/files/sum.txt
> http://dev.gentoo.org/~tove/files/mean.txt
> http://dev.gentoo.org/~tove/files/median.txt
Very useful. Thank you very much 
-- 
Markos Chandras (hwoarang)
Gentoo Linux Developer
Web: http://hwoarang.silverarrow.org

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/07/10 14:00, Ben de Groot wrote:
> On 7 April 2010 13:20, Sebastian Pipping <sping@gentoo.org> wrote:
>>  - herds.xml does not hold membership lists for
>>   all projects, not even herds.  The load
>>   evaluator needs access to complete mappings
>>   to generate output close to reality.
>>   How can such a mapping be made without
>>   duplicating data?
> 
> herds.xml should be the definitive resource for herds, so any herds
> that are not on there should be added, and outdated information should
> be brought up to date.

they are in there but keep their member list elsewhere.
an example is herd "bsd":

  <herd>
    <name>bsd</name>
    <email>bsd@nospam</email>
    <description>Support for *BSD system packages,
        and *BSD derived packages.</description>
    <maintainingproject>
      /proj/en/gentoo-alt/bsd/index.xml
    </maintainingproject>
  </herd>

now that i have a closer look maybe the herds itself are manageable,
besides the annoyance of being in another CVS tree:

  <subproject ref="/proj/en/gentoo-alt/bsd/obsd/index.xml"
      inheritmembers="yes" />
  <subproject ref="/proj/en/gentoo-alt/bsd/fbsd/index.xml"
      inheritmembers="yes" />
  <subproject ref="/proj/en/gentoo-alt/bsd/nbsd/index.xml"
      inheritmembers="yes" />

  <dev role="lead">welp</dev>
  <dev role="lead" description="Deputy lead">drizzt</dev>
  <dev role="member">aballier</dev>
  <dev role="member">the_paya</dev>

that looks usable.

with the herd "embedded" it's difficult though: the page
<http://www.gentoo.org/proj/en/base/embedded/index.xml>
lists members for it but the page source does not
hold it, explicitly.  where does it come from?



> Other projects should have membership listed on their project pages.
> Again, where these are outdated, they should be brought up to date.

seems like they use the same syntax, too.  i start to like guidexml :-)

some of the mail aliases i don't handle yet, could also be a problem
(especially those i don't have permission to look at).



sebastian

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Ben de Groot]

On 7 April 2010 20:55, Sebastian Pipping <sping@gentoo.org> wrote:
>  <dev role="lead">welp</dev>
>  <dev role="lead" description="Deputy lead">drizzt</dev>
>  <dev role="member">aballier</dev>
>  <dev role="member">the_paya</dev>
>
> that looks usable.

Except that it's out of date as welp is retired.

> with the herd "embedded" it's difficult though: the page
> <http://www.gentoo.org/proj/en/base/embedded/index.xml>
> lists members for it but the page source does not
> hold it, explicitly.  where does it come from?

It does, as can be seen in
http://www.gentoo.org/proj/en/base/embedded/index.xml?passthru=1

-- 
Ben de Groot
Gentoo Qt project lead developer
Gentoo Wiki project lead

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/07/10 21:24, Ben de Groot wrote:
>> with the herd "embedded" it's difficult though: the page
>> <http://www.gentoo.org/proj/en/base/embedded/index.xml>
>> lists members for it but the page source does not
>> hold it, explicitly.  where does it come from?
> 
> It does, as can be seen in
> http://www.gentoo.org/proj/en/base/embedded/index.xml?passthru=1

you refer to the members of the embedded team.
i was refering to the members of the embedded herd.
the rendered page lists

   dagger, kumba, pebenito, solar

to be working for that herd.  the link you gave does not
contain the word "dagger".  how does it get in there?



sebastian

[gentoo-dev] Re: [Gentoo Pheonix] Heartbeat team force

[Torsten Veller]

* Sebastian Pipping <sping@gentoo.org>:
> i was refering to the members of the embedded herd.
> the rendered page lists
> 
>    dagger, kumba, pebenito, solar
> 
> to be working for that herd.  the link you gave does not
> contain the word "dagger".  how does it get in there?

From herds.xml. Really.

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On Wed, Apr 07, 2010 at 01:20:11PM +0200, Sebastian Pipping wrote:
> Current results
> ===============
> Bug load per developer
> ----------------------
> http://dev.gentoo.org/~sping/bug-heartbeat/report--bug-count-by-person.html
What's the actual math that you're using (it wasn't clear in your repo).

> Question to be answered
> =======================
>  - herds.xml does not hold membership lists for
>    all projects, not even herds.  The load
>    evaluator needs access to complete mappings
>    to generate output close to reality.
>    How can such a mapping be made without
>    duplicating data?
herds, project files, aliases: 
- 3 sets of non-overlapping data.
- lets ignore the private aliases for now, very few of them should get
  package bugs.

Report on unresolved bugs owned by hidden aliases:
- 11 of the 93 aliases have bugs
- 258 bugs in total
- Those with > 10 bugs:
	- infra, 78
	- bugzilla, 47
	- retirement, 28
	- mirror-admin, 21
	- forum-mods, 20
	- devrel, 19
	- recruiters, 19

>  - Pulling XML out of Bugzilla does no longer
>    feel right with this amount of data:
>    it's 15,000 open bugs that need to be
>    refreshed periodically in chunks of 100 bugs
>    (Bugzilla's limit) each.  That's 150 request
>    for a full sync.  How can that be improved?
Lets talk more about what queries you're using, and we can probably work
something out.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] [Gentoo Pheonix] Heartbeat team force

[Sebastian Pipping]

On 04/08/10 21:16, Robin H. Johnson wrote:
> On Wed, Apr 07, 2010 at 01:20:11PM +0200, Sebastian Pipping wrote:
>> Current results
>> ===============
>> Bug load per developer
>> ----------------------
>> http://dev.gentoo.org/~sping/bug-heartbeat/report--bug-count-by-person.html
> What's the actual math that you're using (it wasn't clear in your repo).

let me try in python:

load[dev] = bugs[dev] \
            + reduce(sum, [bugs(h)/members(h) for h in herds[dev]])

it's personal bugs plus a fraction of all herds you're in.
makes sense?


>>  - Pulling XML out of Bugzilla does no longer
>>    feel right with this amount of data:
>>    it's 15,000 open bugs that need to be
>>    refreshed periodically in chunks of 100 bugs
>>    (Bugzilla's limit) each.  That's 150 request
>>    for a full sync.  How can that be improved?
> Lets talk more about what queries you're using, and we can probably work
> something out.

speaking of queries would limit me in what i may ask the data in the
future.  i have a full dump on open bugs more or less so i can ask them
whatever i like.  it's more flexible to me and makes much easier code
than SQL stuff would.

i guess that doesn't make it easier?



sebastian