[gentoo-soc] Congratulations!

[gentoo-soc] Congratulations!

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Congratulations to all of you who successfully passed your final 
evaluations! You've worked hard, and we really appreciate everything 
you've done to make it this far.

We'd love to feature your projects in a summary of this year's GSoC for 
Gentoo, which could show up on the main Gentoo homepage as well as the 
Google Open Source blog. If you're interested in getting some free 
publicity, please send a short (1-2 paragraphs, 3-5 sentences each) 
summary of your project and why it matters to this list in response to 
my email. I'll put together the responses into the complete summary.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

On 14:02 Mon 24 Aug     , Donnie Berkholz wrote:
> Congratulations to all of you who successfully passed your final 
> evaluations! You've worked hard, and we really appreciate everything 
> you've done to make it this far.
> 
> We'd love to feature your projects in a summary of this year's GSoC for 
> Gentoo, which could show up on the main Gentoo homepage as well as the 
> Google Open Source blog. If you're interested in getting some free 
> publicity, please send a short (1-2 paragraphs, 3-5 sentences each) 
> summary of your project and why it matters to this list in response to 
> my email. I'll put together the responses into the complete summary.

Oh yeah, it would also be really useful to know the status of your code. 
Is it already integrated into Gentoo, what's the plans for integrating 
it if not, etc.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Donnie Berkholz wrote:
> If you're interested in getting some free 
> publicity, please send a short (1-2 paragraphs, 3-5 sentences each) 
> summary of your project and why it matters to this list in response to 
> my email.

I'm unsure about the point of view to use for such a summary.
Let me try this:

  The stats project was about creation of a client and server collecting
  information about user setups to be able to better prioritize
  development efforts in Gentoo.  The data collected ranges from
  details about installed packages to compile flags to the selection of
  mirrors.  What data is submitted can be configured freely to match
  the need of privacy of the machine's administrator.

  With the collected data Gentoo will be able to answer all sorts of
  questions including:
  - Which non-system packages deserve extra attention?
  - What common setup mistakes do people make?
  - How well is the usage of mirrors balanced?

  The implementation is based on Smolt[1].  The stats project was and
  will be done in cooperation with Smolt upstream.

What do you think?



Sebastian


[1] https://fedorahosted.org/smolt/

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Donnie Berkholz wrote:
> Oh yeah, it would also be really useful to know the status of your code. 
> Is it already integrated into Gentoo, what's the plans for integrating 
> it if not, etc.

Part of the code is integrated at Smolt upstream.
The remaining part is planned to be merged upstream, too.
Neither upstream nor I want a fork and if upstream resources can handle
the extra Gentoo load we will just upgrade the instance running behind
http://www.smolts.org/ and submit the Gentoo data to that, too.

I should mention that load is not to be neglected, both looking at
processing time of a single submission as well as storage needs.  I hope
to further improve on the SQL alchemy logic used in my code to make
submission handling faster.

If any SQL alchemy experts around feel like mentoring me through the
process they are welcome to do so.


Anything else you want to know?



Sebastian

Re: [gentoo-soc] Congratulations!

[Nirbheek Chauhan]

On Sat, Aug 29, 2009 at 12:12 AM, Sebastian
Pipping<webmaster@hartwork.org> wrote:
> Part of the code is integrated at Smolt upstream.
> The remaining part is planned to be merged upstream, too.
> Neither upstream nor I want a fork and if upstream resources can handle
> the extra Gentoo load we will just upgrade the instance running behind
> http://www.smolts.org/ and submit the Gentoo data to that, too.
>

If this is combined with some sort of gentoo-infra based backups of
the data, it would be nice.

> I should mention that load is not to be neglected, both looking at
> processing time of a single submission as well as storage needs.  I hope
> to further improve on the SQL alchemy logic used in my code to make
> submission handling faster.
>

You mentioned that upstream needs to check if they can handle Gentoo
load, was this the bottleneck in that?

>
> Anything else you want to know?
>

I would say: Is there anything else *you* want to tell us? :)

-- 
~Nirbheek Chauhan

GNOME+Mozilla Team, Gentoo

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Nirbheek Chauhan wrote:
> If this is combined with some sort of gentoo-infra based backups of
> the data, it would be nice.

Any ideas how we could organize that?


> You mentioned that upstream needs to check if they can handle Gentoo
> load, was this the bottleneck in that?

As I understand load is quite high already (I think due to a high rate
of submissions).  The gentoo stuff adds to the per-submission processing
time considerably:  A machine with 500-1000 packages installed adds a
few thousand DB rows easily.

I have introduced batch processing (..) but it may need to be taken
further and it's not upstream yet.  I'm positive about it, upstream is
just a bit busy these days.


>> Anything else you want to know?
> 
> I would say: Is there anything else *you* want to tell us? :)

If I had a wish I'd say please update the running Redmine installation
to a recent version (without killing the current data, please :-)).

Also I'd really like to see more reviews, alchemy mentoring and
contributions from others.

There may be news on PackageMap again, too.



Sebastian

Re: [gentoo-soc] Congratulations!

[Nirbheek Chauhan]

On Sun, Aug 30, 2009 at 1:28 AM, Sebastian
Pipping<webmaster@hartwork.org> wrote:
> Nirbheek Chauhan wrote:
>> If this is combined with some sort of gentoo-infra based backups of
>> the data, it would be nice.
>
> Any ideas how we could organize that?
>

I don't know how the database is structured, but database backups of
gentoo-specific data would be nice to have. Simplest after that would
be for it to then just be stored on your dev.gentoo.org account (once
you're recruited ;). I believe dev.gentoo.org home directories are
part of the gentoo-infra backup system.

> As I understand load is quite high already (I think due to a high rate
> of submissions).  The gentoo stuff adds to the per-submission processing
> time considerably:  A machine with 500-1000 packages installed adds a
> few thousand DB rows easily.
>
> I have introduced batch processing (..) but it may need to be taken
> further and it's not upstream yet.  I'm positive about it, upstream is
> just a bit busy these days.
>

This sounds like a db-load distribution nightmare. Do keep us informed :)

> If I had a wish I'd say please update the running Redmine installation
> to a recent version (without killing the current data, please :-)).
>

I've actually been pondering that for a long time now, but I've been lazy :p

There are lots of improvements in the current upstream release. I'll
try to do the upgrade sometime this week, but it may take longer if
there are db migration problems.

> Also I'd really like to see more reviews, alchemy mentoring and
> contributions from others.
>

The key to that is to get a wider audience. Right now your audience is
limited to gentoo-soc + parts of gentoo-dev. If you could manage to
(say) get it into the handbooks as a recommended package, and/or into
the 10.0 release DVD, you'd get a flurry of activity w.r.t new users
and hence new contributors.

-- 
~Nirbheek Chauhan

GNOME+Mozilla Team, Gentoo

Re: [gentoo-soc] Congratulations!

[Eitan Mosenkis]

> We'd love to feature your projects in a summary of this year's GSoC for
> Gentoo, which could show up on the main Gentoo homepage as well as the
> Google Open Source blog. If you're interested in getting some free
> publicity, please send a short (1-2 paragraphs, 3-5 sentences each)
> summary of your project and why it matters to this list in response to
> my email.

Ingenue is a new project intended to provide a web-based system to
build complete linux systems on-demand.  It is built with a
frontend-backend split, with both ends written in PHP and
communicating mainly through a MySQL database.  It is currently in a
working state, though there is still much room for improvements and
will hopefully be deployed soon once a virtual server becomes
available for it.

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Nirbheek Chauhan wrote:
> I don't know how the database is structured,

All gentoo data except ..

 - log/batch entries
 - machine UUIDs

.. are in separate tables.  About 30 of them or so.

You can run

  # python database-admin.py  --fake-mysql create

in ./smoon/ to get all the table definitions printed.
Alternatively, I could send you a copy of that, just ask.


> but database backups of
> gentoo-specific data would be nice to have. Simplest after that would
> be for it to then just be stored on your dev.gentoo.org account (once
> you're recruited ;). I believe dev.gentoo.org home directories are
> part of the gentoo-infra backup system.

I guess my concerns were more about getting the data of the DB and
transfered in a well-organized manner.

Mike, can you imagine a way to do that best?


> I've actually been pondering that for a long time now, but I've been lazy :p
> 
> There are lots of improvements in the current upstream release. I'll
> try to do the upgrade sometime this week, but it may take longer if
> there are db migration problems.

Okay, cool.

Have you seen the Redmine ebuild in Bugzilla [1] or are you using that
already?


>> Also I'd really like to see more reviews, alchemy mentoring and
>> contributions from others.
> 
> The key to that is to get a wider audience. Right now your audience is
> limited to gentoo-soc + parts of gentoo-dev. If you could manage to
> (say) get it into the handbooks as a recommended package, and/or into
> the 10.0 release DVD, you'd get a flurry of activity w.r.t new users
> and hence new contributors.

I think that will not work as that's stuff that need to be done before
deploying the "real thing".  That's a chicken-and-egg thing, it seems.



Sebastian


[1] http://bugs.gentoo.org/show_bug.cgi?id=260575

Re: [gentoo-soc] Congratulations!

[Nirbheek Chauhan]

On Tue, Sep 1, 2009 at 8:18 AM, Sebastian Pipping<webmaster@hartwork.org> wrote:
> Okay, cool.
>
> Have you seen the Redmine ebuild in Bugzilla [1] or are you using that
> already?
>

Nah, it's safer to do it manually (while using svn to do updates). The
last release of redmine (0.8.x) needs Rails 2.1.2 which is masked for
removal in 60 days. For now I've updated 0.7 for security issues.

> I think that will not work as that's stuff that need to be done before
> deploying the "real thing".  That's a chicken-and-egg thing, it seems.
>

Well, chicken and egg problems need extra effort to resolve. But on
the bright side you *know* what the turning point will be :)


-- 
~Nirbheek Chauhan

GNOME+Mozilla Team, Gentoo

Re: [gentoo-soc] Congratulations!

[Stanislav Ochotnicky]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

On 14:19 Mon 24 Aug     , Donnie Berkholz wrote:
> On 14:02 Mon 24 Aug     , Donnie Berkholz wrote:
> > Congratulations to all of you who successfully passed your final 
> > evaluations! You've worked hard, and we really appreciate everything 
> > you've done to make it this far.
> > 
> > We'd love to feature your projects in a summary of this year's GSoC for 
> > Gentoo, which could show up on the main Gentoo homepage as well as the 
> > Google Open Source blog. If you're interested in getting some free 
> > publicity, please send a short (1-2 paragraphs, 3-5 sentences each) 
> > summary of your project and why it matters to this list in response to 
> > my email. I'll put together the responses into the complete summary.
> 
> Oh yeah, it would also be really useful to know the status of your code. 
> Is it already integrated into Gentoo, what's the plans for integrating 
> it if not, etc.

Hi,

hope it's not too late. I was away for a few days.

Let me try the summary:
Collagen is an automated build system for Gentoo aiming to create
database of package metadata, content and other information. This
database can be queried through web frontend built on top of Django
framework. Because of this, Collagen is database agnostic. As such
it also provides features to improve quality of Gentoo ebuilds by
testing different combinations of package dependencies. Collagen
still needs a lot of testing and polishing, but it already found
some faulty ebuilds. I also have a few ideas for future improvements
that I didn't include for now.

It is currently not deployed anywhere on Gentoo infrastructure, but
I plan to negotiate some testing machine where a lot of compilations
would not disturb normal operations :-) 

Enjoy your day/evening/night,

-- 
Stanislav Ochotnicky
Working for Gentoo Linux http://www.gentoo.org
Implementing Tree-wide collision checking and provided files database
http://soc.gentooexperimental.org/projects/show/collision-database
Blog: http://inputvalidation.blogspot.com/search/label/gsoc


jabber: sochotnicky@gmail.com
icq: 74274152
PGP: https://dl.getdropbox.com/u/165616/sochotnicky-key.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Nirbheek Chauhan wrote:
>> Have you seen the Redmine ebuild in Bugzilla [1] or are you using that
>> already?
> 
> Nah, it's safer to do it manually (while using svn to do updates).

What's safer about it?


> The
> last release of redmine (0.8.x) needs Rails 2.1.2 which is masked for
> removal in 60 days. For now I've updated 0.7 for security issues.

Thank you.


>> I think that will not work as that's stuff that need to be done before
>> deploying the "real thing".  That's a chicken-and-egg thing, it seems.
> 
> Well, chicken and egg problems need extra effort to resolve. But on
> the bright side you *know* what the turning point will be :)

I don't get a word :-)



Sebastian

Re: [gentoo-soc] Congratulations!

[Nirbheek Chauhan]

On Tue, Sep 1, 2009 at 9:48 PM, Sebastian Pipping<webmaster@hartwork.org> wrote:
> Nirbheek Chauhan wrote:
>> Nah, it's safer to do it manually (while using svn to do updates).
>
> What's safer about it?
>

Manual updation is more reliable than webapp-config updation,
especially with a non-dev ebuild :)

>>> I think that will not work as that's stuff that need to be done before
>>> deploying the "real thing".  That's a chicken-and-egg thing, it seems.
>>
>> Well, chicken and egg problems need extra effort to resolve. But on
>> the bright side you *know* what the turning point will be :)
>
> I don't get a word :-)
>

I mean, you need to be a komodo dragon to solve a chicken and egg
problem, but on the bright side you know that when you solve the
chicek and egg problem the project is self-sustaining.


-- 
~Nirbheek Chauhan

GNOME+Mozilla Team, Gentoo

Re: [gentoo-soc] Congratulations!

[Arne Babenhauserheide]

[-- Attachment #1: Type: Text/Plain, Size: 369 bytes --]

Am Freitag, 28. August 2009 20:42:21 schrieb Sebastian Pipping:
> Anything else you want to know?

Jepp: Since I'm a bit ouf of date: How can I submit data? 

:) 

Best wishes, 
Arne

--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- 
   - singing a part of the history of free software -
              http://infinite-hands.draketo.de


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Arne Babenhauserheide wrote:
> Jepp: Since I'm a bit ouf of date: How can I submit data? 
> 
> :) 

Thanks for asking.

These three steps should work:

  * Enable the sping overlay
  * Install app-admin/gentoo-smolt-9999
  * Run # smoltSendProfile --server=http://smolt.hartwork.org:45678/

Please let me know if it worked.



Sebastian

Re: [gentoo-soc] Congratulations!

[Arne Babenhauserheide]

[-- Attachment #1: Type: Text/Plain, Size: 494 bytes --]

Am Sonntag, 6. September 2009 17:57:15 schrieb Sebastian Pipping:
> Please let me know if it worked.

http://smolt.hartwork.org:45678/client/show/pub_efb5a4a3-974a-4ec2-9df3-4185a610b3d1

Does this also contain my profile with installes programs and such? 

Many thanks for your work! 

Best wishes, 
Arne

--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- 
   - singing a part of the history of free software -
              http://infinite-hands.draketo.de


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Arne Babenhauserheide wrote:
> http://smolt.hartwork.org:45678/client/show/pub_efb5a4a3-974a-4ec2-9df3-4185a610b3d1
> 
> Does this also contain my profile with installes programs and such? 

At the moment profile pages only show what upstream Smolt is collecting.

Would you rather like to see more on that page or be concerned about
privacy if we did?


I have triggered report re-creation, your data is now included:
http://smolt.hartwork.org:45678/static/stats/gentoo.html



Sebastian

Re: [gentoo-soc] Congratulations!

[Jeremy Olexa]

Sebastian Pipping wrote:

> I have triggered report re-creation, your data is now included:
> http://smolt.hartwork.org:45678/static/stats/gentoo.html

I notice an inconsistency. Everything is sorted by popularity, except 
"Archs", "Chosts", & "System profiles" - By design? It is hard to read 
at first glance and could get worse as more data is submitted.

-Jeremy

Re: [gentoo-soc] Congratulations!

[Arne Babenhauserheide]

[-- Attachment #1: Type: Text/Plain, Size: 1027 bytes --]

Am Montag, 7. September 2009 01:58:43 schrieb Sebastian Pipping:
> Arne Babenhauserheide wrote:
> > 
http://smolt.hartwork.org:45678/client/show/pub_efb5a4a3-974a-4ec2-9df3-4185a610b3d1
> Would you rather like to see more on that page or be concerned about
> privacy if we did?

Maybe you could include the installed portage systems (portage, pkgcore, 
paludis) :) 

Otherwise I think your concern about privacy is justified.  As long as the 
data is in the general report, I don't need it in my personal profile. 

> I have triggered report re-creation, your data is now included:
> http://smolt.hartwork.org:45678/static/stats/gentoo.html

Looks great! 

Could you post that to the statistics thread in the forums along with the 
instructions you sent me? 

- http://forums.gentoo.org/viewtopic-t-714162.html

Best wishes, 
Arne

--- --- --- --- --- --- --- --- --- 
Unpolitisch sein
heißt politisch sein, 
ohne es zu merken. 
- Arne (http://draketo.de)
--- --- --- --- --- --- --- --- --- 


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Arne Babenhauserheide wrote:
> Maybe you could include the installed portage systems (portage, pkgcore, 
> paludis) :) 

As of now it works with portage only ..


> Otherwise I think your concern about privacy is justified.  As long as the 
> data is in the general report, I don't need it in my personal profile. 

I see.


> Could you post that to the statistics thread in the forums along with the 
> instructions you sent me? 
> 
> - http://forums.gentoo.org/viewtopic-t-714162.html

Seems like Fauli was faster than me.



Sebastian

Re: [gentoo-soc] Congratulations!

[Arne Babenhauserheide]

[-- Attachment #1: Type: Text/Plain, Size: 958 bytes --]

Am Montag, 7. September 2009 20:52:27 schrieb Sebastian Pipping:
> Arne Babenhauserheide wrote:
> > Maybe you could include the installed portage systems (portage, pkgcore,
> > paludis) :)
>
> As of now it works with portage only ..

I don't mean work with it, but list it as especially interesting information 
:) 

Is it (easily) possible to query for special reports like "how many people 
have syste, XYZ installed"? 

Can we do these queries ourselves (maybe on an anonymized local copy)? 

> > - http://forums.gentoo.org/viewtopic-t-714162.html
>
> Seems like Fauli was faster than me.

:) 

Rightly so, why should you have all the work where others can (and want to) 
fill in the role just as well :) 

Best wishes, and congratulation once more! 

Arne

--- --- --- --- --- --- --- --- --- 
Unpolitisch sein
heißt politisch sein, 
ohne es zu merken. 
- Arne (http://draketo.de)
--- --- --- --- --- --- --- --- --- 


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Arne Babenhauserheide wrote:
> Is it (easily) possible to query for special reports like "how many people 
> have syste, XYZ installed"? 

Not at the moment.  Before we can offer such a feature we will need an
answer to this question:

  How much freedom in query building do users get?
  We cannot allow them to execute their own SQL but if it get's too
  limited the whole point of custom queries becomes useless.

That and probably more.


> Can we do these queries ourselves (maybe on an anonymized local copy)? 

Before providing SQL dumps there will also be question to be answered,
especially privacy related questions.
It starts with the mapping from public UUIDs to machine UUIDs.
If we published that anybody could send in data to update the entry for
your machine.  But that's an easy one.  The questions is what more needs
to be done or if that's enough already.


> Best wishes, and congratulation once more! 

Glad you like it.



Sebastian

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Arne Babenhauserheide wrote:
>>> Maybe you could include the installed portage systems (portage, pkgcore,
>>> paludis) :)
>> As of now it works with portage only ..
> 
> I don't mean work with it, but list it as especially interesting information 
> :) 

Actually I had the same idea, too.

I'm wondering if it should be done more in a generic way
(as in top N for each category) or a selected way with hardcoded sets of
packages to compare (selected package managers, selected audio players,
selected text editors, ...)

Extending the report generation code can takes quite some time so I try
to understand as much as possible about the report's future before I
actually do changes to it.

I feel like we could use a "big vision" first to save me refactorings.



Sebastian

Re: [gentoo-soc] Congratulations!

[Sebastian Pipping]

Jeremy Olexa wrote:
> Sebastian Pipping wrote:
> 
>> I have triggered report re-creation, your data is now included:
>> http://smolt.hartwork.org:45678/static/stats/gentoo.html
> 
> I notice an inconsistency. Everything is sorted by popularity, except
> "Archs", "Chosts", & "System profiles" - By design? It is hard to read
> at first glance and could get worse as more data is submitted.

By design, yes.  I made an exception with these as I felt it would not
be right with these tables.

In general a user can seek for the answer to (at least) these two
different questions:
- What's the top 1, top 2, top 3
- How does item X (e.g. use flag "mp3") rank

Doing both in a static table can only be done with JavaScript magic
that's currently beyond my expertise.
As a result I had to choose between sort-by-popularity and
sort-alphabetically.  While a user can still answer the "how does item X
rank" question using browser text search determining the top N from the
table in his head alone can be quite a hard task.  So I think the
sort-by-popularity approach mainly "hurts less".

I'm open to suggestions on how to improve presentation of the data.



Sebastian

Re: [gentoo-soc] Congratulations!

[Jeremy Olexa]

Sebastian Pipping wrote:
> Jeremy Olexa wrote:
>> Sebastian Pipping wrote:
>>
>>> I have triggered report re-creation, your data is now included:
>>> http://smolt.hartwork.org:45678/static/stats/gentoo.html
>> I notice an inconsistency. Everything is sorted by popularity, except
>> "Archs", "Chosts", & "System profiles" - By design? It is hard to read
>> at first glance and could get worse as more data is submitted.
> 
> By design, yes.  I made an exception with these as I felt it would not
> be right with these tables.
> 
> In general a user can seek for the answer to (at least) these two
> different questions:
> - What's the top 1, top 2, top 3
> - How does item X (e.g. use flag "mp3") rank
> 
> Doing both in a static table can only be done with JavaScript magic
> that's currently beyond my expertise.
> As a result I had to choose between sort-by-popularity and
> sort-alphabetically.  While a user can still answer the "how does item X
> rank" question using browser text search determining the top N from the
> table in his head alone can be quite a hard task.  So I think the
> sort-by-popularity approach mainly "hurts less".
> 
> I'm open to suggestions on how to improve presentation of the data.

Thanks for the explanation, I would just make it all sorted by 
popularity. Of course, just my opinion. :)

-Jeremy