[gentoo-dev] Stats test server running, please check it out

[gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Hello again!


I have set up a test server of the current stats code.  If you have a
minute to check it out that would rock.  I'm very interested in overall
feedback and bug reports.

To check it out please do as following:


0)  Make sure you have these packages installed:
        sys-apps/portage
        dev-util/git
        dev-python/rhpl
        dev-python/urlgrabber
        dev-python/simplejson
        dev-python/dbus-python

1)  Check out the code
        # git clone git://git.goodpoint.de/smolt-gentoo.git
        # cd smolt-gentoo/client
        # git checkout --track -b gentoo origin/gentoo

2)  Verify my server is running at the very moment by browsing to
        http://smolt.hartwork.org:45678/static/stats/gentoo.html

3)  Run the client  (which asks and shows details before submission)
        # python sendProfile.py \
            --server=http://smolt.hartwork.org:45678/


Please note that a summary update is triggered manually so gentoo.html
will not update instantly.  You can ping me on Freenode though, to make
me trigger an update earlier.

To disable certain classes of data for privacy reasons you can put this
in ~/.smolt/client.cfg

    [gentoo]
    arch_related = True
    compile_flags = True
    features = True
    global_use_flags = True
    installed_packages = True
    installed_packages_use_flags = True
    mirrors_sync = True
    mirrors_distfiles = True
    package_mask = True
    repositories = True
    system_profile = True

and turn "True" to "False" where necessary.

Have fun and please share your experience with it.



Sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Sebastian Pipping wrote:
> 0)  Make sure you have these packages installed:
>         sys-apps/portage
>         dev-util/git
>         dev-python/rhpl
>         dev-python/urlgrabber
>         dev-python/simplejson
>         dev-python/dbus-python

From what I hear not everyone has the HAL daemon installed.
HAL is required, too:

        sys-apps/hal



Sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Sebastian Pipping wrote:
> 3)  Run the client  (which asks and shows details before submission)
>         # python sendProfile.py \
>             --server=http://smolt.hartwork.org:45678/

I forgot to mention you need to create a random machine id one way or
another before you can submit data.  An easy and transparent way to do
that would be running this command:

  # sudo sh -c 'cat /proc/sys/kernel/random/uuid > /etc/smolt/hw-uuid'

The smolt ebuild is doing the very same.


We have data from 8 machines so far, you can be number 9.
http://smolt.hartwork.org:45678/static/stats/gentoo.html




Sebastian

[gentoo-dev] Re: Stats test server running, please check it out

[Mark Bateman]

Sebastian Pipping <webmaster <at> hartwork.org> writes:

> 
> Hello again!
> 
> I have set up a test server of the current stats code.  If you have a
> minute to check it out that would rock.  I'm very interested in overall
> feedback and bug reports.
> 
> To check it out please do as following:
> 
> 0)  Make sure you have these packages installed:
>         sys-apps/portage
>         dev-util/git
>         dev-python/rhpl
>         dev-python/urlgrabber
>         dev-python/simplejson
>         dev-python/dbus-python
> 

emerge rhpl -va

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ] net-wireless/wireless-tools-29  USE="nls -multicall" 288 kB
[ebuild  N    ] dev-python/rhpl-0.213  236 kB

Total: 2 packages (2 new), Size of downloads: 523 kB

Would you like to merge these packages? [Yes/No] 




is this correct? 

Re: [gentoo-dev] Stats test server running, please check it out

[Dirkjan Ochtman]

On Mon, Aug 10, 2009 at 12:08, Sebastian Pipping<webmaster@hartwork.org> wrote:
> 0)  Make sure you have these packages installed:
>        dev-python/rhpl
>        dev-python/urlgrabber
>        dev-python/dbus-python

What do you need these for?

Cheers,

Dirkjan

Re: [gentoo-dev] Re: Stats test server running, please check it out

[Sebastian Pipping]

Mark Bateman wrote:
> emerge rhpl -va
> 
> These are the packages that would be merged, in order:
> 
> Calculating dependencies... done!
> [ebuild  N    ] net-wireless/wireless-tools-29  USE="nls -multicall" 288 kB
> [ebuild  N    ] dev-python/rhpl-0.213  236 kB
> 
> Total: 2 packages (2 new), Size of downloads: 523 kB
> 
> Would you like to merge these packages? [Yes/No] 
> 
> 
> 
> 
> is this correct? 

yes,  thanks for asking.

(though my code neither uses that component nor do i know what it does)



sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Dirkjan Ochtman wrote:
> On Mon, Aug 10, 2009 at 12:08, Sebastian Pipping<webmaster@hartwork.org> wrote:
>> 0)  Make sure you have these packages installed:
>>        dev-python/rhpl
>>        dev-python/urlgrabber
>>        dev-python/dbus-python
> 
> What do you need these for?

a short and correct answer would be that smolt 1.2 already depends on
them.  i cannot say much more, except that urlgrabber might be
replaceable.  i don't use any of them in the gentoo-specific extensions.



sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Federico Ferri]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastian Pipping wrote:
> Hello again!
>
>
> I have set up a test server of the current stats code.  If you have
> a minute to check it out that would rock.  I'm very interested in
> overall feedback and bug reports.

1st: you could make an ebuild for it ;)

2nd: how to improve the output: you could make every data an
hyperlink: that would help understand better the contents
example:
Archs:
  hyperlink arch name to wikipedia perhaps

CFLAGS
  can link the flag name to GCC user guide relevant section, or to
http://en.gentoo-wiki.com/wiki/CFLAGS#<flagname>
same for other flags and MAKEOPTS

FEATURES
  can link feature to make.conf man page:
http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
rendered with no anchors over variable/features, so you could do
better ^__^

USE flags
  can link to
http://gentoo-portage.com/Search?search=&use=<useflagname> (now
appears down)

System profiles:
  can link to profiles in cvs

Package atoms can link to packages.g.o or to gentoo-portage.com



those percentages are just funny:

33 (275.0 %)    58 (483.3 %)  25 (208.3 %)

Total 12 (100.0 %)  [[ yeah, total is 100%, sometimes ]]

kde 3 (25.0 %) others 554 (4616.67 %) Total 1066 (8883.33 %)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqAzOkACgkQV/B5axfzrPu9YwCfekbUmNJ0LfJ1PxqgAcCuFiD8
STYAn1xjdrMNmWusEV+EZlHPrL5Hr8gH
=ZcgR
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Federico Ferri wrote:
> 1st: you could make an ebuild for it ;)

i just made one but it's not that useful yet, as the code is not
runnable from any location yet...


> 2nd: how to improve the output: you could make every data an
> hyperlink: that would help understand better the contents

actually there's a task for that open already:
http://soc.gentooexperimental.org/issues/show/51

i completely agree about it.


> FEATURES
>   can link feature to make.conf man page:
> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
> rendered with no anchors over variable/features, so you could do
> better ^__^

the output of neither

  # man2html -r make.conf.5 > make.conf.5.html

nor

  # groff -man -Thtml make.conf.5 > make.conf.5.html

make me really happy

do you know any alternative coming with more anchors and external CSS
support?


> those percentages are just funny:
> 
> 33 (275.0 %)    58 (483.3 %)  25 (208.3 %)
> 
> Total 12 (100.0 %)  [[ yeah, total is 100%, sometimes ]]
> 
> kde 3 (25.0 %) others 554 (4616.67 %) Total 1066 (8883.33 %)

most of these numbers show how much of something users have on average.
i agree this is not "user friendly".  Task now over here:
http://soc.gentooexperimental.org/issues/show/54



sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Federico Ferri]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastian Pipping wrote:
> Federico Ferri wrote:
>> FEATURES
>>   can link feature to make.conf man page:
>> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
>> rendered with no anchors over variable/features, so you could do
>> better ^__^
>
> the output of neither
>
>   # man2html -r make.conf.5 > make.conf.5.html
>
> nor
>
>   # groff -man -Thtml make.conf.5 > make.conf.5.html
>
> make me really happy
>
> do you know any alternative coming with more anchors and external CSS
> support?

# nope for css, but you could do something like this:
# imagine you have the feature list in a file
# (I don't know how to extract it automatically, but perhaps someone does)

$ cat featurelist
assume-digests
buildpkg
buildsyspkg
...
userpriv
usersandbox
usersync
webrsync-gpg

$ bzcat /usr/share/man/man5/make.conf.5.bz2 | man2html -r > make.html

# basically now you have to match <DT><B>featurename</B>
# and add a anchor tag to it:

$ eval `echo sed; cat featurelist | sed 's,^\(.*\),-e
'\''s:<DT><B>\1</B>:<DT><B><A NAME="feature_\1">\1</A></B>:'\'',';
echo make.html` > make_anchors.html

this will give you anchor to features, e.g.
make_anchor.html#feature_buildpkg


a bit tricky the sed - could be easier, but this version is fast
because it builds a giant sed expression instead of calling sed n times.

- --
Federico Ferri

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqBqKwACgkQV/B5axfzrPsegwCdFo95i26IF9+jeUSLntVI4nhS
msgAnRw4I4D1MwPUf1yBY8gJh3PHtX9S
=cx8u
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Stats test server running, please check it out

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

Hi,

Sebastian Pipping <webmaster@hartwork.org>:
> Have fun and please share your experience with it.

 Works fine...really great work.  This Summer of code seems to bring
out some results for Gentoo finally.

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Federico Ferri wrote:
> 1st: you could make an ebuild for it ;)

Done.  Look for  app-admin/gentoo-smolt-9999  in the "sping" overlay.

Running

  # smoltSendProfile --server=http://smolt.hartwork.org:45678/

should work fine after.



Sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Federico Ferri wrote:
> 2nd: how to improve the output: you could make every data an
> hyperlink: that would help understand better the contents
> example:
> Archs:
>   hyperlink arch name to wikipedia perhaps

Done, linking to

  http://packages.gentoo.org/arch/${arch}


> CFLAGS
>   can link the flag name to GCC user guide relevant section, or to
> http://en.gentoo-wiki.com/wiki/CFLAGS#<flagname>
> same for other flags and MAKEOPTS

Later, server seems down.


> FEATURES
>   can link feature to make.conf man page:
> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
> rendered with no anchors over variable/features, so you could do
> better ^__^

Done,  linking to
http://smolt.hartwork.org:45678/static/man/man5/make.conf.5.html#${feature}

If anyone feels like playing with  manpage-screen.css  please do and
come back to me.

Also, the man page does not seem to list these features:

  cvs
  preserve-libs
  split-debug
  splitdebug
  unmerge-logs

Either people use invalid features (you tell me) or the man page needs
an update.  Maybe both ("split-debug" vs "splitdebug")?


> USE flags
>   can link to
> http://gentoo-portage.com/Search?search=&use=<useflagname> (now
> appears down)

Still down, linking to

  http://gpo.zugaina.org/Search?search=&use=${use}

for now.


> System profiles:
>   can link to profiles in cvs

Are you sure about this?  Will that be fun to look at?


> Package atoms can link to packages.g.o or to gentoo-portage.com

Done, linking to

  http://packages.gentoo.org/package/${CATEGORY}/${PN}


To be able to link all packages properly we'd need a version of
http://gpo.zugaina.org/  that includes the trees "gentoo" and "funtoo".

ycarus, could the server handle that load-wise?  would it be easy to
extend the current service or set up an all-packages-version in parallel?



Sebastian

[gentoo-dev] Re: Stats test server running, please check it out

[Duncan]

Sebastian Pipping posted on Wed, 12 Aug 2009 14:31:48 +0200 as excerpted:

> Also, the man page does not seem to list these features:
> 
>   cvs
>   preserve-libs
>   split-debug
>   splitdebug
>   unmerge-logs
> 
> Either people use invalid features (you tell me) or the man page needs
> an update.  Maybe both ("split-debug" vs "splitdebug")?

Note that some FEATURES may in fact trigger various portage bashrc 
customizations.  While Ed Catmur disappeared some time ago now, I still 
use some of his bashrc customizations, including FEATURES=patchtree, for 
instance.  There were others, but many of them were more dev focused than 
I as a user am, so it's likely that various devs still use a possibly 
further customized version of the same scripts, today.

The point being, due to customization it's not reasonable to expect a 
full mapping, FEATURES or other variable, tho at least some of the above 
would appear to be legitimate mainline portage if perhaps not yet well 
documented FEATURES.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Yannick Chabanois wrote:
> Not really a problem, I can had gentoo and funtoo trees in
> gpo.zugaina.org. This will be available very soon ( next week ?) with
> the new version of the site.
> All source code of gpo.zugaina.org will be made available in the same time.

That's great news!

Please keep me up to date about this.
And thank you for such a great service.



Sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

On Wednesday 12 August 2009, Sebastian Pipping wrote:
>   cvs

Missing documentation, but valid, see make.conf.example

>   split-debug

This is probably invalid.

>   preserve-libs
>   splitdebug
>   unmerge-logs

These are documented in Portage 2.2 make.conf(5).


Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Robert Buchholz wrote:
>>   preserve-libs
>>   splitdebug
>>   unmerge-logs
> 
> These are documented in Portage 2.2 make.conf(5).

Just updated to make.conf from revision 13844.
http://smolt.hartwork.org:45678/static/man/man5/make.conf.5.html


My "man2tidyhtml" wrapper around manServer [1] is now hosted here [2].

An ebuild for manServer is in the pipeline, currently waiting for the
next reply from upstream.  An ebuild for man2tidyhtml will follow.



Sebastian


[1] http://www.squarebox.co.uk/users/rolf/download/manServer.shtml
[2] http://git.goodpoint.de/?p=man2tidyhtml.git;a=summary

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Sebastian Pipping wrote:
> An ebuild for manServer is in the pipeline, currently waiting for the
> next reply from upstream.

manServer ebuild here, new 1.08 release from upstream
http://git.goodpoint.de/?p=overlay-sping.git;a=tree;f=app-text/manserver



Sebastian

Re: [gentoo-dev] Stats test server running, please check it out

[Sebastian Pipping]

Sebastian Pipping wrote:
> An ebuild for man2tidyhtml will follow.

Here it is:
http://git.goodpoint.de/?p=overlay-sping.git;a=tree;f=app-text/man2tidyhtml



Sebastian