[gentoo-dev] A constructive reommendation on stablity improvemt

[gentoo-dev] A constructive reommendation on stablity improvemt

[Igor]

Hi all,

Hereby the summary of my personal suggestions to increase GENTOO stability and help it's maintainers 
and developers. 

1. make.conf

   Add 

   BUG_REPORT_URL  "http://"(or similar name)
   BUG_REPORT ON/OFF
   BUG_REPORT_LEVEL 

   to make.conf  

   (several URLs should be supported for several sever-side contributors versions)

   by default reporting is turned off. 

2. USE flag 

   use some flag like curl for emerge to disable bug reporting functionality for those 
   who are not trusting, afraid of leaks or who is not willing to participate. 

3. Python URL reporting class 
   Add HTTP based bug reporting mechanics to emerge WITHOUT implementing 
   server-side mechanics. 

4. Carefully document bug reporting mechanics. 

5. Wait for server-side implementations to appear. They will appear once 
   emerge can report. And once it's clear for the rest that there is seriously 
   going to be a change soon. 

   I do honestly believe it should be started by portage team, not by any one else.
   They're the key. Reporting callbacks are not difficult and not time consuming. 
   But only the emerge team has the experience to start it right. They have to get 
   into this idea themselves, approving it by introducing the interface. 

   In my point of view - it's fair. 

Expecting: 

  Once interface is there several teams would develop sever side platform. I know there 
  are people out there who will present their own versions. 
  The platforms would be tested with existing emerge reporting interface and emerge reporting 
  interface will evolve with time. 

  With time the best server platform will be adopted by http://bugreport.gentoo.org  (or 
  something like that). 

  
When this stage is over:

  The new tools will be developed, like developer e-mail notifications, automated bug reporting, 
  difficult to imagine that far.

  GENTOO would get over portage hell and a new sub-system addressing GETNOO functional 
  stabilization are developed.

I expect we all get many new users for GENTOO, 

Common guys GENTOO has a lot of potential, problems are fixable once you know more about 
them, GENTOO just needs a little love, some statistics based stabilizations tools,
a compass for developers, a lot of experienced developers here - they will use gathered data 
to make changes nobody even can think about now. Turn your friendly face to the users - 
they're afraid of gentoo only because it's difficult to assemble packages, if this 
shortcoming is fixed - it might be the best linux distro on the planet. All it requires 
is a small reporting class in portage from an experienced portage developer. 1000 lines 
of code max and there is a new future for everyone.


-- 
Best regards,
 Igor                          mailto:lanthruster@gmail.com

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

On 10 August 2014 04:18, Igor <lanthruster@gmail.com> wrote:

> 5. Wait for server-side implementations to appear. They will appear once
>    emerge can report. And once it's clear for the rest that there is
> seriously
>    going to be a change soon.
>

It really needs to be designed from the server side, not the client side.

And defaulting on is a bad default.

The reason the server part has to come first is the server side serves as
reference implementation of how the communication protocol is to work, and
how reports are to be aggregated.

The best part here, is the server can also be designed without needing to
modify the portage source.

The server can be created, and a dummy client can be created that speaks
its language and submits "sample" reports of some kind.

Once the server is doing what it needs to do as a basic feature set in
recording and storing reported data, the dummy client can be enhanced to be
an out-of-band too that reports information by scraping portage logs.

And using that process means we can iteratively refine what needs to happen
without hamstringing us by being tightly coupled with the portage release
process.

Once we have a reference and useful enough server and protocol
specification, *THEN* we can talk about integrating it with portage and
other clients.

And its much more likely we'll have competing clients in circulation that
speak the protocol than it is that we'll have competing servers all working
with a unified client.

^ This much is apparent from observing the CPAN model, we have a variety of
client libraries because different ones prove more practical for end users
for certain workflows, but we only have one server that is the recipient
for the reports.

There is of course value in being *ABLE* to provide competing servers, but
competing servers are way outside the problem domain that proves relevant
to Gentoo.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 3017 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Igor]

[-- Attachment #1: Type: text/plain, Size: 2892 bytes --]

Kent,

Saturday, August 9, 2014, 9:42:18 PM, you wrote:

Thank you for your opinion.
I really don't think data processing unit comes first.

Communication protocol is already there - it's HTTP, method POST
HTTP protocol is already with Python - CURL, WGET
A reliable server ready to accept data from portage is all so there - 
it's Apache web server. 

What we don't have to start with portage class is just a number of 
parameters to submit. 

If you're experienced with portage - help work out parameters you consider useful 
for server.

Example: 

system_arch : 
system_name : 
system_environment: 

portage_message :
portage_tree: 

package_failed: xorg-1.2.3 /example/ 


etc. 

Once the list is ready -> add it reporter to portage -> and the server side data processing unit 
will appear. 



On 10 August 2014 04:18, Igor <lanthruster@gmail.com> wrote:
5. Wait for server-side implementations to appear. They will appear once
   emerge can report. And once it's clear for the rest that there is seriously
   going to be a change soon.
It really needs to be designed from the server side, not the client side.

And defaulting on is a bad default.

The reason the server part has to come first is the server side serves as reference implementation of how the communication protocol is to work, and how reports are to be aggregated.

The best part here, is the server can also be designed without needing to modify the portage source.

The server can be created, and a dummy client can be created that speaks its language and submits "sample" reports of some kind.

Once the server is doing what it needs to do as a basic feature set in recording and storing reported data, the dummy client can be enhanced to be an out-of-band too that reports information by scraping portage logs.

And using that process means we can iteratively refine what needs to happen without hamstringing us by being tightly coupled with the portage release process.

Once we have a reference and useful enough server and protocol specification, *THEN* we can talk about integrating it with portage and other clients.

And its much more likely we'll have competing clients in circulation that speak the protocol than it is that we'll have competing servers all working with a unified client.

^ This much is apparent from observing the CPAN model, we have a variety of client libraries because different ones prove more practical for end users for certain workflows, but we only have one server that is the recipient for the reports.

There is of course value in being *ABLE* to provide competing servers, but competing servers are way outside the problem domain that proves relevant to Gentoo.


-- 
Kent 

KENTNL - https://metacpan.org/author/KENTNL





-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 4349 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Jeroen Roovers]

On Sat, 9 Aug 2014 22:15:17 +0400
Igor <lanthruster@gmail.com> wrote:

> I really don't think data processing unit comes first.

The thing is, almost everything for automated build failure reporting
is in place on the client side. Build logs are automatically generated
and `emerge --info =$CATEGORY/${PF}' is easy, which is what we already
require for bug reports.

Uploading two files to a web server is trivial viewed from the client
side.

> Communication protocol is already there - it's HTTP, method POST
> HTTP protocol is already with Python - CURL, WGET
> A reliable server ready to accept data from portage is all so there - 
> it's Apache web server. 

That's a matter of implementation. It doesn't concern the design, which
is what you want to be dealing with right now.

> What we don't have to start with portage class is just a number of 
> parameters to submit. 

Yes, we do. See above. Improvements are suggested and implemented from
time to time (see perhaps bug #436294).

> Once the list is ready -> add it reporter to portage -> and the
> server side data processing unit will appear. 

Processing what?

Example 1: Dr. A. Spammer wants to reach Gentoo developers because he
wants to sell them stuff, marry them or give them ONE MILLION DOLLARS.
He can spam mailing lists and aliases but they have spam filters or
require subscriptions or both. Or he can trivially upload his spam to
the build failure log processing server, which will do all the work of
distributing those "bug reports".

The problem: How do you propose to keep that new service clean? We'd
need a spam filter and user authentication, which is a lot harder to do
on the client side (than merely uploading to an open server) and
requires proper user management on the server side too, which comes
with security concerns and manual involvement.

Example 2: Mr. I.B. User configured his system with CFLAGS=-fomg-faster
and now it generates a ton of build failures. All of these should go
to /dev/null, but there we are running an automated service that cannot
be taught how to distinguish between genuine bug reports and PEBKAC.

The problem: How do you propose to filter out all the junk and promote
genuine issues to "bug reports"?

Both examples should make clear why we have a bug tracker and not an
automated build failure reporting facility - it works rather well and
it stops dead a lot of spam and bad reports.

It currently takes a handful of people a lot of dedication and time to
weed out the crappy bug reports (support requests, misconfiguration,
out of date packages, duplicates) from the real ones, but since filing
bug reports takes an actual effort by the reporter, we probably don't
see that many bad ones.

With an automated build failure service in place, we should be prepared
to start manually processing an order of magnitude more reports and
still decide intelligently which are good and which aren't. Without a
really good interface (and a larger team of people dedicated to that
job), people will probably start ignoring the entire thing pretty
quickly.

Creating a better (command line) interface to bugzilla.g.o is probably
what we should be working on: extend pybugz (and the server side API)
somewhat and create a simple UI that asks some additional questions
(about steps to reproduce, a useful Summary/Description and so on) and
automatically submits that information. If a user wants to, he can then
flip a FEATURES switch which will automatically invoke that bugzilla
client, feeding it some preliminary information and a couple of files
to attach.

A better bug reporting client will not deal with all the problems of bad
reports (and probably exacerbate that problem), but it will stop spam
and should encourage users to file more bug reports, and would be
based on a vetted implementation that has already addressed many of
the concerns that your proposal would need to address from the ground
up.


     jer

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On 10 August 2014 06:15, Igor <lanthruster@gmail.com> wrote:

> Communication protocol is already there - it's HTTP, method POST
> HTTP protocol is already with Python - CURL, WGET
> A reliable server ready to accept data from portage is all so there -
> it's Apache web server.



For the sake of this discussion, those protocols serve only as a channel.

You have to send some kind of data structure over that channel indicating
data about state.

This requires the data to have a defined format, and potentially a
versioning scheme for that format.

And not just a defined format, like JSON for instance, but a defined
structure of JSON representation with keys and values being clearly
stipulated.

You can't just send arbitrary freeform JSON to a random server and hope it
pulls sense out of fat air.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1685 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

On 10 August 2014 07:22, Jeroen Roovers <jer@gentoo.org> wrote:

>
> Example 2: Mr. I.B. User configured his system with CFLAGS=-fomg-faster
> and now it generates a ton of build failures. All of these should go
> to /dev/null, but there we are running an automated service that cannot
> be taught how to distinguish between genuine bug reports and PEBKAC.
>
> The problem: How do you propose to filter out all the junk and promote
> genuine issues to "bug reports"?
>
> Both examples should make clear why we have a bug tracker and not an
> automated build failure reporting facility - it works rather well and
> it stops dead a lot of spam and bad reports.


^ In practice this doesn't prove to be an issue for CPAN, *because* it uses
a seperate channel from the bug reports.

The isolation means you can either choose to delve into the metrics, or
simply ignore them, they don't affect your workflow unless you want them to.

Users with misconfigured systems do occur from time to time, and you just
get a knack for knowing which reports are reports you can fix, and which
you just leave to rot.

And there's no *onus* on you to fix anything, because its only an automated
report, its intended to give you installation context *if* and *when* you
want it, as opposed to interrupting you and demanding attention.

For instance: http://matrix.cpantesters.org/?dist=IO+1.25

This shows the CPAN version of IO can't be installed on >5.18, on windows
and a few darwin boxes.

The maintainer of IO doesn't *HAVE* to do anything here, its just
information they can go swimming for if they want to.

=~ Its up to the maintainer if they want to respond to this information or
not.

If a user desires a response, they still have to file a bug report.


Though it does prove useful as a user filing a bug report to be able to see
all the other failure reports and know "Hey, its not just me, this is a
real problem and I'm not making it up", and be able to link to a bunch of
them, which cross comparison quickly reveals common elements to investigate
as probable failing points.

But I'm very much in agreement that such as system should NEVER be attached
as a pump handle to bugzilla itself.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 3408 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Tim Boudreau]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

 FWIW:  I have worked on a project for years where exception reporting was
used as a "pump handle" for Bugzilla.  It can be done;  the trick is
getting good data *in* and automating recognition of which failures are the
same failure, doing NOTHING until some threshold number of failures are
logged, and having a way to flag certain flavors of report as known-bogus.
 Here is an example failure report and resulting bug report:

http://statistics.netbeans.org/exceptions/detail.do?id=205871
https://netbeans.org/bugzilla/show_bug.cgi?id=239261

That being said, it was done for ONE language in ONE application, where the
error messages were detailed, meaningful and in a standard, Java-specific
format.

Doing that across the multiple languages, myriad bug tracking systems
(including none at all), for all packages anyone ever might build on Linux
sounds like a doomed enterprise.  I'm not saying don't do it - such
statistics might be interesting in aggregate - but don't have high hopes
for it solving the world's problems, and plan on simply publishing those
stats in a consumable way, promote their existence and plan on *attracting*
developers and projects to *want* to consume those, as opposed to
force-feeding every bug tracker in the universe, which I assure you, will
not win friends.

But the bottom line is:  Write a patch.  Prototype the server-side piece.
 People respond to working code;  hypothetical discussions about what
somebody else could or should do inevitably go nowhere.  If you write
something, nobody can say you're not committed to it, and *everybody* will
agree on what the thing does because they can see it, rather than guessing
at what "files a bug report" means and getting into arguments because
people are using the same words for different things.  You'll probably get
a better sense of the problem space and what's easy and what's hard about
it in the process, which will result in a better proposal.

If it's just telling other people what they ought to do, then it looks like
you're a dilettante, and people are rightly wary of that.

-Tim

[-- Attachment #2: Type: text/html, Size: 2819 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Igor]

[-- Attachment #1: Type: text/plain, Size: 2960 bytes --]

Hello Tim,

Sunday, August 10, 2014, 4:41:09 AM, you wrote:

I have no doubts that it could be done. 

What I'm perplexed about... 

The project consists of 3 major parts:

Reporting (portage -> Database <- Processing) 
Database   
Processing

Wouldn't it be logical to have Data Reporting unit working to some extent before everything else?
It's difficult to code imagining the most important part of the game - portage Reporting unit.

Database and Processing units both would require an expensive hardware to work, there should 
be some kind invitation from portage to spend a few thousand bucks. Not to mention time.

If the invitation in Portage is there a lot of people can try out their own Database and Processing 
units. Sounds fair to me.


FWIW:  I have worked on a project for years where exception reporting was used as a "pump handle" for Bugzilla.  It can be done;  the trick is getting good data *in* and automating recognition of which failures are the same failure, doing NOTHING until some threshold number of failures are logged, and having a way to flag certain flavors of report as known-bogus.  Here is an example failure report and resulting bug report:

http://statistics.netbeans.org/exceptions/detail.do?id=205871
https://netbeans.org/bugzilla/show_bug.cgi?id=239261

That being said, it was done for ONE language in ONE application, where the error messages were detailed, meaningful and in a standard, Java-specific format. 

Doing that across the multiple languages, myriad bug tracking systems (including none at all), for all packages anyone ever might build on Linux sounds like a doomed enterprise.  I'm not saying don't do it - such statistics might be interesting in aggregate - but don't have high hopes for it solving the world's problems, and plan on simply publishing those stats in a consumable way, promote their existence and plan on *attracting* developers and projects to *want* to consume those, as opposed to force-feeding every bug tracker in the universe, which I assure you, will not win friends.

But the bottom line is:  Write a patch.  Prototype the server-side piece.  People respond to working code;  hypothetical discussions about what somebody else could or should do inevitably go nowhere.  If you write something, nobody can say you're not committed to it, and *everybody* will agree on what the thing does because they can see it, rather than guessing at what "files a bug report" means and getting into arguments because people are using the same words for different things.  You'll probably get a better sense of the problem space and what's easy and what's hard about it in the process, which will result in a better proposal.

If it's just telling other people what they ought to do, then it looks like you're a dilettante, and people are rightly wary of that.

-Tim





-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 4175 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Igor]

[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]

Hello Kent,

Saturday, August 9, 2014, 11:49:43 PM, you wrote:

We have a lot of uncertainty even about what to send and how. It is
important for a data processor to understand what data is going to be fed before it's designed. 
If you have 1000 requests/second JSON parsing might be a problem and might require different 
solutions/hardware. 

If portage implements data the way it sees is - bad or good - that would reduce uncertainty and 
allow tools implemented.

OK, let's have another year for everyone to think over it.



On 10 August 2014 06:15, Igor <lanthruster@gmail.com> wrote:
Communication protocol is already there - it's HTTP, method POST
HTTP protocol is already with Python - CURL, WGET
A reliable server ready to accept data from portage is all so there - 
it's Apache web server. 


For the sake of this discussion, those protocols serve only as a channel.

You have to send some kind of data structure over that channel indicating data about state.

This requires the data to have a defined format, and potentially a versioning scheme for that format.

And not just a defined format, like JSON for instance, but a defined structure of JSON representation with keys and values being clearly stipulated.

You can't just send arbitrary freeform JSON to a random server and hope it pulls sense out of fat air.

-- 
Kent 

KENTNL - https://metacpan.org/author/KENTNL





-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 2758 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Hi,

On 09/08/14 18:18, Igor wrote:
> Hi all,
> 
> Hereby the summary of my personal suggestions to increase GENTOO stability and help it's maintainers 
> and developers. 
> 
> 1. make.conf
> 
>    Add 
> 
>    BUG_REPORT_URL  "http://"(or similar name)
>    BUG_REPORT ON/OFF
>    BUG_REPORT_LEVEL 
> 
>    to make.conf  

Apart from the huge work to implement this, I think any automatic
reporting of build failures will not be useful due to a very bad
signal to noise ratio.  Bug wrangling will be insane.  What are
the experiences in the industry.  Is Mozilla getting anything
useful out of their automatic crash reports?

Cheers,
Thomas


-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 536 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On 09/08/14 21:22, Jeroen Roovers wrote:
> A better bug reporting client will not deal with all the problems of bad
> reports (and probably exacerbate that problem), but it will stop spam
> and should encourage users to file more bug reports, and would be
> based on a vetted implementation that has already addressed many of
> the concerns that your proposal would need to address from the ground
> up.

Absolutely!



-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 536 bytes --]

Re: [gentoo-dev] A constructive reommendation on stablity improvemt

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/08/14 05:03 PM, Thomas Kahle wrote:
> Hi,
> 
> On 09/08/14 18:18, Igor wrote:
>> Hi all,
>> 
>> Hereby the summary of my personal suggestions to increase GENTOO
>> stability and help it's maintainers and developers.
>> 
>> 1. make.conf
>> 
>> Add
>> 
>> BUG_REPORT_URL  "http://"(or similar name) BUG_REPORT ON/OFF 
>> BUG_REPORT_LEVEL
>> 
>> to make.conf
> 
> Apart from the huge work to implement this, I think any automatic 
> reporting of build failures will not be useful due to a very bad 
> signal to noise ratio.  Bug wrangling will be insane.  What are the
> experiences in the industry.  Is Mozilla getting anything useful
> out of their automatic crash reports?
> 
> Cheers, Thomas
> 
> 


I haven't been able to track down the right people in mozilla to ask,
yet.  I have seen some bugs where the developers that work on fixing
an issue are indeed referencing the crash dumps that were submitted; i
don't know if or how well the crash report submitting links multiple
crashes together to the same issue, though.

Mozilla also has a very hefty 'try' (tinderbox) system, though, that
runs builds on every single commit across every platform that is
supported -- I believe they have a number of tools that help aggregate
the errors and warnings from those logs to provide useful output to
support development, and I think that model might be more comparable
to what we would need in Gentoo -- essentially, this project Igor is
suggesting would turn the Gentoo community into one massive tinderbox.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlPqIO0ACgkQ2ugaI38ACPBfaAD+IKBIQWf0L4WF2pd6iCzdDUU0
l11GSV6NfnpQYG/qilYBAI+bEUHYqxA75Uhrg+m9lqJ0CzwBpm4Tn0ya1MzmiXxC
=gi7Q
-----END PGP SIGNATURE-----