From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4382B138247 for ; Thu, 9 Jan 2014 12:44:14 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id F2050E0BD2; Thu, 9 Jan 2014 12:44:08 +0000 (UTC) Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 7E4D9E0B8E for ; Thu, 9 Jan 2014 12:44:07 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id l4so2240621lbv.21 for ; Thu, 09 Jan 2014 04:44:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:reply-to:message-id:to:subject:in-reply-to:references :mime-version:content-type; bh=SGi33cxU++Q2rcSkaKir+i2ugOxlyWRnEoj8zU3bbWU=; b=hQXmiNdn0EI/c9/c41bURlbXZVf9lKl+Zcqzu0CewXyiz+aVbpGffSI5zWnaco50x2 1JZAWY+HtkYUrpV6AKZa1oGCYGhW1LgYsv6Z9ZWrOZOAS3QAsKZJ8Rk4nReqHTtYPL92 HfApFrNN4vKVst/N/iUnwMXOyhmxh6hPTMEf9Ceq4sX35Bu6Jk69YVCVASvqkbId607E /Z/J86dGjTxrjuAWWYGsEAM8vMBorkdijtMQ3qunTeGnnHp5/cZkOG5gQm3fd5dI+l/8 RVwe6QBxwg+MiheaIqq0tSL1vZ4yDZrhC/Fep1kU7JB+M8WwjhmgwXVDlf4Y9GU315/a JDsA== X-Received: by 10.112.130.35 with SMTP id ob3mr1144610lbb.2.1389271445693; Thu, 09 Jan 2014 04:44:05 -0800 (PST) Received: from [192.168.60.64] (office.healtech.ru. [89.208.21.2]) by mx.google.com with ESMTPSA id xl4sm1530107lac.9.2014.01.09.04.44.03 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 09 Jan 2014 04:44:04 -0800 (PST) Date: Thu, 9 Jan 2014 16:44:02 +0400 From: Igor X-Priority: 3 (Normal) Message-ID: <52ce9994.24f5980a.0660.342e@mx.google.com> To: Alec Warner Subject: Re: [gentoo-dev] Portage QOS In-Reply-To: References: <52ce4eab.463f700a.4b43.16bd@mx.google.com> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----------1251052442FA0252B" X-Archives-Salt: e425093b-1c12-4e6a-91cd-0274429f17c8 X-Archives-Hash: 57a082de2451b1b189e111b0d96535b1 ------------1251052442FA0252B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello Alec, Thursday, January 9, 2014, 12:12:18 PM, you wrote: On Wed, Jan 8, 2014 at 11:24 PM, LTHR wrote: Hi All, I want to start off by discussing your premise, before embarking on the ove= rall goals. You wrote: "I'm with Gentoo for many years. For various reasons many techs were not im= plemented and now Gentoo is in a kind of stagnation. But we can give Gentoo= a new birth with relatively little efforts and bring the distro to the who= le new level. " I don't actually believe your premise of stagnation. But I can put aside my= disagreement for now. True. There is no data to tell what happens with Gentoo (to give that data = is one of the goals of the=20 project). We only have some formal esteems from unreliable sources.=20 According to distro watch:=20 http://web.archive.org/web/20130701000000*/http://distrowatch.com/dwres.php= ?resource=3Dpopularity In February 2012, Gentoo distro was in 19th place.=20 In December 2012, Gentoo went to 22nd place.=20 In December 2013, Gentoo is down to 32nd place=20 According to Linux Counter=20 http://web.archive.org/web/20120101000000*/http://linuxcounter.net/distribu= tions/stats.html In January 2012, Gentoo distro had 5.32% In January 2012, Gentoo had 4.04% In November 2013, Gentoo had 4,21%=20 And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at= least not gaining new users. If in several years the number of users is not increased - we can tell abou= t stagnation.=20 Why new users are not with Gentoo and how to get them - good question, let'= s find out! Lets talk about how the overall goal of 'bringing the distro to a whole ne= w level' and how 'Portage QoS' will help us get there. I don't think you co= vered these points well in your post (I will talk about the goals more belo= w...) I'll re-phrase the goals.=20 It's all not very well thought after at this stage but immediate goals are = like this:=20 * Knowledge of the number of Gentoo distros installed world wide - knowing = the trend how many users choose=20 Gentoo and where Gentoo is really going down|up|stands still.=20 =20 You can then try different features and see how a feature is met - if the= number of systems increase=20 then this feature is probably useful. It's a strategical job, somebody at= the very top of the project should=20 analyze databases and make conclusions.=20 * Knowledge of the ebuild popularity - what ebuilds are popular and what ar= e not - what ebuild to give an extra focus=20 and what ebuild could wait * Knowledge of ebuild quality. If some ebuilds fail on many systems - somet= hing is wrong and ebuild and may be portage=20 need fixing. It's especially useful to make sure that all ebuilds have co= rrect dependencies, missing dependencies, etc. * A formal esteem of portage quality=20 PortageQ =3D (the number of successful ebuilds/the number of all ebuild a= ttempts) Portage speed efficiency: =20 Average time before build starts (or download starts) How many times portage fails itself.=20 * Immediate problem detection. If the number of PortageQ went down last day= - there is some problem.=20 (then you go to ebuild stats and see what is failing) * Reducing load on bugtracker folks - the build problems will be detected a= utomatically and solved according=20 to their importance. There will be no need to supply bug tracker with ebu= ild logs and emerge --info if=20 somebody wants to report a problem.=20 * Team efficiency esteem. The stats will tell what ebuilds are failing most= often.=20 * Team automated info. When failure rate of a certain ebuild increase the m= aintainer is automatically=20 informed and he can login in web-interface and see details how exactly eb= uild failed.=20 The same for the portage itself. Next day a maintainer could push a new e= build in the portage and the=20 problem might be solved. It's not possible not to make mistakes. But it's possible to react on the= ir consequences fast.=20 * Knowledge what kernels are used by Gentoo users, how often they update th= eir systems, what flags=20 are used=20 2nd turn goals:=20 * to integrate forums.gentoo.org and bug tracker. People are offering great= workarounds and solutions. But=20 they're not known to the majority of Gentoo users.=20 If a e-build fails - may be there is already a solution - and we can offe= r the solutions automatically from=20 portage. Like: There might be some work-arounds on this problem:=20 [Gentoo Forum - qt-core ebuild fails - SOLVED]=20 htpp://forums.gentoo.org/.....=20 There is a known bug on this ebuild:=20 [Gentoo Bug - qt-core ebuild fails]=20 htpp://forums.gentoo.org/.....=20 * to make Bug Tracker almost unmanned. We can use gathered infromation on f= ailed e-builds to=20 create bugs in Bug Tracker automatically and automatically set priorities= according to the=20 severity.=20 The severity could be assigned automatically from package popularity and = failure rate stats. The users with the problems could receive e-mail automatically to follow = up the quick arounds and solutions.=20 No need to change bug-tracker version, you just need a robot who would ma= intain bug-tracker=20 analyazing PortageQOS databases.=20 I'm sure there could be more applications of the system.=20 Back to the premise of bringing the distro to a whole new level. Some of th= e items above I think have merits on their own, but they still don't guide = me to your ultimate goal. You outlined some of what I presume to be defects= in Gentoo today. It's hard to see more goals at this stage. It's like when you're building a= hammer to set a nail and=20 then when you have it you discover that you may use it to pull a nail too. = It's for the future developers=20 to decide.=20 Knowledge: So I think in general I agree, insofar as more information can b= e helpful in making decisions. I think you should take note that there are = at least 4-5 'gentoo stats' projects that have been tried and my understand= ing is that none of them are in operation today. True, one of the reasons - the whole system planning shouldn't be apart fro= m Gentoo developers.=20 I could write the whole system on paper, with tools, soft used and the syst= em design not narrowing to the table structures. Then everyone could contribute his experience to it or see a problem at des= ign time.=20 I know how to design the system to make handling significant load. It has a= good chance=20 to work fast and to be scalable.=20 With the design approved by others, programming is easy.=20 It's not going to be very complex. I estimate the whole system as 10 000 - 15 000 lines max in different langu= ages. The most difficult part would=20 be to make it scalable and fast as we don't know how many ebuilds are done = world wide per second and we=20 have to be prepared to hot-plug hardware at any time not redesigning the sy= stem. And the system shouldn't get=20 overloaded/loose functionality.=20 Package blocks that portage does not resolve automatically Slot conflicts that portage does not resolve automatically Compile failures ...What else? So part of the Portage QoS system is that users will submit their failures = and Portage QoS will serve as a knowledge base of known issues. To me, that= is still a pretty bad User Experience. Can't we just get portage to handle= these issues transparently, as per http://forums.gentoo.org/viewtopic-t-97= 7936.html If we ask users about questions they don't know answers to it will result i= n users loss. But it's totally=20 out of my expertise if the portage interaction is GOOD=20 Users are not doing anything usual than they do right now. The reports are = sent transparently by=20 portage to the PorageQOS load balancer and the suggestions are received. Just a brief question - does anyone know how many ebuilds are assembled wor= ld=20 wide each second? I bet you more apks are installed per second ;) -A =20 --=20 Best regards, LTHR mailto:lanthruster@gmail.com --=20 Best regards, Igor mailto:lanthruster@gmail.com ------------1251052442FA0252B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Re: [gentoo-dev] Portage QOS Hello Alec,

Thursday, January 9, 2014, 12:12:18 PM, you wrote:


 lan= thruster@gmail.com> wrote:
Hi All,


I want to start off by discussing your premise, before embarking on the ove= rall goals.

You wrote:
"I'm with Gentoo for many years. For various reasons many techs were not im= plemented and now Gentoo is in a kind of stagnation. But we can give Gentoo= a new birth with relatively little efforts and bring the distro to the who= le new level. "

I don't actually believe your premise of stagnation. But I can put aside my= disagreement for now.


True. There is= no data to tell what happens with Gentoo (to give that data is one of the = goals of the 
project). We only have some formal esteems from unreliable sources. 
According to distro watch: 

http://web.archive.org/web/20130701000000*/http= ://distrowatch.com/dwres.php?resource=3Dpopularity

In February 20= 12, Gentoo distro was in 19th place. 
In December 2012, Gentoo went to 22nd place. 
In December 2013, Gentoo is down to 32nd place 

According to Linux Counter 
http://web.archive.org/web/20120101000000*/http://linuxcoun= ter.net/distributions/stats.html

In January 201= 2, Gentoo distro had 5.32%
In January 2012, Gentoo had 4.04%
In November 2013, Gentoo had 4,21% 

And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at= least not gaining new users.
If in several years the number of users is not increased - we can tell abou= t stagnation. 

Why new users are not with Gentoo and how to get them - good question, let'= s find out!


 I'll re-phrase= the goals. 

It's all not very well thought after at this stage but immediate goals are = like this: 


* Knowledge of the number of Gentoo distros installed world wide - knowing = the trend how many users choose 
  Gentoo and where Gentoo is really going down|up|stands still. <= br>   
  You can then try different features and see how a feature is met - i= f the number of systems increase 
  then this feature is probably useful. It's a strategical job, somebo= dy at the very top of the project should 
  analyze databases and make conclusions. 

* Knowledge of the ebuild popularity - what ebuilds are popular and what ar= e not - what ebuild to give an extra focus 
  and what ebuild could wait

* Knowledge of ebuild quality. If some ebuilds fail on many systems - somet= hing is wrong and ebuild and may be portage 
  need fixing. It's especially useful to make sure that all ebuilds ha= ve correct dependencies, missing dependencies, etc.

* A formal esteem of portage quality 
  PortageQ =3D (the number of successful ebuilds/the number of all ebu= ild attempts)

  Portage speed efficiency:  
  Average time before build starts (or download starts)

  How many times portage fails itself. 

* Immediate problem detection. If the number of PortageQ went down last day= - there is some problem. 
  (then you go to ebuild stats and see what is failing)

* Reducing load on bugtracker folks - the build problems will be detected a= utomatically and solved according 
  to their importance. There will be no need to supply bug tracker wit= h ebuild logs and emerge --info if 
  somebody wants to report a problem. 

* Team efficiency esteem. The stats will tell what ebuilds are failing most= often. 

* Team automated info. When failure rate of a certain ebuild increase the m= aintainer is automatically 
  informed and he can login in web-interface and see details how exact= ly ebuild failed. 
  The same for the portage itself. Next day a maintainer could push a = new ebuild in the portage and the 
  problem might be solved.

  It's not possible not to make mistakes. But it's possible to react o= n their consequences fast. 

* Knowledge what kernels are used by Gentoo users, how often they update th= eir systems, what flags 
  are used 

2nd turn goals: 

* to integrate forums.gentoo.org and bug tracker. People are offering great= workarounds and solutions. But 
  they're not known to the majority of Gentoo users. 

  If a e-build fails - may be there is already a solution - and we can= offer the solutions automatically from 
  portage. Like:

  There might be some work-arounds on this problem: 
  [Gentoo Forum - qt-core ebuild fails - SOLVED] 
  htpp://forums.gentoo.org/..... 

  There is a known bug on this ebuild: 
  [Gentoo Bug - qt-core ebuild fails] 
  htpp://forums.gentoo.org/..... 

* to make Bug Tracker almost unmanned. We can use gathered infromation on f= ailed e-builds to 
  create bugs in Bug Tracker automatically and automatically set prior= ities according to the 
  severity. 

  The severity could be assigned automatically from package popularity= and failure rate stats.

  The users with the problems could receive e-mail automatically to fo= llow up the quick arounds
  and solutions. 

  No need to change bug-tracker version, you just need a robot who wou= ld maintain bug-tracker 
  analyazing PortageQOS databases. 


I'm sure there could be more applications of the system. 



 It's hard to s= ee more goals at this stage. It's like when you're building a hammer to set= a nail and 
then when you have it you discover that you may use it to pull a nail too. = It's for the future developers 
to decide. 

Knowledge: So I think in general I agree, insofar as more information ca= n be helpful in making decisions. I think you should take note that there a= re at least 4-5 'gentoo stats' projects that have been tried and my underst= anding is that none of them are in operation today.

True, one of the reasons - the whole system planning shouldn't be apart= from Gentoo developers. 

I could write the whole system on paper, with tools, soft used and the syst= em design not narrowing to the table structures.
Then everyone could contribute his experience to it or see a problem at des= ign time. 

I know how to design the system to make handling significant load. It has a= good chance 
to work fast and to be scalable. 


With the design approved by others, programming is easy. 

It's not going to be very complex.
I estimate the whole system as 10 000 - 15 000 lines max in different langu= ages. The most difficult part would 
be to make it scalable and fast as we don't know how many ebuilds are done = world wide per second and we 
have to be prepared to hot-plug hardware at any time not redesigning the sy= stem. And the system shouldn't get 
overloaded/loose functionality. 

Package blocks= that portage does not resolve automatically
Slot conflicts that portage does not resolve automatically
Compile failures
...What else?

So part of the Portage QoS system is that users will submit their failures = and Portage QoS will serve as a knowledge base of known issues. To me, that= is still a pretty bad User Experience. Can't we just get portage to handle= these issues transparently, as per http://forums.gentoo.org/viewtopic-t-977936.html




If we ask user= s about questions they don't know answers to it will result in users loss. = But it's totally 
out of my expertise if the portage interaction is GOOD 

Users are not doing anything usual than they do right now. The reports are = sent transparently by 
portage to the PorageQOS load balancer and the suggestions are received.

Just a brief q= uestion - does anyone know how many ebuilds are assembled world 
wide each second?

I bet you more apks are installed per second ;)

-A
 
--=  
Best regards,
 LTHR                   &= nbsp;      -- 
Best regards,
 Igor                   &= nbsp;